What are Python Tokens?
What are Tokens in context with programming Languages?
Before we look into – What are Python Tokens? let’s first understand What are tokens?. Tokens are nothing but Lexical components or units which have an independent meaning in the programming language.
Python Tokens
When we write a Python program, interpreter breaks each logical line into a sequence of elementary lexical units. These lexical components are known as tokens.
These tokens can further be classified into following types for better understanding: identifiers,
- keywords
- operators
- delimiters, and
- literals
Keywords
Python supported keywords as part of Python Tokens are listed below. Observe all these keywords are in lowercase.
| and | del | for | is | raise |
| assert | elif | from | lambda | return |
| break | else | global | not | try |
| class | except | if | or | while |
| continue | exec | import | pass | with (2.5) |
| def | finally | in | yield |
Identifiers which have some special meaning attached by Python and are thus are reserved by python are known as keywords.
Operators
Python uses following python tokens to represent operators in Python. You will observe all of them are non-alphanumeric characters. Here we are discussing Operators as part of Python Tokens. Their functionality and usage will be discussed in the later part of the course.
| + | – | * | / | % | ** | // | << | >> | & |
| | | ^ | ~ | < | <= | > | >= | <> | != | == |
Delimeters
The following symbols and symbol combinations are used by Python interpreter as delimiters. (Delimeter is othing but a sequence o one or more characters used to seperate tokens from one another, like we use space and even comma (,) to separate two or more words.)
| ( | ) | [ | ] | { | } |
| , | : | . | ` | = | ; |
| += | -= | *= | /= | //= | %= |
| &= | |= | ^= | >>= | <<= | **= |
Literals
Literals or constants are the values that are represented in the source code (program) exactly the same way as it should be interpreted. It can be a number, String or any other data type written directly as it is in the program.
For example:
X=10 X=X+5 print(10) print(X)
In the code above, 10 is literal, its value can’t change and is written as it is. On the contrary X is a variable and its value can change during program run.
