What are Python Tokens?

What are Tokens in context with programming Languages?

Before we look into – What are Python Tokens? let’s first understand What are tokens?. Tokens are nothing but Lexical components or units which have an independent meaning in the programming language.

Python Tokens

When we write a Python program, interpreter breaks each logical line into a sequence of elementary lexical  units. These lexical components are known as tokens.

These tokens can further be classified into following types for better understanding: identifiers,

  • keywords
  • operators
  • delimiters, and
  • literals


Python supported keywords as part of Python Tokens are listed below. Observe all these keywords are in lowercase.

and del for is raise
assert elif from lambda return
break else global not try
class except if or while
continue exec import pass with (2.5)
def finally in print yield

Identifiers which have some special meaning attached by Python and are thus are reserved by python are known as keywords.


Python uses following python tokens to represent operators in Python. You will observe all of them are non-alphanumeric characters. Here we are discussing Operators as part of Python Tokens. Their functionality and usage will be discussed in the later part of the course.

+ * / % ** // << >> &
| ^ ~ < <= > >= <> != ==


The following symbols and symbol combinations are used by Python interpreter as delimiters. (Delimeter is othing but a sequence o one or more characters used to seperate tokens from one another, like we use space and even comma (,) to separate two or more words.)

( ) [ ] { }
, : . ` = ;
+= -= *= /= //= %=
&= |= ^= >>= <<= **=


Literals or constants are the values that are represented in the source code (program) exactly the same way as it should be interpreted. It can be a number, String  or any other data type written directly as it is in the program.

For example:


In the code above, 10 is literal, its value can’t change and is written as it is. On the contrary X is a variable and its value can change during program run.