How to describe tokens?

. Programming language tokens can be described by regular languages

. Regular languages

- Are easy to understand

- There is a well understood and useful theory

- They have efficient implementation

. Regular languages have been discussed in great detail in the "Theory of Computation" course

Here we address the problem of describing tokens. Regular expression is an important notation for specifying patterns. Each pattern matches a set of strings, so regular expressions will serve as names for set of strings. Programming language tokens can be described by regular languages. The specification of regular expressions is an example of an recursive definition. Regular languages are easy to understand and have efficient implementation. The theory of regular languages is well understood and very useful. There are a number of algebraic laws that are obeyed by regular expression which can be used to manipulate regular expressions into equivalent forms. We will look into more details in the subsequent slides.