How to specify tokens?

. How to describe tokens

2.e0 20.e-01 2.000

. How to break text into token

if (x==0) a = x << 1;

iff (x==0) a = x < 1;

. How to break input into token efficiently

- Tokens may have similar prefixes

- Each character should be looked at only once

The various issues which concern the specification of tokens are:

1. How to describe the complicated tokens like e0 20.e-01 2.000

2. How to break into tokens the input statements like if (x==0) a = x << 1; iff (x==0) a = x < 1;

3. How to break the input into tokens efficiently?There are the following problems that are encountered:

- Tokens may have similar prefixes

- Each character should be looked at only once