Project, Phase 1: Lexical Analysis

Assigned: Wednesday, 7 February 2001
Due:Friday, 16 February 2001

Summary: In this stage of the project, you will design and build the lexical analyzer for your Pascal compilers.

Warning: You may be required to use each others' lexers at the next stage of the project.

Group Work: You should work in groups of size 3. I am likely to reassign groups for the next stage of the project.

Building a Lexical Analyzer

1. Begin by deciding on the natural tokens for Pascal. You should do this in the next day or so.

2. Design Java classes for tokens. You should have a general Token interface or abstract class and then make classes for the indivdiual tokens that subclass or implement Token (or that subclass a subclass or implement a subinterface of Token).

3. Write regular expressions for the tokens.

4. Design a generic Lexer class that provides appropriate methods, such as nextToken and peekToken.

5. Implement the lexical analyzer (yeah, you knew there had to be a hard part, didn't you?).

6. Write a test program that reads in files and prints out their tokens (and, possibly, reports on errors as it encounters them).

Implementation Options

You are free to implement the lexical analyzer in any manner you see fit. There are a number of options.

You can hand code the analyzer. This solution potentially gives you the most freedom (and perhaps even efficiency). For example, you can probably deal with some non-regular issues with this solution.

You can use an existing Java lexical analyzer generator. This solution is what many commercial compilers do. However, you will probably need to put a wrapper class around the results to create a more generic result. You will also need to learn the specifics of one of these systems.

You might consider using

• ANTLR (or at least its lexical analysis phase).
• JavaCC (or at least its lexical analysis phase).
• JFlex

There are also probably others out there.

You can rely on built-in Java classes, particularly StringTokenizer and StreamTokenizer.

You can write your own lexical analyzer generator. You'll certainly learn a lot of lexical analysis, regular expressions, and automata if you choose this solution. However, it is also requires much more effort than the other options.