CSE340
Project 1: Lexical Analysis
The goal of this project is to give you hands-on experience with lexical analysis. You will extend the
provided lexical analyzer to support more token types. The next section lists all new token types that you
need to implement.
1. Token Types
Modify the lexer to support the following 3 token types:
REALNUM
NUM DOT digit digit*
BASE08NUM ((pdigit8 digit8*) + 0) (x) (08)
BASE16NUM
=
((pdigit16 digit16*) + 0) (x) (16)
Where
digit16
pdigit16
digit
=
012 + 3 + 4 + 5 + 6 + 7+
=
=
+ 2 + 3 + 4 + 5 + 6 + 7 + 8 +
01 + 2 + 3 + 4 + 5 + 6 +
pdigit
digit8
pdigit8
=
8+ 9+ A + B + C + D + E + F
9 + A + B + C + D + E + F
7+ 8+ 9
1 + 2 + 3 + 4 + 5 + 6 + 7+ 8+ 9
=
= 1 + 2 + 3 + 4 + 5
01 + 2 + 3 + 4 + 5 + 6 + 7
+ 6 + 7
Note that NUM and DOT are already defined in the lexer, but here are the regular expressions for the
sake of completeness:
=
NUM
DOT = "."
(pdigit digit*) + 0
Note that DOT is a single dot character, the quotes are used to avoid ambiguity. The list of valid tokens including the existing tokens in the code would be as follows:
IF
WHILE
DO
THEN
PRINT
PLUS
MINUS
DIV
MULT
EQUAL
COLON
COMMA
SEMICOLON
LBRACE
RBRACE
LPAREN
RPAREN
NOTEQUAL
GREATER
LESS
LTEQ
GTEQ
DOT
NUM
ID
REALNUM
BASE08NUM
BASE16NUM
This list should be used to determine the token if the input matches more than one regular expression. 2. How-To
Follow these steps:
• Download the lexer.cc, lexer.h, inputbuf.cc and inputbuf.h files accompanying this project
description. Note that these files might be a little different than the code you've seen in class or
elsewhere.
• Add your code to the files to support the token types listed in the
previous section.
• Compile your code using GCC compiler in Ubuntu 19.04 or higher. You will need to use the
g++ command to compile your code in a terminal window.
Note that you are required to compile and test your code in Ubuntu 19.04 or higher using the
GCC compilers. You are free to use any IDE or text editor on any platform, however, using tools
available in Ubuntu 19.04 g++ version 7.5.0 (or tools that you could install on Ubuntu 19.04 could
save time in the development/compile/test cycle. See next section for more details on how to compile
using GCC.
• Test your code to see if it passes the provided test cases. You will need to extract the test cases
from the zip file and run the test script test1.sh. More details on this in the next section.
Submit your code in Gradescope before the deadline:
For this project you need to update lexer.cc and lexer.h. Since inputbuf.h and inputbuf.cc should
not be modified, you do not need to upload them to Gradescope. 3. Compile & Test
3.1 Compiling Code with GCC
You should compile your programs with the GCC compilers which are available in g++ 7.5.0 in Ubuntu
19.04. The GCC is a collection of compilers for many programming languages. There are separate
commands for compiling C and C++ programs:
•
Use gcc command to compile C programs
Use g++ to compile C++ programs
Here is an example of how to compile a simple C++ program:
$ g++ test_program.cpp
If the compilation is successful, gcc will generate an executable file named a.out in the same folder as the
program. You can change the output file name by specifying the -o switch:
$ g++ test_program.cpp -o hello.out
To enable all warning messages of the GCC compiler, use the -Wall
switch:
$ g++ -Wall test_program.cpp -o hello.out
The same options can be used with gcc to compile C programs.
We use c++11 on Gradescope to compile the codes; so to make sure your code compiles correctly on
Gradescope, use the following option to compile your code with c++11:
$ g++ -std=c++11 -Wall_test_program.cpp -o hello.out
Compiling projects with multiple files
If your program is written in multiple source files that should be linked together, you can compile and
link all files together with one command:
$ g++ filel.cpp file2.cpp file3.cpp
Or you can compile them separately and then link:
$ g++ file1.cpp
$ g++ file2.cpp
$ g++ file3.cpp
$ g++ filel.o file2.0 file3.0 The files with the .o extension are object files but are not executable. They are linked together
with the last statement and the final executable will be a.out.
NOTE: you can replace g++ with gcc in all examples listed above to compile C programs.
3.2 Testing your code with I/O Redirection
Your programs should not explicitly open any file. You can only use the standard input e.g.
std::cin in C++, getchar(), scanf() in C and standard output e.g. std::cout in C++, putchar(), printf()
in C for input/output.
However, this restriction does not limit our ability to feed input to the program from files nor does it
mean that we cannot save the output of the program in a file. We use a technique called standard IO
redirection to achieve this.
Suppose we have an executable program a.out, we can run it by issuing the following command in a terminal
(the dollar sign is not part of the command):
$ ./a.out
If the program expects any input, it waits for it to be typed on the keyboard and any output generated
by the program will be displayed on the terminal screen. Now to feed input to the program from a
file, we can redirect the standard input to a file:
$ ./a.out <input_data.txt
Now, the program will not wait for keyboard input, but rather read its input from the specified file. We
can redirect the output of the program as well:
$ ./a.out output_file.txt
In this way, no output will be shown in the terminal window, but rather it will be saved to the specified
file. Note that programs have access to another standard interface which is called standard error e.g.
std::cerr in C++, fprintf(stderr,...) in C.
