CSE340 Project 1: Lexical Analysis The goal of this project is to give you hands-on experience with lexical analysis. You will extend the provided lexical analyzer to support more token

types. The next section lists all new token types that you need to implement. 1. Token Types Modify the lexer to support the following 3 token types: REALNUM NUM DOT digit digit* BASE08NUM ((pdigit8 digit8*) + 0) (x) (08) BASE16NUM = ((pdigit16 digit16*) + 0) (x) (16) Where digit16 pdigit16 digit = 012 + 3 + 4 + 5 + 6 + 7+ = = + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 01 + 2 + 3 + 4 + 5 + 6 + pdigit digit8 pdigit8 = 8+ 9+ A + B + C + D + E + F 9 + A + B + C + D + E + F 7+ 8+ 9 1 + 2 + 3 + 4 + 5 + 6 + 7+ 8+ 9 = = 1 + 2 + 3 + 4 + 5 01 + 2 + 3 + 4 + 5 + 6 + 7 + 6 + 7 Note that NUM and DOT are already defined in the lexer, but here are the regular expressions for the sake of completeness: = NUM DOT = "." (pdigit digit*) + 0 Note that DOT is a single dot character, the quotes are used to avoid ambiguity. The list of valid tokens including the existing tokens in the code would be as follows: IF WHILE DO THEN PRINT PLUS MINUS DIV MULT EQUAL COLON COMMA SEMICOLON LBRACE RBRACE LPAREN RPAREN NOTEQUAL GREATER LESS LTEQ GTEQ DOT NUM ID REALNUM BASE08NUM BASE16NUM This list should be used to determine the token if the input matches more than one regular expression. 2. How-To Follow these steps: • Download the lexer.cc, lexer.h, inputbuf.cc and inputbuf.h files accompanying this project description. Note that these files might be a little different than the code you've seen in class or elsewhere. • Add your code to the files to support the token types listed in the previous section. • Compile your code using GCC compiler in Ubuntu 19.04 or higher. You will need to use the g++ command to compile your code in a terminal window. Note that you are required to compile and test your code in Ubuntu 19.04 or higher using the GCC compilers. You are free to use any IDE or text editor on any platform, however, using tools available in Ubuntu 19.04 g++ version 7.5.0 (or tools that you could install on Ubuntu 19.04 could save time in the development/compile/test cycle. See next section for more details on how to compile using GCC. • Test your code to see if it passes the provided test cases. You will need to extract the test cases from the zip file and run the test script test1.sh. More details on this in the next section. Submit your code in Gradescope before the deadline: For this project you need to update lexer.cc and lexer.h. Since inputbuf.h and inputbuf.cc should not be modified, you do not need to upload them to Gradescope. 3. Compile & Test 3.1 Compiling Code with GCC You should compile your programs with the GCC compilers which are available in g++ 7.5.0 in Ubuntu 19.04. The GCC is a collection of compilers for many programming languages. There are separate commands for compiling C and C++ programs: • Use gcc command to compile C programs Use g++ to compile C++ programs Here is an example of how to compile a simple C++ program: $ g++ test_program.cpp If the compilation is successful, gcc will generate an executable file named a.out in the same folder as the program. You can change the output file name by specifying the -o switch: $ g++ test_program.cpp -o hello.out To enable all warning messages of the GCC compiler, use the -Wall switch: $ g++ -Wall test_program.cpp -o hello.out The same options can be used with gcc to compile C programs. We use c++11 on Gradescope to compile the codes; so to make sure your code compiles correctly on Gradescope, use the following option to compile your code with c++11: $ g++ -std=c++11 -Wall_test_program.cpp -o hello.out Compiling projects with multiple files If your program is written in multiple source files that should be linked together, you can compile and link all files together with one command: $ g++ filel.cpp file2.cpp file3.cpp Or you can compile them separately and then link: $ g++ file1.cpp $ g++ file2.cpp $ g++ file3.cpp $ g++ filel.o file2.0 file3.0 The files with the .o extension are object files but are not executable. They are linked together with the last statement and the final executable will be a.out. NOTE: you can replace g++ with gcc in all examples listed above to compile C programs. 3.2 Testing your code with I/O Redirection Your programs should not explicitly open any file. You can only use the standard input e.g. std::cin in C++, getchar(), scanf() in C and standard output e.g. std::cout in C++, putchar(), printf() in C for input/output. However, this restriction does not limit our ability to feed input to the program from files nor does it mean that we cannot save the output of the program in a file. We use a technique called standard IO redirection to achieve this. Suppose we have an executable program a.out, we can run it by issuing the following command in a terminal (the dollar sign is not part of the command): $ ./a.out If the program expects any input, it waits for it to be typed on the keyboard and any output generated by the program will be displayed on the terminal screen. Now to feed input to the program from a file, we can redirect the standard input to a file: $ ./a.out