Although this manual covers all aspects of the CUP system, it is relatively brief, and assumes you have at least a little bit of knowledge of LR parsing. A working knowledge of YACC is also very helpful in understanding how CUP specifications work. A number of compiler construction textbooks (such as [2,3]) cover this material, and discuss the YACC system (which is quite similar to this one) as a specific example.
Using CUP involves creating a simple specification based on the grammar for which a parser is needed, along with construction of a scanner capable of breaking characters up into meaningful tokens (such as keywords, numbers, and special symbols).
As a simple example, consider a system for evaluating simple arithmetic expressions over integers. This system would read expressions from standard input (each terminated with a semicolon), evaluate them, and print the result on standard output. A grammar for the input to such a system might look like:
expr_list ::= expr_list expr_part | expr_part expr_part ::= expr ';' expr ::= expr '+' expr | expr '-' expr | expr '*' expr | expr '/' expr | expr '%' expr | '(' expr ')' | '-' expr | numberTo specify a parser based on this grammar, our first step is to identify and name the set of terminal symbols that will appear on input, and the set of non-terminal symbols. In this case, the non-terminals are:
expr_list, expr_part and expr .For terminal names we might choose:
SEMI, PLUS, MINUS, TIMES, DIVIDE, MOD, NUMBER, LPAREN, and RPARENThe experienced user will note a problem with the above grammar. It is ambiguous. An ambiguous grammar is a grammar which, given a certain input, can reduce the parts of the input in two different ways such as to give two different answers. Take the above grammar, for example. given the following input:
// CUP specification for a simple expression evaluator (no actions) import java_cup.runtime.*; /* Preliminaries to set up and use the scanner. */ init with {: scanner.init(); :}; scan with {: return scanner.next_token(); :}; /* Terminals (tokens returned by the scanner). */ terminal SEMI, PLUS, MINUS, TIMES, DIVIDE, MOD; terminal UMINUS, LPAREN, RPAREN; terminal Integer NUMBER; /* Non terminals */ non terminal expr_list, expr_part; non terminal Integer expr, term, factor; /* Precedences */ precedence left PLUS, MINUS; precedence left TIMES, DIVIDE, MOD; precedence left UMINUS; /* The grammar */ expr_list ::= expr_list expr_part | expr_part; expr_part ::= expr SEMI; expr ::= expr PLUS expr | expr MINUS expr | expr TIMES expr | expr DIVIDE expr | expr MOD expr | MINUS expr %prec UMINUS | LPAREN expr RPAREN | NUMBER ;
To produce a parser from this specification we use the CUP generator. If this specification were stored in a file parser.cup, then (on a Unix system at least) we might invoke CUP using a command like:
java java_cup.Main < parser.cupIn this case, the system will produce two Java source files containing parts of the generated parser: sym.java and parser.java. As you might expect, these two files contain declarations for the classes sym and parser. The sym class contains a series of constant declarations, one for each terminal symbol. This is typically used by the scanner to refer to symbols (e.g. with code such as "return new Symbol(sym.SEMI);" ). The parser class implements the parser itself.
The specification above, while constructing a full parser, does not perform any semantic actions &emdash; it will only indicate success or failure of a parse. To calculate and print values of each expression, we must embed Java code within the parser to carry out actions at various points. In CUP, actions are contained in code strings which are surrounded by delimiters of the form {: and :} (we can see examples of this in the init with and scan with clauses above). In general, the system records all characters within the delimiters, but does not try to check that it contains valid Java code.
A more complete CUP specification for our example system (with actions
embedded at various points in the grammar) is shown below:
// CUP specification for a simple expression evaluator (w/ actions) import java_cup.runtime.*; /* Preliminaries to set up and use the scanner. */ init with {: scanner.init(); :}; scan with {: return scanner.next_token(); :}; /* Terminals (tokens returned by the scanner). */ terminal SEMI, PLUS, MINUS, TIMES, DIVIDE, MOD; terminal UMINUS, LPAREN, RPAREN; terminal Integer NUMBER; /* Non-terminals */ non terminal expr_list, expr_part; non terminal Integer expr; /* Precedences */ precedence left PLUS, MINUS; precedence left TIMES, DIVIDE, MOD; precedence left UMINUS; /* The grammar */ expr_list ::= expr_list expr_part | expr_part; expr_part ::= expr:e {: System.out.println("= " + e); :} SEMI ; expr ::= expr:e1 PLUS expr:e2 {: RESULT = new Integer(e1.intValue() + e2.intValue()); :} | expr:e1 MINUS expr:e2 {: RESULT = new Integer(e1.intValue() - e2.intValue()); :} | expr:e1 TIMES expr:e2 {: RESULT = new Integer(e1.intValue() * e2.intValue()); :} | expr:e1 DIVIDE expr:e2 {: RESULT = new Integer(e1.intValue() / e2.intValue()); :} | expr:e1 MOD expr:e2 {: RESULT = new Integer(e1.intValue() % e2.intValue()); :} | NUMBER:n {: RESULT = n; :} | MINUS expr:e {: RESULT = new Integer(0 - e.intValue()); :} %prec UMINUS | LPAREN expr:e RPAREN {: RESULT = e; :} ;
expr:e1 PLUS expr:e2 {: RESULT = new Integer(e1.intValue() + e2.intValue()); :}the first non-terminal expr has been labeled with e1, and the second with e2. The left hand side value of each production is always implicitly labeled as RESULT.
Each symbol appearing in a production is represented at runtime by an object of type Symbol on the parse stack. The labels refer to the instance variable value in those objects. In the expression expr:e1 PLUS expr:e2, e1 and e2 refer to objects of type Integer. These objects are in the value fields of the objects of type Symbol representing those non-terminals on the parse stack. RESULT is of type Integer as well, since the resulting non-terminal expr was declared as of type Integer. This object becomes the value instance variable of a new Symbol object.
For each label, two more variables accessible to the user are declared. A left and right value labels are passed to the code string, so that the user can find out where the left and right side of each terminal or non-terminal is in the input stream. The name of these variables is the label name, plus left or right. for example, given the right hand side of a production expr:e1 PLUS expr:e2 the user could not only access variables e1 and e2, but also e1left, e1right, e2left and e2right. these variables are of type int.
The final step in creating a working parser is to create a scanner (also
known as a lexical analyzer or simply a lexer). This routine is
responsible for reading individual characters, removing things things like
white space and comments, recognizing which terminal symbols from the
grammar each group of characters represents, then returning Symbol objects
representing these symbols to the parser.
The terminals will be retrieved with a call to the
scanner function. In the example, the parser will call
scanner.next_token(). The scanner should return objects of
type java_cup.runtime.Symbol. This type is very different than
older versions of CUP's java_cup.runtime.symbol. These Symbol
objects contains the instance variable value of type Object,
which should be
set by the lexer. This variable refers to the value of that symbol, and
the type of object in value should be of the same type as declared in
the terminal and non terminal declarations. In the
above example, if the lexer wished to pass a NUMBER token, it should
create a Symbol with the value instance variable
filled with an object of type Integer.
The code contained in the init with clause of the specification
will be executed before any tokens are requested. Each token will be
requested using whatever code is found in the scan with clause.
Beyond this, the exact form the scanner takes is up to you; however
note that each call to the scanner function should return a new
instance of
In the next section a more detailed and formal
explanation of all parts of a CUP specification will be given.
Section 3 describes options for running the
CUP system. Section 4 discusses the details
of how to customize a CUP parser, while section 5
discusses the scanner interface added in CUP 0.10j. Section
6 considers error recovery. Finally, Section 7
provides a conclusion.
Symbol
objects corresponding to terminals and non-terminals with no value
have a null value field.java_cup.runtime.Symbol
(or a subclass).
These symbol objects are annotated with parser information and pushed
onto a stack; reusing objects will result in the parser annotations
being scrambled. As of CUP 0.10j, Symbol
reuse should be
detected if it occurs; the parser will throw an Error
telling you to fix your scanner.2. Specification Syntax
Now that we have seen a small example, we present a complete description of all
parts of a CUP specification. A specification has four sections with
a total of eight specific parts (however, most of these are optional).
A specification consists of:
package name;where name name is a Java package identifier, possibly in several parts separated by ".". In general, CUP employs Java lexical conventions. So for example, both styles of Java comments are supported, and identifiers are constructed beginning with a letter, dollar sign ($), or underscore (_), which can then be followed by zero or more letters, numbers, dollar signs, and underscores.
After an optional package declaration, there can be zero or more import declarations. As in a Java program these have the form:
import package_name.class_name;or
import package_name.*;The package declaration indicates what package the sym and parser classes that are generated by the system will be in. Any import declarations that appear in the specification will also appear in the source file for the parser class allowing various names from that package to be used directly in user supplied action code.
action code {: ... :};where {: ... :} is a code string whose contents will be placed directly within the action class class declaration.
After the action code declaration is an optional parser code declaration. This declaration allows methods and variable to be placed directly within the generated parser class. Although this is less common, it can be helpful when customizing the parser &emdash; it is possible for example, to include scanning methods inside the parser and/or override the default error reporting routines. This declaration is very similar to the action code declaration and takes the form:
parser code {: ... :};Again, code from the code string is placed directly into the generated parser class definition.
Next in the specification is the optional init declaration which has the form:
init with {: ... :};This declaration provides code that will be executed by the parser before it asks for the first token. Typically, this is used to initialize the scanner as well as various tables and other data structures that might be needed by semantic actions. In this case, the code given in the code string forms the body of a void method inside the parser class.
The final (optional) user code section of the specification indicates how the parser should ask for the next token from the scanner. This has the form:
scan with {: ... :};As with the init clause, the contents of the code string forms the body of a method in the generated parser. However, in this case the method returns an object of type java_cup.runtime.Symbol. Consequently the code found in the scan with clause should return such a value. See section 5 for information on the default behavior if the
scan with
section is omitted.As of CUP 0.10j the action code, parser code, init code, and scan with sections may appear in any order. They must, however, precede the symbol lists.
terminal classname name1, name2, ...; non terminal classname name1, name2, ...; terminal name1, name2, ...;and
non terminal name1, name2, ...;where classname can be a multiple part name separated with "."s. The classname specified represents the type of the value of that terminal or non-terminal. When accessing these values through labels, the users uses the type declared. the classname can be of any type. If no classname is given, then the terminal or non-terminal holds no value. a label referring to such a symbol with have a null value. As of CUP 0.10j, you may specify non-terminals the declaration "
nonterminal
" (note, no
space) as well as the original "non terminal
" spelling.Names of terminals and non-terminals cannot be CUP reserved words; these include "code", "action", "parser", "terminal", "non", "nonterminal", "init", "scan", "with", "start", "precedence", "left", "right", "nonassoc", "import", and "package".
precedence left terminal[, terminal...]; precedence right terminal[, terminal...]; precedence nonassoc terminal[, terminal...];The comma separated list indicates that those terminals should have the associativity specified at that precedence level and the precedence of that declaration. The order of precedence, from highest to lowest, is bottom to top. Hence, this declares that multiplication and division have higher precedence than addition and subtraction:
precedence left ADD, SUBTRACT; precedence left TIMES, DIVIDE;Precedence resolves shift reduce problems. For example, given the input to the above example parser 3 + 4 * 8, the parser doesn't know whether to reduce 3 + 4 or shift the '*' onto the stack. However, since '*' has a higher precedence than '+', it will be shifted and the multiplication will be performed before the addition.
CUP assigns each one of its terminals a precedence according to these declarations. Any terminals not in this declaration have lowest precedence. CUP also assigns each of its productions a precedence. That precedence is equal to the precedence of the last terminal in that production. If the production has no terminals, then it has lowest precedence. For example, expr ::= expr TIMES expr would have the same precedence as TIMES. When there is a shift/reduce conflict, the parser determines whether the terminal to be shifted has a higher precedence, or if the production to reduce by does. If the terminal has higher precedence, it it shifted, if the production has higher precedence, a reduce is performed. If they have equal precedence, associativity of the terminal determine what happens.
An associativity is assigned to each terminal used in the precedence/associativity declarations. The three associativities are left, right and nonassoc Associativities are also used to resolve shift/reduce conflicts, but only in the case of equal precedences. If the associativity of the terminal that can be shifted is left, then a reduce is performed. This means, if the input is a string of additions, like 3 + 4 + 5 + 6 + 7, the parser will always reduce them from left to right, in this case, starting with 3 + 4. If the associativity of the terminal is right, it is shifted onto the stack. hence, the reductions will take place from right to left. So, if PLUS were declared with associativity of right, the 6 + 7 would be reduced first in the above string. If a terminal is declared as nonassoc, then two consecutive occurrences of equal precedence non-associative terminals generates an error. This is useful for comparison operations. For example, if the input string is 6 == 7 == 8 == 9, the parser should generate an error. If '==' is declared as nonassoc then an error will be generated.
All terminals not used in the precedence/associativity declarations are treated as lowest precedence. If a shift/reduce error results, involving two such terminals, it cannot be resolved, as the above conflicts are, so it will be reported.
start with non-terminal;This indicates which non-terminal is the start or goal non-terminal for parsing. If a start non-terminal is not explicitly declared, then the non-terminal on the left hand side of the first production will be used. At the end of a successful parse, CUP returns an object of type java_cup.runtime.Symbol. This Symbol's value instance variable contains the final reduction result.
The grammar itself follows the optional start declaration. Each production in the grammar has a left hand side non-terminal followed by the symbol "::=", which is then followed by a series of zero or more actions, terminal, or non-terminal symbols, followed by an optional contextual precedence assignment, and terminated with a semicolon (;).
Each symbol on the right hand side can optionally be labeled with a name.
Label names appear after the symbol name separated by a colon (:). Label
names must be unique within the production, and can be used within action
code to refer to the value of the symbol. Along with the label, two
more variables are created, which are the label plus left and
the label plus right. These are int values that
contain the right and left locations of what the terminal or
non-terminal covers in the input file. These values must be properly
initialized in the terminals by the lexer. The left and right values
then propagate to non-terminals to which productions reduce.
If there are several productions for the same non-terminal they may be
declared together. In this case the productions start with the non-terminal
and "::=". This is followed by multiple right hand sides each
separated by a bar (|). The full set of productions is then terminated by a
semicolon.
Actions appear in the right hand side as code strings (e.g., Java code inside
{: ... :} delimiters). These are executed by the parser
at the point when the portion of the production to the left of the
action has been recognized. (Note that the scanner will have returned the
token one past the point of the action since the parser needs this extra
lookahead token for recognition.)
Contextual precedence assignments follow all the symbols and actions of
the right hand side of the production whose precedence it is assigning.
Contextual precedence assignment allows a production to be assigned a
precedence not based on the last terminal in it. A good example is
shown in the above sample parser specification:
In addition to the specification file, CUP's behavior can also be changed
by passing various options to it. Legal options are documented in
precedence left PLUS, MINUS;
precedence left TIMES, DIVIDE, MOD;
precedence left UMINUS;
expr ::= MINUS expr:e
{: RESULT = new Integer(0 - e.intValue()); :}
%prec UMINUS
Here, there production is declared as having the precedence of UMINUS.
Hence, the parser can give the MINUS sign two different precedences,
depending on whether it is a unary minus or a subtraction operation.
3. Running CUP
As mentioned above, CUP is written in Java. To invoke it, one needs
to use the Java interpreter to invoke the static method
java_cup.Main(), passing an array of strings containing options.
Assuming a Unix machine, the simplest way to do this is typically to invoke it
directly from the command line with a command such as:
java java_cup.Main options < inputfile
Once running, CUP expects to find a specification file on standard input
and produces two Java source files as output. Main.java
and include:
interface
rather than as a class
.
This option is typically used to work-around the java bytecode
limitations on table initialization code sizes. However, CUP
0.10h introduced a string-encoding for the parser tables which
is not subject to the standard method-size limitations.
Consequently, use of this option should no longer be required
for large grammars.
java_cup.runtime.Scanner
. By default, the
generated parser refers to this interface, which means you cannot
use these parsers with CUP runtimes older than 0.10j. If your
parser does not use the new scanner integration features, then you
may specify the -noscanner
option to suppress the
java_cup.runtime.Scanner
references and allow
compatibility with old runtimes. Not many people should have reason
to do this.
-version
flag will cause it
to print out the working version of CUP and halt. This allows
automated CUP version checking for Makefiles, install scripts and
other applications which may require it.
4. Customizing the Parser
Each generated parser consists of three generated classes. The
sym class (which can be renamed using the -symbols
option) simply contains a series of int constants,
one for each terminal. Non-terminals are also included if the -nonterms
option is given. The source file for the parser class (which can
be renamed using the -parser option) actually contains two
class definitions, the public parser class that implements the
actual parser, and another non-public class (called CUP$action) which
encapsulates all user actions contained in the grammar, as well as code from
the action code declaration. In addition to user supplied code, this
class contains one method: CUP$do_action which consists of a large
switch statement for selecting and executing various fragments of user
supplied action code. In general, all names beginning with the prefix of
CUP$ are reserved for internal uses by CUP generated code.
The parser class contains the actual generated parser. It is a subclass of java_cup.runtime.lr_parser which implements a general table driven framework for an LR parser. The generated parser class provides a series of tables for use by the general framework. Three tables are provided:
Beyond the parse tables, generated (or inherited) code provides a series of methods that can be used to customize the generated parser. Some of these methods are supplied by code found in part of the specification and can be customized directly in that fashion. The others are provided by the lr_parser base class and can be overridden with new versions (via the parser code declaration) to customize the system. Methods available for customization include:
getScanner().next_token()
.