3.5

Input Elements and Tokens

LEXICAL STRUCTURE

numbers produced by a Java compiler or other Java system component. It also

specifies the termination of the

//

 form of a comment ( 3.7).

LineTerminator:

the ASCII

LF

 character, also known as  newline 

the ASCII

CR

 character, also known as  return 

the ASCII

CR

 character followed by the ASCII

LF

 character

InputCharacter:

UnicodeInputCharacter

 but not

CR

 or

LF

Lines are terminated by the ASCII characters

CR

, or

LF

, or

CR LF

. The two

characters

CR

 immediately followed by

LF

 are counted as one line terminator, not

two. The result is a sequence of line terminators and input characters, which are

the terminal symbols for the third step in the tokenization process.

3.5   Input Elements and Tokens

The input characters and line terminators that result from escape processing ( 3.3)

and then input line recognition ( 3.4) are reduced to a sequence of

input elements

.

Those input elements that are not white space ( 3.6) or comments ( 3.7) are

tokens

. The tokens are the terminal symbols of the Java syntactic grammar ( 2.3).

This process is specified by the following productions:

Input:

InputElements

opt

Sub

opt

InputElements:

InputElement

InputElements InputElement

InputElement:

WhiteSpace

Comment

Token

Token:

Identifier

Keyword

Literal

Separator

Operator

14