CSC%203315%20Lexical%20and%20Syntax%20Analysis

1
CSC 3315Lexical and Syntax Analysis
Hamid Harroud School of Science and Engineering,
Akhawayn University http//www.aui.ma/H.Harroud/c
sc3315/
2
Lexical Analysis

• Convert source file characters into token stream.
• Remove content-free characters (comments,
  whitespace, ...)
• Detect lexical errors (badly-formed literals,
  illegal characters, ...)
• Output of lexical analysis is input to syntax
  analysis.
• Idea Look for patterns in input character
  sequence, convert to tokens with attributes, and
  pass them to parser in stream.

3
Lexical Analysis Example
4
Specifying Lexical Analysers

• Can define lexical analyzer via list of pairs
• (regular expression, action)
• where regular expression describes token pattern
• and action is a piece of code, parameterized by
  the matching lexeme, that returns a (token,
  attribute) pair
• Example
• (digit, return new Token(NUM,parseInt(lexeme))
  )
• (alpha(alphadigit), return new
  Token(ID,lexeme))
• (spacetabnewline, )
• (.,.)
• So R.E