CS 152 - Lecture 7

Cay S. Horstmann

Language Translation Process

• Input: Source code
• Lexical Analysis: Group input characters into reserved words, identifiers, constants, operators, comments
• Parsing: Build data representation of program, report syntax errors
• Code generation: Use data representation to generate, optimize code
• Output: Machine or VM code
• Interpreter directly executes the parsed program

Lexical Analysis

• Remove white space
• Remove comments
• Break input into tokens
• if ( radix >= 0.0) y = sqrt(radix)
• Each token has type (reserved word, identifier, constant, operator, etc.) and text
• Use regular expressions to define token types (see below)
• Process header files (C/C++)

Notes on White Space

• Free form language: All white space is optional
• Line-oriented language: Line endings are significant. Ex.: Fortran
• Scala is in the middle: Newline is a token and can be equivalent to a ;
• Python: Indentation indicates block structure

  if radix >= 0:
      y = sqrt(radix)
      print(y)
  else:
      print('error')

Parsing

• Translates stream of tokens into data structure
• Parse tree for expression 3 + 4 * x

  

• Higher-level features have more complex structure

  Ex. function has

  • name
  • return type
  • list of parameters
  • body

  Body has

  • list of expressions (Scala)/statements (Java)

Code Generation

• Parse tree translated into machine or VM instructions
• Example:

  

  movl $4, %eax
  mull x, %eax
  addl $3, %eax

Code Optimization

• Register allocation

  result = (a + b) * (c + d)

  movl a, %eax
  addl b, %eax
  
  movl c, %ebx
  addl b, %ebx
  
  mull %ebx, %eax
  movl %eax, result

• Loop optimization

  for (i = 0; i < n; i++) {
     t = u * v;    
     a[i] = t + i;
  }

• Inline functions

  int max(int x, int y) { return x > y ? x : y; }

  Translate max(x, 0) into x > 0 ? x : 0, avoid cost of function call

• Many others

Regular Expressions

• Describe “regular” sets of strings
• Symbols other than ( ) | * stand for themselves
• Concatenation α β: First part matches α, second part β
• α | β = Match α or β
• α* = 0 or more matches of α
• Use () for grouping
• Example: E(0|1|2|3|4|5|6|7|8|9)*

  An E followed by a (possibly empty) sequence of digits

  E123
  E9
  E

Regular Expressions

Convenience Notation

• α+ = one or more (i.e. αα*)
• α? = 0 or 1 (i.e. (|α))
• [xyz] = x|y|z
• [x-y] = all characters from x to y, e.g. [0-9] = all ASCII digits
• [^x-y] = all characters other than [x-y]
• \p{Name}, where Name is a Unicode category (ex. L, N, Z for letter, number, space)
• \P{Name}: complement of \p{Name}
• . matches any character
• \ is an escape. For example, \. is a period, \\ a backslash

Regex Examples

• Integers: [+-]?[0-9]+, or maybe [+-]?\p{N}+
  • Note: + loses its normal meaning inside [], and a - just before ] denotes itself
• Hexadecimal numbers 0[Xx][0-9A-Fa-f]+
• Quoted Java strings: ".*"
• Well, actually not; the . will match a quote. Better: "[^"]*"
• Well, actually not; you can have a \" in a quoted string. "([^"\\]|\\.)*"
• Are we done yet? What about \x1B?

The Scala Regex Library

• Find all matches

  import scala.util.matching._
  val regex = "[0-9]+".r
  regex.findAllIn("99 bottles, 98 bottles").toList
  List[String] = List(99, 98)

  Check whether beginning matches

  regex.findPrefixOf("99 bottles, 98 bottles").getOrElse(null)
  String = 99

• Groups

  val regex = "([0-9]+) bottles".r
  val matches = regex.findAllIn("99 bottles, 98 bottles, 97 cans").matchData.toList 
  matches : List[Regex.Match] = List(99 bottles, 98 bottles)
  matches.map(_.group(1))
  List[String] = List(99, 98)

Lab

• You work with a buddy
• One of you writes the code (coder), the other types up answers (scribe)
• When you get stuck, ask your buddy first!
• Switch roles each lab. The previous scribe is the coder for this lab.
• The scribe submits lab work in lab6/report.txt inside the Git repo. Include the coder's name in the report!
• If the coder wants to submit something, that's cool too. Do whatever is easy.

Step 1: Regular Expressions

Write down regular expressions for

• Java floating-point literals
• Java character constants
• C-style comments, i.e. /* ... */

Step 2: Check Your Regexes

Use the Scala Regex.findAllIn method to check your work from Step 1 with the strings

• "3.14 -3.14 +3.14 3. 3 .3 3.14E2 3.14E+2 3.14E-2 3.14e100"

• "'a' '\\n' '\\\\' '\\'' '\\x1b' '\\033'"

• "/* foo */ /** foo */ /**/**/**/ /*/*/*/"

What results do you get?

Tip: In Java, it is unpleasant to deal with strings containing backslashes and quotes. For example, the regular expression ([^"\\]|\\.)*, as a Java string, is "([^\"\\\\]|\\\\.)*" Scala has an alternate way of specifying strings. When strings are enclosed in """...""", nothing inside is escaped. (Of course, you can't have a """ inside.) For example, """([^"\\]|\\.)*""".

Step 3: A Simple Lexer

In a lexer, we specify a set of patterns that are tried in sequence. For example, a simple language might have the following token types:

• Reserved words: if|def|val
• Identifiers: \p{L}(\p{L}|\p{N}|_)*
• Literals: [+-]?\p{N}+
• Operators: [+*/%<=>-]
• Syntactical marks: [(){};]
• White space: \p{Z}+

Note that the order matters. We want if, def, val recognized as reserved words, not identifiers.

In this step, write a function firstMatch(input : String, patterns : List[Regex]): String that returns the first match in the input string for any of the regular expressions or null if there is no match. For example,

val patterns = List("if|def|val".r, """\p{L}(\p{L}|\p{N}|_)*""".r,
   """[+-]?\p{N}+""".r, "[+*/%<=>-]".r, "[(){};]".r, """\p{Z}+""".r
)
val input = "if(x<0) 0 else root(x);"
firstMatch(input, patterns)
String : if
firstMatch(input.substring(2), patterns)
String : (

What is the code for your function?

Hint: This is simple recursion. If the first regex matches, return the match, otherwise call firstMatch(input, patterns.tail).

Step 4: Complete the Lexer

Write a function tokens(input : String, patterns : List[Regex]) : List[String] that returns a list of matching tokens. For example,

tokens(input, patterns)
List[String] = List(if, (, x, <, 0, ), , 0, , else, , root, (, x, ), ;, )

That's again simple recursion. If the input is empty, return the empty list. Otherwise, get the first match. If it's null, return the empty list. Otherwise, recursively call tokens(input.substring(first.length), patterns).

What is the code of your function?