Class: Rubycc::Front::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/rubycc/front/lexer.rb

Overview

Hand-written lexer for the C subset. Tracks 1-based line/column positions and keeps each source line around so tokens (and errors) can be reported with source excerpts. Handles // and /* */ comments and whitespace. The spelling-to-value decoding of numbers, string/character literals and identifiers is delegated to the shared LexemeReader, which the preprocessor reuses so both token sources agree exactly.

Instance Method Summary collapse

Constructor Details

#initialize(source, filename:) ⇒ Lexer

Returns a new instance of Lexer.



16
17
18
19
20
21
22
23
24
 
# File 'lib/rubycc/front/lexer.rb', line 16

def initialize(source, filename:)
  @src = source
  @filename = filename
  @pos = 0
  @line = 1
  @column = 1
  # -1 keeps a trailing empty field, so line numbers map 1:1 to entries.
  @lines = source.split("\n", -1)
end

Instance Method Details

#tokenize ⇒ Object 



26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/rubycc/front/lexer.rb', line 26

def tokenize
  tokens = []
  loop do
    skip_whitespace_and_comments
    break if at_end?

    tokens << next_token
  end
  tokens << make_token(:eof, nil, @line, @column)
  tokens
end