Class: Rubycc::Preprocess::TokenConverter

Inherits:
Object
  • Object
show all
Defined in:
lib/rubycc/preprocess/token_converter.rb

Overview

Ends translation phase 4 by turning preprocessing tokens into the Front::Token stream the parser consumes. Numbers are folded, string and character literals decoded, and identifiers classified as keywords, all via the shared LexemeReader so the result matches the streaming lexer exactly. Newlines are dropped here; a "#"/"##" that survives to this point is a stray operator with no macro meaning (directive lines are diagnosed earlier), so it is rejected.

This is where translation phases 5-7 sit in this pipeline, so adjacent string-literal concatenation (phase 6, ISO C 6.4.5p5) happens here: a run of neighbouring string tokens folds into one, its bytes the per-literal decodings laid end to end. The streaming Front::Lexer takes the direct path and does not concatenate — nothing but this converter ever feeds the parser in the real compile pipeline (which always runs through the preprocessor), so the lexer's simpler single-literal path serves its unit-test-only role without the phase-6 fold.

Instance Method Summary collapse

Instance Method Details

#convert(pp_tokens) ⇒ Object 



27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/rubycc/preprocess/token_converter.rb', line 27

def convert(pp_tokens)
  tokens = []
  index = 0
  while index < pp_tokens.length
    pp = pp_tokens[index]
    case pp.type
    when :newline
      index += 1
    when :eof
      tokens << front_token(pp, :eof, nil)
      index += 1
    when :identifier
      spelling = Front::LexemeReader.keyword_spelling(pp.text)
      tokens << front_token(pp, spelling ? :keyword : :ident, spelling || pp.text)
      index += 1
    when :pp_number
      tokens << convert_number(pp)
      index += 1
    when :string
      value, index = concatenate_strings(pp_tokens, index)
      tokens << front_token(pp, :string, value)
    when :char
      tokens << num_token(pp, decode_char(pp), 10, "")
      index += 1
    when :punct
      if pp.text == "#" || pp.text == "##"
        raise_at(pp, "stray '#' in program")
      end
      tokens << front_token(pp, :punct, pp.text)
      index += 1
    when :other
      raise_at(pp, "unexpected character #{pp.text.inspect}")
    end
  end
  tokens
end