Class: Rubycc::Front::LexemeReader

Inherits:
Object
  • Object
show all
Defined in:
lib/rubycc/front/lexeme_reader.rb

Overview

The shared spelling-to-value decoder for the C subset's lexemes. Both the streaming Front::Lexer and the preprocessor's token converter delegate here so the two paths fold numeric constants, resolve escapes and classify identifiers with byte-for-byte identical rules. The reader walks a plain string with its own cursor (no line/column bookkeeping); positions are the caller's concern. Each read_* leaves pos just past the lexeme.

Defined Under Namespace

Classes: Result

Constant Summary collapse

KEYWORDS = 
%w[int char void short long signed unsigned _Bool float double struct union
enum typedef static extern const volatile inline _Noreturn register auto
return if else while do for break continue
switch case default goto sizeof
_Static_assert _Alignof _Alignas _Atomic __int128
__builtin_va_start __builtin_va_arg __builtin_va_end __builtin_va_copy
__builtin_expect __builtin_alloca __builtin_offsetof
__builtin_constant_p __builtin_choose_expr
__builtin_ctz __builtin_ctzll __builtin_clz __builtin_clzll
__builtin_unreachable __builtin_memcpy
__builtin_add_overflow __builtin_sub_overflow __builtin_mul_overflow
__atomic_load_n __atomic_store_n __atomic_exchange_n
__atomic_compare_exchange_n
__atomic_fetch_add __atomic_fetch_sub
__atomic_add_fetch __atomic_sub_fetch __atomic_or_fetch
__atomic_thread_fence
__sync_fetch_and_add __sync_fetch_and_sub
__sync_add_and_fetch __sync_sub_and_fetch __sync_or_and_fetch
__sync_lock_test_and_set __sync_lock_release __sync_synchronize
__sync_bool_compare_and_swap __sync_val_compare_and_swap
__asm__
__attribute__ __extension__].freeze
KEYWORD_SET =

Membership lookup for every identifier the lexer produces, so it must be O(1); KEYWORDS.include? showed up as a linear scan in profiling.

KEYWORDS.to_h { |word| [word, true] }.freeze
KEYWORD_ALIASES =

gcc's reserved "__x"/"x" alternate spellings for a handful of keywords (6.10.8.4's rationale: a header built with strict-ISO options such as -ansi still needs the keyword's meaning without colliding with a user identifier of the plain name). glibc's uapi-derived headers lean on these unconditionally, e.g. asm-generic/int-ll64.h's "typedef signed char __s8". Each maps straight to the plain keyword's own spelling, so every downstream check keyed on that spelling (DECL_SPECIFIER_KEYWORDS, "const"/"volatile"/"inline" in Front::Parser, ...) sees an ordinary keyword token and needs no separate case for the alias.

{
  "__signed" => "signed", "__signed__" => "signed",
  "__const" => "const", "__const__" => "const",
  "__volatile" => "volatile", "__volatile__" => "volatile",
  "__inline" => "inline", "__inline__" => "inline"
}.freeze
ESCAPES =

The eleven simple escape sequences of 6.4.4.4p1, shared by character constants and string literals, mapping the character after the backslash to the byte value it denotes. "?" is the one whose escaped and plain spellings mean the same byte ('?'): it exists only so a source line can avoid accidentally spelling a trigraph, and gcc accepts it in both constructs. "\x" (hexadecimal, any number of digits) and octal ("\ooo", 1-3 digits, "0" included) are handled separately in #read_escaped_byte since their value comes from digits rather than a fixed table lookup.

{
  "n" => 10, "t" => 9, "r" => 13, "\\" => 92,
  "'" => 39, "\"" => 34, "?" => 63,
  "a" => 7, "b" => 8, "f" => 12, "v" => 11
}.freeze
PUNCTUATORS_3 =

Three-character punctuators, matched before the shorter lists so the longest one always wins: "<<=" must beat "<<" (and "<="/"<"), ">>=" must beat ">>" (and ">="/">"), and "..." (the variadic-parameter ellipsis) must beat a lone "." (which is not a two-character punctuator, so ".." never forms; three consecutive dots are the only way "..." arises).

%w[<<= >>= ...].freeze
PUNCTUATORS_2 =

Two-character punctuators, matched before the single-character list so the lexer always prefers the longest punctuator ("==" over two "=", "&&" over two "&", "++" or "+=" over a lone "+", "->" over a lone "-", "<<" over two "<", "&=" over "&"/"&&").

%w[== != <= >= && || += -= *= /= %= ++ -- -> << >> &= |= ^=].freeze
PUNCTUATORS_1 =

Single-character punctuators used by this slice. "&" is both the address-of operator and the bitwise-and operator, "*" doubles as dereference and pointer-declarator marker, "|" "^" "~" are the remaining bitwise operators, "[" "]" bracket array declarators and subscripts, "?" ":" form the conditional operator, and "." selects a struct member.

%w[+ - * / % ( ) { } ; = , < > ! & | ^ ~ [ ] ? : .].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(text, start = 0) ⇒ LexemeReader

Returns a new instance of LexemeReader.



129
130
131
132
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 129

def initialize(text, start = 0)
  @text = text
  @pos = start
end

Instance Attribute Details

#pos ⇒ Object (readonly)

Returns the value of attribute pos.



127
128
129
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 127

def pos
  @pos
end

Class Method Details

.keyword?(name) ⇒ Boolean

Whether name (an identifier's spelling) is a reserved keyword.

Returns:

  • (Boolean) 


113
114
115
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 113

def self.keyword?(name)
  KEYWORD_SET.key?(name)
end

.keyword_spelling(name) ⇒ Object

The keyword token name denotes: itself when it already is one of KEYWORDS, or the plain spelling it aliases (see KEYWORD_ALIASES) when it is one of gcc's reserved "__x"/"x" spellings. nil when name names neither, so the caller lexes it as an ordinary identifier.



121
122
123
124
125
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 121

def self.keyword_spelling(name)
  return name if KEYWORD_SET.key?(name)

  KEYWORD_ALIASES[name]
end

Instance Method Details

#read_char ⇒ Object

A character constant 'c' or '\n' including its quotes: returns a :num result whose value is the character's byte code, since an ISO C character constant has type int (6.4.4.4). The constant must hold exactly one character; an empty '', a multi-character 'ab', an unterminated ' and an unknown escape are all rejected.



186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 186

def read_char
  start = @pos
  advance # opening quote
  if at_end? || current == "\n"
    raise LexError.new("unterminated character constant", start)
  elsif current == "'"
    raise LexError.new("empty character constant", start)
  end
  value = read_escaped_byte(start, "character constant")
  if at_end? || current == "\n"
    raise LexError.new("unterminated character constant", start)
  elsif current != "'"
    raise LexError.new("multi-character character constant", start)
  end
  advance # closing quote
  Result.new(:num, value, 10, "")
end

#read_identifier ⇒ Object

An identifier or keyword run "[A-Za-z_][A-Za-z0-9_]*". The caller must position the cursor on the leading identifier character.



136
137
138
139
140
141
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 136

def read_identifier
  name = +""
  name << advance while identifier_char?(current)
  spelling = LexemeReader.keyword_spelling(name)
  Result.new(spelling ? :keyword : :ident, spelling || name, nil, nil)
end

#read_number ⇒ Object

A numeric constant. A "0x"/"0X" prefix is hexadecimal (an integer, or a hexadecimal floating constant this subset does not lower yet); otherwise a "." or an exponent anywhere in the run marks a decimal floating constant, and its absence a decimal or octal integer. The floating check precedes the octal one so "08.5" reads as the float 8.5 rather than tripping the octal-digit rule on its '8'.



149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 149

def read_number
  if current == "0" && (peek == "x" || peek == "X")
    read_hexadecimal_constant
  elsif current == "0" && (peek == "b" || peek == "B")
    read_binary_constant
  elsif floating_constant_ahead?
    read_floating_constant
  else
    read_integer_constant
  end
end

#read_string ⇒ Object

A string literal "abc" including its surrounding quotes: returns the escape-resolved bytes (ASCII-8BIT), without the NUL terminator the generator later appends. An unterminated literal (newline or end before the closing quote) and an unknown escape are rejected.



165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/rubycc/front/lexeme_reader.rb', line 165

def read_string
  start = @pos
  advance # opening quote
  bytes = +"".b
  loop do
    if at_end? || current == "\n"
      raise LexError.new("unterminated string literal", start)
    end
    break if current == "\""

    bytes << read_escaped_byte(start, "string literal")
  end
  advance # closing quote
  Result.new(:string, bytes, nil, nil)
end