Class: Rubycc::Preprocess::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/rubycc/preprocess/scanner.rb

Overview

Turns raw source into a stream of preprocessing tokens, carrying out translation phases 2 and 3: backslash-newline line splicing and the replacement of comments by whitespace. Newlines survive as explicit tokens so the directive layer (added later) stays line-oriented, and every token keeps the physical location of its first character.

Splicing works by deleting every backslash-newline pair up front and remembering where each deletion happened (@splice_points): tokens are then matched over the spliced text with StringScanner-driven regexps — one match per token instead of a method call per character, which is what makes the preprocessor's dominant cost scale. A token's physical line/column stays truthful because the recorded splice points replay the deleted line breaks: everything at or past a splice point is located on the following physical line (see #sync). Positions are byte offsets (StringScanner's native unit); columns are converted back to character counts only when a token is actually made, and only for non-ASCII source.

Constant Summary collapse

PP_PUNCTUATORS_3 =

Two- and one-character punctuators gain the preprocessor-only "##" and "#" over the shared punctuator tables; the three-character set is unchanged.

Front::LexemeReader::PUNCTUATORS_3
PP_PUNCTUATORS_2 = 
(Front::LexemeReader::PUNCTUATORS_2 + ["##"]).freeze
PP_PUNCTUATORS_1 = 
(Front::LexemeReader::PUNCTUATORS_1 + ["#"]).freeze
PUNCTUATOR_RE =

Longest alternative first, so one anchored attempt is a longest match.

Regexp.union(PP_PUNCTUATORS_3 + PP_PUNCTUATORS_2 + PP_PUNCTUATORS_1)
HORIZONTAL_WS_RE = 
/[ \t\r\v\f]+/
LINE_COMMENT_RE =

A // comment runs to (but not over) the end of the physical line; a trailing backslash-newline was already spliced away, so the comment continues onto the next line just as it does under gcc.

%r{//[^\n]*}
BLOCK_OPEN_RE = 
%r{/\*}
BLOCK_CLOSE_RE = 
%r{\*/}
NEWLINE_RE = 
/\n/
IDENTIFIER_RE = 
/[A-Za-z_][A-Za-z0-9_]*/
PP_NUMBER_RE =

A preprocessing number (6.4.8): a digit, or a "." then a digit, and then digits, identifier characters, ".", and a sign immediately after an 'e'/'E'/'p'/'P'. Whether the run is a valid C constant is decided later, at conversion time.

/(?:[0-9]|\.[0-9])(?:[eEpP][+-]|[0-9A-Za-z_.])*/
STRING_RE =

A string literal or character constant, kept verbatim (quotes and escapes included). Only the boundary is recognized here, matching the streaming lexer: a backslash shields the following character so an escaped quote does not close the literal. An unterminated literal keeps whatever was read (the trailing ("|\)? picks up a lone backslash cut off by end-of-line or end-of-file); the converter re-scans it and raises the positioned error.

/"(?:[^"\\\n]|\\[^\n])*(?:"|\\)?/
CHAR_RE = 
/'(?:[^'\\\n]|\\[^\n])*(?:'|\\)?/
WIDE_CHAR_RE =

A wide character constant L'c' or wide string literal L"..." (6.4.4.4, 6.4.5): recognized only when the "L" abuts the quote, so an identifier named "L" is unaffected. The "L" is kept in the spelling so the converter can tell a wide literal from a plain one; it decides a wide character's value (int here, as for a plain constant) and rejects a wide string. The u/U/u8 prefixes are out of scope, so they still scan as an identifier.

/L'(?:[^'\\\n]|\\[^\n])*(?:'|\\)?/
WIDE_STRING_RE = 
/L"(?:[^"\\\n]|\\[^\n])*(?:"|\\)?/

Instance Method Summary collapse

Constructor Details

#initialize(source, filename:) ⇒ Scanner

Returns a new instance of Scanner.



71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/rubycc/preprocess/scanner.rb', line 71

def initialize(source, filename:)
  @filename = filename
  # -1 keeps a trailing empty field so line numbers map 1:1 to entries.
  @lines = source.split("\n", -1)
  splice(source)
  @scanner = StringScanner.new(@spliced)
  @ascii_only = @spliced.ascii_only?
  @line = 1
  # Byte offset in @spliced where the current physical line starts; a
  # column is the distance from here (plus one). @column_pos/@column_chars
  # are the incremental cursor #column_at advances (see there); they are
  # only meaningful for non-ASCII source, and reset_line_start resets all
  # three together so they can never drift apart.
  reset_line_start(0)
  # Index into @splice_points of the first point not yet replayed.
  @next_splice = 0
end

Instance Method Details

#scan ⇒ Object 



89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
 
# File 'lib/rubycc/preprocess/scanner.rb', line 89

def scan
  tokens = []
  ss = @scanner
  loop do
    before = ss.pos
    skip_whitespace_and_comments(ss, tokens)
    # Whether any whitespace or comment separated this token from the
    # previous one on the same logical line. The directive layer needs it
    # to tell "#define F(x)" (function-like) from "#define F (x)" (an
    # object macro whose replacement begins with a parenthesis).
    space_before = ss.pos != before
    break if ss.eos?

    start = ss.pos
    sync(start)
    if ss.skip(NEWLINE_RE)
      tokens << make_token(:newline, "\n", @line, column_at(start), space_before)
      @line += 1
      reset_line_start(ss.pos)
    else
      tokens << scan_token(ss, @line, column_at(start), space_before)
    end
  end
  sync(@spliced.bytesize)
  tokens << make_token(:eof, nil, @line, column_at(@spliced.bytesize))
  tokens
end