Class: PGN::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/pgn/lexer.rb

Overview

Lexer is a StringScanner-based tokenizer for PGN. It reuses the same terminal patterns as the legacy whittle parser so tokenization is byte-compatible, but performs far fewer Ruby allocations because the scanning happens in the C-backed StringScanner.

The lexer also records, per game, the byte offset of the game's first non-discarded token (#game_starts). The parser uses these offsets to slice the verbatim Game#pgn raw text out of the original input, reproducing the legacy accumulator's output exactly.

All offsets are byte offsets (StringScanner works in bytes); slicing is done with String#byteslice so multibyte (UTF-8) input stays intact.

Defined Under Namespace

Classes: Token

Constant Summary collapse

WSP =

Discarded: insignificant whitespace.

/\s+/
PGN_COMMENT =

Discarded: a PGN "rest of line" comment beginning with %.

/% .*/
STRING =

A tag value string. Allows unescaped double-quotes inside the value (a form seen in real-world PGN files) — only a bare backslash starts an escape. Matches the legacy parser's relaxed string rule.

/
  "                          # beginning of string
  (
    [[:print:]&&[^\\]] |    # printing characters except backslash
    \\\\                |    # escaped backslashes
    \\"                      # escaped quotation marks
  )*                         # zero or more of the above
  "                          # end of string
/x
COMMENT =

A brace-delimited comment, with recursive nesting via \g<1>.

/
  (
    \{                           # beginning of comment
    (
      [[:print:]&&[^\\{}]] |   # printing characters except brace and backslash
      \n                     |
      \\\\                   |   # escaped backslashes
      \\\{|\\\}              |   # escaped braces
      \n                     |   # newlines
      \g<1>                      # recursive
    )*                           # zero or more of the above
      \}                           # end of comment
  )
/x
GAME_TERMINATION =

Game termination marker.

%r{
  1-0       |    # white wins
  0-1       |    # black wins
  1/2-1/2 |    # draw
  \*             # ?
}x
SAN_MOVE =

A move in standard algebraic notation (incl. castling, promotion, check/mate, the -- "don't care" move).

%r{
  (
    --                           |    # "don't care" move (used in variations)
    [O0](-[O0]){1,2}             |    # castling (O-O, O-O-O)
    [a-h][1-8]                   |    # pawn moves (e4, d7)
    [BKNQR][a-h1-8]?x?[a-h][1-8] |    # major piece moves w/ optional specifier
    [a-h][1-8]?x[a-h][1-8]            # pawn captures
  )
  (
    =[BNQR]                            # optional promotion (d8=Q)
  )?
  (
    \+                            |    # check (g5+)
    \#                                 # checkmate (Qe7#)
  )?
}x
MOVE_NUMBER =

A move number indication, e.g. 1., 12., 1....

/[[:digit:]]+\.*/
TAG_NAME =

A tag name (letters, digits, underscores).

/[A-Za-z0-9_]+/
NAG =

A numeric annotation glyph ($1) or a punctuation annotation (?!, !?, ??, ...).

/
  \$\d+       | # dollar sign followed by an integer
  [?!][?!]?   # support the most used annotations directly
/x
RULES =

Order matters: more specific / longer tokens are tried first so that e.g. 1-0 (termination) wins over 1 (move number), and 0-0 (castling) wins over 0 (move number). Whitespace and % comments are discarded (consumed but not emitted). Beyond that constraint, rules are ordered most- to least-frequent (one san_move/move_number per ply/full-move vs. a handful of comments/strings per game) so the common case fails the fewest regexes before matching.

[
  [:wsp,              WSP,              true],   # discarded
  [:pgn_comment,      PGN_COMMENT,      true],   # discarded
  [:game_termination, GAME_TERMINATION, false],
  [:san_move,         SAN_MOVE,         false],
  [:move_number,      MOVE_NUMBER,      false],
  [:nag,              NAG,              false],
  [:comment,          COMMENT,          false],
  [:string,           STRING,           false],
  [:tag_name,         TAG_NAME,         false]
].freeze.each(&:freeze)
LITERAL_BYTES =

Single-character literals, matched by their byte value: [type, frozen value].

{
  91 => [:lbracket, '['], # [
  93 => [:rbracket, ']'], # ]
  40 => [:lparen,   '('], # (
  41 => [:rparen,   ')'] # )
}.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(input) ⇒ Lexer

Returns a new instance of Lexer.



128
129
130
131
132
133
134
 
# File 'lib/pgn/lexer.rb', line 128

def initialize(input)
  @input = input
  @ss = StringScanner.new(input)
  @line = 1
  @game_starts = []
  @between_games = true # at start we are "between" games
end

Instance Attribute Details

#game_starts ⇒ Object (readonly)

Returns the value of attribute game_starts.



136
137
138
 
# File 'lib/pgn/lexer.rb', line 136

def game_starts
  @game_starts
end

Instance Method Details

#next_token ⇒ Object

Returns the next Token, or nil at end of input.



148
149
150
151
152
153
 
# File 'lib/pgn/lexer.rb', line 148

def next_token
  type, value = next_token_pair
  return nil unless type

  Token.new(type: type, value: value, offset: @last_offset, line: @line)
end

#next_token_pair ⇒ Object

Fast path for the parser: returns [type, value] for the next non-discarded token, or nil at end of input. Does not allocate a Token Struct, and scan_one returns the matched string directly (stashing its type/discarded flag in ivars) so the only array allocated per token is the [type, value] pair Racc requires.



160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/pgn/lexer.rb', line 160

def next_token_pair
  until @ss.eos?
    off = @ss.pos

    if (lit = LITERAL_BYTES[@input.getbyte(off)])
      type, value = lit
      @ss.pos = off + 1
      note_token(type, off)
      @last_offset = off
      return [type, value]
    end

    value = scan_one
    advance_line(value)
    next if @scan_discarded

    note_token(@scan_type, off)
    @last_offset = off
    return [@scan_type, value]
  end
  nil
end

#tokens ⇒ Object

The list of Tokens for the whole input. Convenience for specs.



139
140
141
142
143
144
145
 
# File 'lib/pgn/lexer.rb', line 139

def tokens
  result = []
  while (t = next_token)
    result << t
  end
  result
end