Class: PGN::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/pgn/lexer.rb

Overview

Lexer is a StringScanner-based tokenizer for PGN. It reuses the same terminal patterns as the legacy whittle parser so tokenization is byte-compatible, but performs far fewer Ruby allocations because the scanning happens in the C-backed StringScanner.

The lexer also records, per game, the byte offset of the game's first non-discarded token (#game_starts). The parser uses these offsets to slice the verbatim Game#pgn raw text out of the original input, reproducing the legacy accumulator's output exactly.

All offsets are byte offsets (StringScanner works in bytes); slicing is done with String#byteslice so multibyte (UTF-8) input stays intact.

Defined Under Namespace

Classes: Token

Constant Summary collapse

WSP =

Discarded: insignificant whitespace.

/\s+/.freeze
PGN_COMMENT =

Discarded: a PGN "rest of line" comment beginning with %.

/% .*/.freeze
STRING =

A tag value string. Allows unescaped double-quotes inside the value (a form seen in real-world PGN files) — only a bare backslash starts an escape. Matches the legacy parser's relaxed string rule.

/
  "                          # beginning of string
  (
    [[:print:]&&[^\\]] |    # printing characters except backslash
    \\\\                |    # escaped backslashes
    \\"                      # escaped quotation marks
  )*                         # zero or more of the above
  "                          # end of string
/x.freeze
COMMENT =

A brace-delimited comment, with recursive nesting via \g<1>.

/
  (
    \{                           # beginning of comment
    (
      [[:print:]&&[^\\\{\}]] |   # printing characters except brace and backslash
      \n                     |
      \\\\                   |   # escaped backslashes
      \\\{|\\\}              |   # escaped braces
      \n                     |   # newlines
      \g<1>                      # recursive
    )*                           # zero or more of the above
      \}                           # end of comment
  )
/x.freeze
GAME_TERMINATION =

Game termination marker.

%r{
  1-0       |    # white wins
  0-1       |    # black wins
  1\/2-1\/2 |    # draw
  \*             # ?
}x.freeze
SAN_MOVE =

A move in standard algebraic notation (incl. castling, promotion, check/mate, the -- "don't care" move).

%r{
  (
    --                           |    # "don't care" move (used in variations)
    [O0](-[O0]){1,2}             |    # castling (O-O, O-O-O)
    [a-h][1-8]                   |    # pawn moves (e4, d7)
    [BKNQR][a-h1-8]?x?[a-h][1-8] |    # major piece moves w/ optional specifier
    [a-h][1-8]?x[a-h][1-8]            # pawn captures
  )
  (
    =[BNQR]                            # optional promotion (d8=Q)
  )?
  (
    \+                            |    # check (g5+)
    \#                                 # checkmate (Qe7#)
  )?
}x.freeze
MOVE_NUMBER =

A move number indication, e.g. 1., 12., 1....

/[[:digit:]]+\.*/.freeze
TAG_NAME =

A tag name (letters, digits, underscores).

/[A-Za-z0-9_]+/.freeze
NAG =

A numeric annotation glyph ($1) or a punctuation annotation (?!, !?, ??, ...).

/
  \$\d+       | # dollar sign followed by an integer
  [\?!][\?!]?   # support the most used annotations directly
/x.freeze
RULES =

Order matters: more specific / longer tokens are tried first so that e.g. 1-0 (termination) wins over 1 (move number), and 0-0 (castling) wins over 0 (move number). Whitespace and % comments are discarded (consumed but not emitted).

[
  [:wsp,              WSP,              true],   # discarded
  [:pgn_comment,      PGN_COMMENT,      true],   # discarded
  [:comment,          COMMENT,          false],
  [:string,           STRING,           false],
  [:game_termination, GAME_TERMINATION, false],
  [:san_move,         SAN_MOVE,         false],
  [:nag,              NAG,              false],
  [:move_number,      MOVE_NUMBER,      false],
  [:tag_name,         TAG_NAME,         false],
].freeze.each(&:freeze)
LITERAL_BYTES =

Single-character literals, matched by their byte value.

{
  91 => :lbracket,   # [
  93 => :rbracket,   # ]
  40 => :lparen,     # (
  41 => :rparen,     # )
}.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(input) ⇒ Lexer

Returns a new instance of Lexer.



125
126
127
128
129
130
131
 
# File 'lib/pgn/lexer.rb', line 125

def initialize(input)
  @input = input
  @ss = StringScanner.new(input)
  @line = 1
  @game_starts = []
  @between_games = true   # at start we are "between" games
end

Instance Attribute Details

#game_starts ⇒ Object (readonly)

Returns the value of attribute game_starts.



133
134
135
 
# File 'lib/pgn/lexer.rb', line 133

def game_starts
  @game_starts
end

Instance Method Details

#next_token ⇒ Object

Returns the next Token, or nil at end of input.



145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/pgn/lexer.rb', line 145

def next_token
  until @ss.eos?
    off = @ss.pos

    if (lit = LITERAL_BYTES[@input.getbyte(off)])
      @ss.pos = off + 1
      note_token(lit, off)
      return Token.new(type: lit, value: @input.byteslice(off, 1),
                       offset: off, line: @line)
    end

    type, value = scan_one
    advance_line(value)
    if type == :wsp || type == :pgn_comment
      # discarded: keep looping without emitting
      next
    end
    note_token(type, off)
    return Token.new(type: type, value: value, offset: off, line: @line)
  end
  nil
end

#tokens ⇒ Object

The list of Tokens for the whole input. Convenience for specs.



136
137
138
139
140
141
142
 
# File 'lib/pgn/lexer.rb', line 136

def tokens
  result = []
  while (t = next_token)
    result << t
  end
  result
end