Class: Markbridge::Parsers::BBCode::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/markbridge/parsers/bbcode/scanner.rb

Overview

High-performance BBCode scanner Tokenizes BBCode in O(n) time with minimal allocations and bounded backtracking

The scanner works entirely in byte offsets (byteindex/byteslice/ getbyte). CRuby has no character-index cache, so character-index operations like str[pos] walk the string from the start on any input containing a multibyte character — O(pos) per call and superlinear per document. Byte offsets keep multibyte input at near-ASCII cost. Invariant: @current_pos always sits on a character boundary — jumps land on matches of ASCII-only patterns and advances step over ASCII bytes or whole matches. Token positions are therefore byte offsets into the input.

Instance Method Summary collapse

Constructor Details

#initialize(input) ⇒ Scanner

Returns a new instance of Scanner.



19
20
21
22
23
 
# File 'lib/markbridge/parsers/bbcode/scanner.rb', line 19

def initialize(input)
  @input = input
  @length = input.bytesize
  @current_pos = 0
end

Instance Method Details

#next_token ⇒ Object 



25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/markbridge/parsers/bbcode/scanner.rb', line 25

def next_token
  return nil if end_of_input?
  start_pos = @current_pos
  # `byteindex` returns 0 for a match at the start — nil-check, never
  # truthiness-check. Can't be 0 here though: callers guarantee
  # @current_pos <= bracket_index.
  bracket_index = @input.byteindex("[", @current_pos)

  if bracket_index.nil?
    text = @input.byteslice(@current_pos, @length - @current_pos)
    @current_pos = @length
    TextToken.new(text:, pos: start_pos)
  elsif bracket_index > @current_pos
    text = @input.byteslice(@current_pos, bracket_index - @current_pos)
    @current_pos = bracket_index
    TextToken.new(text:, pos: start_pos)
  elsif (tag_token = parse_tag_at_cursor)
    tag_token
  else
    @current_pos += 1
    TextToken.new(text: "[", pos: start_pos)
  end
end