Class: Pdfrb::Source::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/pdfrb/source/tokenizer.rb

Overview

Byte-level PDF lexer (s7.2). State machine that emits Token values via next_token / peek. Pull-based so the Parser can stream through arbitrarily large PDFs without materialising the whole token stream.

States:

:top       — between tokens; dispatch by first byte.
:string    — inside (...) with depth tracking and \ escapes.
:hexstring — inside <...> (after distinguishing << as dict open).
:comment   — inside %... until EOL.

The keyword set per s7.2: obj/endobj/stream/endstream/xref/ startxref/trailer/true/false/null. The Tokenizer emits them with type :keyword and value equal to the keyword string; the Parser dispatches on the value.

Constant Summary collapse

WHITESPACE_BYTES = 
PdfConstants::WHITESPACE.bytes.to_set.freeze
DELIMITER_BYTES = 
PdfConstants::DELIMITERS.bytes.to_set.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(io) ⇒ Tokenizer

Returns a new instance of Tokenizer.



33
34
35
36
37
38
 
# File 'lib/pdfrb/source/tokenizer.rb', line 33

def initialize(io)
  @io = io
  @pos = 0
  @pushback = []
  @lookahead = []
end

Instance Attribute Details

#io ⇒ Object (readonly)

Returns the value of attribute io.



31
32
33
 
# File 'lib/pdfrb/source/tokenizer.rb', line 31

def io
  @io
end

#pos ⇒ Object (readonly)

Returns the value of attribute pos.



31
32
33
 
# File 'lib/pdfrb/source/tokenizer.rb', line 31

def pos
  @pos
end

Instance Method Details

#next_token ⇒ Object 



40
41
42
43
44
45
 
# File 'lib/pdfrb/source/tokenizer.rb', line 40

def next_token
  return @pushback.pop unless @pushback.empty?

  fill_lookahead(1) if @lookahead.empty?
  @lookahead.shift
end

#peek(offset = 0) ⇒ Object 



47
48
49
50
 
# File 'lib/pdfrb/source/tokenizer.rb', line 47

def peek(offset = 0)
  fill_lookahead(offset + 1)
  @lookahead[offset]
end

#pushback(token) ⇒ Object 



52
53
54
55
 
# File 'lib/pdfrb/source/tokenizer.rb', line 52

def pushback(token)
  @pushback << token
  self
end