Class: Pdfrb::Source::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/pdfrb/source/tokenizer.rb

Overview

Byte-level PDF lexer (s7.2). State machine that emits Token values via next_token / peek. Pull-based so the Parser can stream through arbitrarily large PDFs without materialising the whole token stream.

States:

:top       — between tokens; dispatch by first byte.
:string    — inside (...) with depth tracking and \ escapes.
:hexstring — inside <...> (after distinguishing << as dict open).
:comment   — inside %... until EOL.

The keyword set per s7.2: obj/endobj/stream/endstream/xref/ startxref/trailer/true/false/null. The Tokenizer emits them with type :keyword and value equal to the keyword string; the Parser dispatches on the value.

Constant Summary collapse

WHITESPACE_BYTES = 
PdfConstants::WHITESPACE.bytes.to_set.freeze
DELIMITER_BYTES = 
PdfConstants::DELIMITERS.bytes.to_set.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(io) ⇒ Tokenizer

Returns a new instance of Tokenizer.



31
32
33
34
35
36
 
# File 'lib/pdfrb/source/tokenizer.rb', line 31

def initialize(io)
  @io = io
  @pos = 0
  @pushback = []
  @lookahead = []
end

Instance Attribute Details

#io ⇒ Object (readonly)

Returns the value of attribute io.



29
30
31
 
# File 'lib/pdfrb/source/tokenizer.rb', line 29

def io
  @io
end

#pos ⇒ Object (readonly)

Returns the value of attribute pos.



29
30
31
 
# File 'lib/pdfrb/source/tokenizer.rb', line 29

def pos
  @pos
end

Instance Method Details

#eof? ⇒ Boolean

Whether the underlying IO is at end of stream.

Returns:

  • (Boolean) 


56
57
58
 
# File 'lib/pdfrb/source/tokenizer.rb', line 56

def eof?
  @io.eof?
end

#next_token ⇒ Object 



38
39
40
41
42
43
 
# File 'lib/pdfrb/source/tokenizer.rb', line 38

def next_token
  return @pushback.pop unless @pushback.empty?

  fill_lookahead(1) if @lookahead.empty?
  @lookahead.shift
end

#peek(offset = 0) ⇒ Object 



45
46
47
48
 
# File 'lib/pdfrb/source/tokenizer.rb', line 45

def peek(offset = 0)
  fill_lookahead(offset + 1)
  @lookahead[offset]
end

#pushback(token) ⇒ Object 



50
51
52
53
 
# File 'lib/pdfrb/source/tokenizer.rb', line 50

def pushback(token)
  @pushback << token
  self
end

#read_byte ⇒ Object

Read one byte from the underlying IO. Returns nil at EOF.



61
62
63
64
65
 
# File 'lib/pdfrb/source/tokenizer.rb', line 61

def read_byte
  b = @io.getbyte
  @pos += 1 if b
  b
end

#skip_whitespace ⇒ Object

Skip whitespace bytes (NUL, HT, LF, FF, CR, SP).



68
69
70
71
72
73
74
 
# File 'lib/pdfrb/source/tokenizer.rb', line 68

def skip_whitespace
  while (b = peek_byte)
    break unless WHITESPACE_BYTES.include?(b)

    advance_byte
  end
end