Class: Postsvg::Source::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/postsvg/source/lexer.rb

Overview

Comment-aware PS lexer.

Distinguishes # line comments from % inside string literals, captures DSC comments (lines beginning with %%), and tags every token with its source position (line, column).

The legacy Tokenizer stripped comments with a global gsub(/%[^\n\r]*/, " ") before lexing, which corrupted % chars inside string literals. This implementation walks the source as a state machine so % inside (...) or <...> survives.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source) ⇒ Lexer

Returns a new instance of Lexer.



18
19
20
21
22
23
24
25
 
# File 'lib/postsvg/source/lexer.rb', line 18

def initialize(source)
  @source = source.to_s
  @position = 0
  @line = 1
  @column = 1
  @tokens = []
  @state = :top
end

Instance Attribute Details

#column ⇒ Object (readonly)

Returns the value of attribute column.



16
17
18
 
# File 'lib/postsvg/source/lexer.rb', line 16

def column
  @column
end

#line ⇒ Object (readonly)

Returns the value of attribute line.



16
17
18
 
# File 'lib/postsvg/source/lexer.rb', line 16

def line
  @line
end

#position ⇒ Object (readonly)

Returns the value of attribute position.



16
17
18
 
# File 'lib/postsvg/source/lexer.rb', line 16

def position
  @position
end

#source ⇒ Object (readonly)

Returns the value of attribute source.



16
17
18
 
# File 'lib/postsvg/source/lexer.rb', line 16

def source
  @source
end

Class Method Details

.tokenize(source) ⇒ Object 



40
41
42
 
# File 'lib/postsvg/source/lexer.rb', line 40

def self.tokenize(source)
  new(source).tokenize
end

Instance Method Details

#tokenize ⇒ Object 



27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/postsvg/source/lexer.rb', line 27

def tokenize
  until eos?
    case @state
    when :top        then scan_top
    when :string     then scan_string_body
    when :hexstring  then scan_hexstring_body
    when :comment    then scan_comment_body
    when :dsc        then scan_dsc_body
    end
  end
  @tokens.freeze
end