Class: Natsuzora::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/natsuzora/lexer.rb,
lib/natsuzora/lexer/compiled_lexer.rb,
lib/natsuzora/lexer/token_processor.rb

Overview

Lexer for Natsuzora template language using LexerKit

Responsibilities:

  • Escape sequence processing ([{] ->

  • Whitespace control via TokenProcessor ({[- and -])

  • Error handling for invalid characters

  • EOF token addition

Defined Under Namespace

Modules: CompiledLexer Classes: TokenProcessor

Constant Summary collapse

ESCAPE_SEQUENCE = 
'{[{]}'
ESCAPED_VALUE = 
'{['

Instance Method Summary collapse

Constructor Details

#initialize(source) ⇒ Lexer

Returns a new instance of Lexer.



20
21
22
 
# File 'lib/natsuzora/lexer.rb', line 20

def initialize(source)
  @source = source
end

Instance Method Details

#tokenize ⇒ Object 



24
25
26
27
28
29
30
 
# File 'lib/natsuzora/lexer.rb', line 24

def tokenize
  stream = CompiledLexer.instance.stream(@source)
  mapped_tokens = map_tokens_from_stream(stream)
  processed_tokens = TokenProcessor.new(mapped_tokens).process
  add_eof(processed_tokens)
  processed_tokens
end