Class: Gemite::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/gemite/lexer.rb

Overview

Gemiteの「コード」部分( の中身や、{ ... } 補間の中身)を トークン列に変換する。HTMLの生テキストは扱わない(それは TemplateParser の仕事)。

Constant Summary collapse

KEYWORDS = 
{
  "def" => :DEF,
  "end" => :END,
  "return" => :RETURN,
  "true" => :TRUE,
  "false" => :FALSE,
  "nil" => :NIL,
  "and" => :AND,
  "or" => :OR,
  "not" => :NOT,
  "begin" => :BEGIN,
  "catch" => :CATCH,
  "in" => :IN
}.freeze
AT_KEYWORDS = 
{
  "if" => :AT_IF,
  "elseif" => :AT_ELSEIF,
  "else" => :AT_ELSE,
  "endif" => :AT_ENDIF,
  "for" => :AT_FOR,
  "end" => :AT_END
}.freeze
OPERATORS = 
[
  ["==", :EQ], ["!=", :NEQ], ["<=", :LTE], [">=", :GTE],
  ["<", :LT], [">", :GT], ["=", :ASSIGN],
  ["+", :PLUS], ["-", :MINUS], ["*", :STAR], ["/", :SLASH], ["%", :PERCENT],
  ["(", :LPAREN], [")", :RPAREN], ["[", :LBRACKET], ["]", :RBRACKET],
  ["{", :LBRACE], ["}", :RBRACE], [",", :COMMA], [":", :COLON], [".", :DOT]
].freeze
CONTINUATION_TOKENS =

直前のトークンがこれらのいずれかである場合、続く改行は「行継続」とみなし NEWLINEトークンを生成しない。これにより comments = db.select(...) のように "=" や演算子・開き括弧・カンマの直後で改行して値を次行に続える 書き方(仕様書28の例で実際に使われている)を、文終端の改行と区別できる。

%i[
  ASSIGN PLUS MINUS STAR SLASH PERCENT
  EQ NEQ LT LTE GT GTE AND OR NOT
  COMMA COLON DOT LPAREN LBRACKET LBRACE
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(source, line: 1) ⇒ Lexer

Returns a new instance of Lexer.



44
45
46
47
48
49
 
# File 'lib/gemite/lexer.rb', line 44

def initialize(source, line: 1)
  @src = source
  @pos = 0
  @len = source.length
  @line = line
end

Instance Method Details

#tokenize ⇒ Object 



63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/gemite/lexer.rb', line 63

def tokenize
  tokens = []
  loop do
    tok = next_token
    if tok.type == :NEWLINE && CONTINUATION_TOKENS.include?(tokens.last&.type)
      next
    end
    tokens << tok
    break if tok.type == :EOF
  end
  tokens
end