Class: RemLint::ExprLexer

Inherits:
Object
  • Object
show all
Defined in:
lib/remlint/expr_lexer.rb

Overview

Tokenises the inside of a Remind command.

Not a parser. Remind's expression grammar is real, but a linter needs far less than a grammar and gets it much more cheaply: what it needs to know is where the string literals are (so it does not count a [ inside one), where the bracketed expressions are, and where the function calls are. Everything else can stay an undifferentiated run of characters.

Strings matter most. MSG see [ansicolor("")] has balanced brackets; MSG a "]" b does not contain a closing bracket at all. Scanning characters without tracking quotes gets both wrong.

Constant Summary collapse

DOUBLE_QUOTED =

Remind takes both quote characters, with backslash escaping inside.

/"(?:[^"\\]|\\.)*"?/
SINGLE_QUOTED = 
/'(?:[^'\\]|\\.)*'?/
SUBSTITUTION =

%a, %_, %" -- the substitution sequences, one character after the percent. Matched before anything else can claim the character, which is what remind.vim's remindSubst pattern does.

/%[^\s]/
SYSVAR =

$SysInclude, $Latitude.

/\$[A-Za-z_]\w*/
FUNCTION =

A name immediately followed by ( is a call; a name not followed by one is a variable read, and the two need different rules applied to them.

/[A-Za-z_]\w*(?=\()/
NAME = 
/[A-Za-z_]\w*/
NUMBER = 
/\d+(?:\.\d+)?/
WHITESPACE = 
/\s+/

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(text) ⇒ ExprLexer

Returns a new instance of ExprLexer.



60
61
62
 
# File 'lib/remlint/expr_lexer.rb', line 60

def initialize(text)
  @scanner = StringScanner.new(text)
end

Class Method Details

.significant(text) ⇒ Object

Just the tokens that carry meaning -- everything but whitespace.



77
78
79
 
# File 'lib/remlint/expr_lexer.rb', line 77

def self.significant(text)
  tokenise(text).reject { |token| token.type == :whitespace }
end

.tokenise(text) ⇒ Object 



64
65
66
 
# File 'lib/remlint/expr_lexer.rb', line 64

def self.tokenise(text)
  new(text).tokenise
end

Instance Method Details

#tokenise ⇒ Object 



68
69
70
71
72
73
74
 
# File 'lib/remlint/expr_lexer.rb', line 68

def tokenise
  [].tap do |tokens|
    until @scanner.eos?
      tokens << next_token
    end
  end
end