Class: Blusher::Shim::TokenStream

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/blusher/shim.rb

Overview

A deferred token stream: returned by the patched ‘RegexLexer#lex` when no block is given, so the lexer + source travel together to the formatter. `Rouge::Formatters::HTML#format` can then fuse lex+format in Rust (one String, no per-token Ruby objects). Any OTHER consumer just iterates it —`#each` lexes via carmine on demand, yielding the same [Token, value] pairs rouge would, so it is a transparent drop-in for `lex`’s old return.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(lexer, source) ⇒ TokenStream

Returns a new instance of TokenStream.



33
34
35
36
 
# File 'lib/blusher/shim.rb', line 33

def initialize(lexer, source)
  @lexer = lexer
  @source = source
end

Instance Attribute Details

#lexer ⇒ Object (readonly)

Returns the value of attribute lexer.



31
32
33
 
# File 'lib/blusher/shim.rb', line 31

def lexer
  @lexer
end

#source ⇒ Object (readonly)

Returns the value of attribute source.



31
32
33
 
# File 'lib/blusher/shim.rb', line 31

def source
  @source
end

Instance Method Details

#each(&b) ⇒ Object 



42
43
44
45
46
47
 
# File 'lib/blusher/shim.rb', line 42

def each(&b)
  return enum_for(:each) unless b
  toks = Blusher::Native.lex(tag, Shim.table_for(tag), @source, QUALNAME)
  toks ||= @lexer.__blusher_rouge_lex(@source).to_a
  toks.each(&b)
end

#tag ⇒ Object 



38
39
40
 
# File 'lib/blusher/shim.rb', line 38

def tag
  @lexer.class.tag
end