Module: TokenReel::Tokenizer

Defined in:
lib/token_reel/tokenizer.rb

Overview

Splits text into the chunks that get revealed one-by-one while streaming. Joining the tokens back together always reproduces the original string exactly.

Class Method Summary collapse

Class Method Details

.tokenize(text, unit) ⇒ Object 



8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
# File 'lib/token_reel/tokenizer.rb', line 8

def self.tokenize(text, unit)
  return [] if text.nil? || text.empty?

  case unit.to_sym
  when :char
    text.each_char.to_a
  when :word
    # "word" + any trailing whitespace travels together, so a token
    # reveals as a whole word (closer to how LLM tokens/BPE chunks
    # tend to land) instead of one raw character at a time.
    text.scan(/\S+\s*|\s+/)
  else
    raise ConfigError, "unknown unit #{unit.inspect} (valid: word, char)"
  end
end