Module: OllamaAgent::Context::TokenCounter

Defined in:
lib/ollama_agent/context/token_counter.rb

Overview

Estimates token count using chars/4 integer division (floor). Auto-upgrades to tiktoken_ruby (gpt-4 tokenizer) if available in host bundle.

Class Method Summary collapse

Class Method Details

.count(text:) ⇒ Object

Keyword API for injectable token counters (e.g. ContextBuilder).



19
20
21
 
# File 'lib/ollama_agent/context/token_counter.rb', line 19

def self.count(text:)
  estimate(text)
end

.estimate(text) ⇒ Object 



11
12
13
14
15
16
 
# File 'lib/ollama_agent/context/token_counter.rb', line 11

def self.estimate(text)
  ensure_tiktoken_loaded
  return @tokenizer.encode(text.to_s).length if @tokenizer

  text.to_s.length / 4
end