Module: Myrr::Tokenizer

Defined in:
lib/myrr/tokenizer.rb

Overview

Token counting for agent responses.

Uses OpenAI's cl100k_base encoding via tiktoken_ruby to count tokens in rendered markdown responses.

Examples:

Myrr::Tokenizer.count("Hello, world!")  # => 4

Constant Summary collapse

ENCODING_MODEL =

Uses cl100k_base encoding

"gpt-4"

Class Method Summary collapse

Class Method Details

.count(text) ⇒ Integer

Count the number of tokens in the given text.

Uses OpenAI's cl100k_base encoding via tiktoken_ruby.

Parameters:

  • text (String) —

    The text to count tokens for

Returns:

  • (Integer) —

    Number of tokens



21
22
23
24
25
 
# File 'lib/myrr/tokenizer.rb', line 21

def count(text)
  return 0 if text.nil? || text.strip.empty?
  enc = Tiktoken.encoding_for_model(ENCODING_MODEL)
  enc.encode(text).length
end