Class: JRRToken::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/j_r_r_token.rb

Overview

This is the main public interface for the gem.

Class Method Summary collapse

Class Method Details

.count(text, model:) ⇒ Integer

Counts tokens for a given text using a specified model name. The underlying Rust extension will handle mapping the model name to the correct tokenizer.

Parameters:

  • text (String) —

    The text to tokenize.

  • model (String) —

    The name of the model (e.g., “gpt-4o”, “gpt-3.5-turbo”).

Returns:

  • (Integer) —

    The number of tokens.



28
29
30
31
32
 
# File 'lib/j_r_r_token.rb', line 28

def self.count(text, model:)
  # This calls the `count_tokens` function from our Rust code,
  # passing both the model and the text.
  JRRToken.count_tokens(model, text.to_s)
end