Class: OllamaChat::TokenEstimator::Crude

Inherits:
Object
  • Object
show all
Defined in:
lib/ollama_chat/token_estimator/crude.rb

Overview

Provides a "best-effort" estimation of token counts based on the character count or byte size of the input content.

Instance Method Summary collapse

Constructor Details

#initialize(arg) ⇒ Crude

Initializes a new crude estimator with the provided source.

Parameters:

  • arg (String, Integer) —

    The content to estimate (string or raw byte count).

Raises:

  • (ArgumentError) —

    if the input is not a string or an integer.



8
9
10
11
12
13
14
15
16
 
# File 'lib/ollama_chat/token_estimator/crude.rb', line 8

def initialize(arg)
  if text = arg.ask_and_send(:to_str)
    @bytes = text.size
  elsif bytes = arg.ask_and_send(:to_int)
    @bytes = bytes
  else
    raise ArgumentError, "#{arg.inspect} cannot be used to estimate"
  end
end

Instance Method Details

#perform ⇒ OllamaChat::TokenEstimator::Estimate

Performs the estimation calculation and returns an Estimate object.

Returns:



21
22
23
24
 
# File 'lib/ollama_chat/token_estimator/crude.rb', line 21

def perform
  tokens = (@bytes.to_f / 3.5).ceil
  OllamaChat::TokenEstimator::Estimate.new(bytes: @bytes, tokens:)
end