Class: CountTokensJob

Inherits:

ApplicationJob

• Object
• ActiveJob::Base
• ApplicationJob
• CountTokensJob

Defined in:

app/jobs/count_tokens_job.rb

Overview

Refines a record’s token_count with the real Anthropic tokenizer count, replacing the local heuristic seeded during creation. Accepts any record that includes TokenEstimation and implements #tokenization_text —Messages, Snapshots, and PinnedMessages share the same pipeline.

Instance Method Summary

• #perform(record) ⇒ Object

Instance Method Details

#perform(record) ⇒ Object

Parameters:

• record (ActiveRecord::Base) —

  any record responding to #tokenization_text and token_count=

# File 'app/jobs/count_tokens_job.rb', line 15

def perform(record)
  count = Providers::Anthropic.new.count_tokens(
    model: Anima::Settings.model,
    messages: [{role: "user", content: record.tokenization_text}]
  )

  record.update!(token_count: count)
end