Class: NitroIntelligence::Client::Handlers::BaseHandler

Inherits:
Object
  • Object
show all
Defined in:
lib/nitro_intelligence/client/handlers/base_handler.rb

Constant Summary collapse

MODALITY_HEADER = 
"nip-modality".freeze
REQUESTED_MODEL_HEADER = 
"nip-requested-model".freeze
TRACE_ID_HEADER =

Correlation headers understood by the inference gateway (LiteLLM). See https://docs.litellm.ai/docs/proxy/request_headers

"x-litellm-trace-id".freeze
SPEND_LOGS_METADATA_HEADER = 
"x-litellm-spend-logs-metadata".freeze
MAX_SPEND_LOGS_METADATA_BYTES =

Headers over ~8KB are rejected by most proxies. Metadata is caller supplied, so cap it rather than turning a large hash into a failed request.

4096
RESPONSE_COST_HEADER =

Cost the inference gateway calculated for a response. See https://docs.litellm.ai/docs/proxy/response_headers

The gateway is the only component that knows which deployment actually served a request, and the same model group can be served by internal capacity or by any of several third-party providers at different rates. Recomputing cost downstream from token counts therefore means maintaining a second price table that silently drifts from the one doing the billing.

"x-litellm-response-cost".freeze
RESPONSE_COST_INPUT_HEADER = 
"x-litellm-response-cost-input".freeze
RESPONSE_COST_OUTPUT_HEADER = 
"x-litellm-response-cost-output".freeze

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ BaseHandler

Returns a new instance of BaseHandler.



31
32
33
 
# File 'lib/nitro_intelligence/client/handlers/base_handler.rb', line 31

def initialize(client:)
  @client = client
end

Instance Method Details

#cost_details(response) ⇒ Object

Cost breakdown for an observation, or nil when the gateway did not price the request.

A deployment the gateway has no price for sends no cost header at all rather than a zero one, so absence has to mean "unknown" here. Recording nil leaves the generation without a cost; recording 0.0 would assert the request was free and quietly understate spend for every model still awaiting a price.

Only the total is guaranteed. Where the cost comes from the upstream provider rather than the gateway's own calculation - OpenRouter reports a real per-request cost of its own, which the gateway passes through - there is no component breakdown, so input and output appear only when sent.



48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/nitro_intelligence/client/handlers/base_handler.rb', line 48

def cost_details(response)
  headers = response_headers(response)
  return nil if headers.nil?

  total = header_amount(headers, RESPONSE_COST_HEADER)
  return nil if total.nil?

  {
    total:,
    input: header_amount(headers, RESPONSE_COST_INPUT_HEADER),
    output: header_amount(headers, RESPONSE_COST_OUTPUT_HEADER),
  }.compact
end