Class: LaunchDarkly::Server::AI::AIConfigTracker

Inherits:
Object
  • Object
show all
Defined in:
lib/server/ai/ai_config_tracker.rb

Overview

The AIConfigTracker records metrics for a single AI run. Unless otherwise noted, the tracker’s methods are not safe for concurrent use.

All events a tracker emits share a runId (a UUIDv4) so LaunchDarkly can correlate them in metrics views. See individual track methods for their specific semantics. Call create_tracker on the AI Config to start a new run. A resumption token preserves the runId, so events emitted by a tracker reconstructed in another process share the original runId.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(ld_client:, run_id:, config_key:, variation_key:, version:, context:, model_name:, provider_name:) ⇒ AIConfigTracker

Initialize a new AIConfigTracker instance.

Parameters:

  • ld_client (LDClient) —

    The LaunchDarkly client instance

  • variation_key (String) —

    The variation key from the flag evaluation

  • config_key (String) —

    The configuration key

  • version (Integer) —

    The version number

  • model_name (String) —

    The name of the AI model being used

  • provider_name (String) —

    The name of the AI provider

  • context (LDContext) —

    The context used for the flag evaluation



67
68
69
70
71
72
73
74
75
76
77
78
 
# File 'lib/server/ai/ai_config_tracker.rb', line 67

def initialize(ld_client:, run_id:, config_key:, variation_key:, version:, context:, model_name:, provider_name:)
  @ld_client = ld_client
  @variation_key = variation_key
  @config_key = config_key
  @version = version
  @model_name = model_name
  @provider_name = provider_name
  @context = context
  @summary = MetricSummary.new
  @run_id = run_id
  @logger = LaunchDarkly::Server::AI.default_logger
end

Instance Attribute Details

#config_key ⇒ Object (readonly)

Returns the value of attribute config_key.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def config_key
  @config_key
end

#context ⇒ Object (readonly)

Returns the value of attribute context.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def context
  @context
end

#ld_client ⇒ Object (readonly)

Returns the value of attribute ld_client.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def ld_client
  @ld_client
end

#model_name ⇒ Object (readonly)

Returns the value of attribute model_name.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def model_name
  @model_name
end

#provider_name ⇒ Object (readonly)

Returns the value of attribute provider_name.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def provider_name
  @provider_name
end

#summary ⇒ Object (readonly)

Returns the value of attribute summary.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def summary
  @summary
end

#variation_key ⇒ Object (readonly)

Returns the value of attribute variation_key.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def variation_key
  @variation_key
end

#version ⇒ Object (readonly)

Returns the value of attribute version.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def version
  @version
end

Class Method Details

.from_resumption_token(token:, ld_client:, context:) ⇒ AIConfigTracker

Reconstructs a tracker from a resumption token.

Parameters:

  • token (String) —

    A URL-safe Base64-encoded JSON resumption token

  • ld_client (LDClient) —

    The LaunchDarkly client instance

  • context (LDContext) —

    The context for track events

Returns:



104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'lib/server/ai/ai_config_tracker.rb', line 104

def self.from_resumption_token(token:, ld_client:, context:)
  json = Base64.urlsafe_decode64(token)
  payload = JSON.parse(json)

  new(
    ld_client: ld_client,
    run_id: payload['runId'],
    config_key: payload['configKey'],
    variation_key: payload.fetch('variationKey', ''),
    version: payload['version'],
    context: context,
    model_name: '',
    provider_name: ''
  )
end

Instance Method Details

#resumption_token ⇒ String

Returns a URL-safe Base64-encoded JSON token that can be used to reconstruct a tracker in a different process (e.g. for deferred feedback).

The token contains: runId, configKey, variationKey, version. modelName and providerName are NOT included.

Returns:

  • (String) —

    the resumption token



89
90
91
92
93
94
 
# File 'lib/server/ai/ai_config_tracker.rb', line 89

def resumption_token
  payload = { runId: @run_id, configKey: @config_key }
  payload[:variationKey] = @variation_key if @variation_key && !@variation_key.empty?
  payload[:version] = @version
  Base64.urlsafe_encode64(JSON.generate(payload), padding: false)
end

#track_bedrock_converse_metrics { ... } ⇒ Hash

Track AWS Bedrock conversation operations. This method tracks the duration, token usage, and success/error status.

Subsequent calls re-run the inner block but emit only metrics not already recorded on this Tracker. Call create_tracker on the AI Config to start a new run.

Yields:

  • The block to track.

Returns:

  • (Hash) —

    The original response hash.



314
315
316
317
318
319
320
321
322
 
# File 'lib/server/ai/ai_config_tracker.rb', line 314

def track_bedrock_converse_metrics(&block)
  result = track_duration_of(&block)
  track_success
  track_tokens(bedrock_to_token_usage(result[:usage])) if result[:usage]
  result
rescue StandardError
  track_error
  raise
end

#track_duration(duration) ⇒ Object

Track the duration of an AI run.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • duration (Integer) —

    The duration in milliseconds



127
128
129
130
131
132
133
134
135
136
137
138
139
 
# File 'lib/server/ai/ai_config_tracker.rb', line 127

def track_duration(duration)
  unless @summary.duration.nil?
    @logger&.warn("Skipping track_duration: duration already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.duration = duration
  @ld_client.track(
    '$ld:ai:duration:total',
    @context,
    flag_data,
    duration
  )
end

#track_duration_of { ... } ⇒ Object

Track the duration of a block of code

Yields:

  • The block to measure

Returns:

  • The result of the block



147
148
149
150
151
152
153
 
# File 'lib/server/ai/ai_config_tracker.rb', line 147

def track_duration_of(&block)
  start_time = Time.now
  yield
ensure
  duration = ((Time.now - start_time) * 1000).to_i
  track_duration(duration)
end

#track_error ⇒ Object

Track an error in AI generation.

Records at most once per Tracker. track_success and track_error share state; only one of the two can record per Tracker, and subsequent calls are ignored.



227
228
229
230
231
232
233
234
235
236
237
238
239
 
# File 'lib/server/ai/ai_config_tracker.rb', line 227

def track_error
  unless @summary.success.nil?
    @logger&.warn("Skipping track_error: success/error already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.success = false
  @ld_client.track(
    '$ld:ai:generation:error',
    @context,
    flag_data,
    1
  )
end

#track_feedback(kind:) ⇒ Object

Track user feedback.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • kind (Symbol) —

    The kind of feedback (:positive or :negative)



184
185
186
187
188
189
190
191
192
193
194
195
196
197
 
# File 'lib/server/ai/ai_config_tracker.rb', line 184

def track_feedback(kind:)
  unless @summary.feedback.nil?
    @logger&.warn("Skipping track_feedback: feedback already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.feedback = kind
  event_name = kind == :positive ? '$ld:ai:feedback:user:positive' : '$ld:ai:feedback:user:negative'
  @ld_client.track(
    event_name,
    @context,
    flag_data,
    1
  )
end

#track_openai_metrics { ... } ⇒ Object

Track OpenAI-specific operations. This method tracks the duration, token usage, and success/error status. If the provided block raises, this method will also raise. A failed operation will not have any token usage data.

Subsequent calls re-run the inner block but emit only metrics not already recorded on this Tracker. Call create_tracker on the AI Config to start a new run.

Yields:

  • The block to track.

Returns:

  • The result of the tracked block.



293
294
295
296
297
298
299
300
301
 
# File 'lib/server/ai/ai_config_tracker.rb', line 293

def track_openai_metrics(&block)
  result = track_duration_of(&block)
  track_success
  track_tokens(openai_to_token_usage(result[:usage])) if result[:usage]
  result
rescue StandardError
  track_error
  raise
end

#track_success ⇒ Object

Track a successful AI generation.

Records at most once per Tracker. track_success and track_error share state; only one of the two can record per Tracker, and subsequent calls are ignored.



206
207
208
209
210
211
212
213
214
215
216
217
218
 
# File 'lib/server/ai/ai_config_tracker.rb', line 206

def track_success
  unless @summary.success.nil?
    @logger&.warn("Skipping track_success: success/error already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.success = true
  @ld_client.track(
    '$ld:ai:generation:success',
    @context,
    flag_data,
    1
  )
end

#track_time_to_first_token(time_to_first_token) ⇒ Object

Track time to first token.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • duration (Integer) —

    The duration in milliseconds



162
163
164
165
166
167
168
169
170
171
172
173
174
175
 
# File 'lib/server/ai/ai_config_tracker.rb', line 162

def track_time_to_first_token(time_to_first_token)
  unless @summary.time_to_first_token.nil?
    @logger&.warn("Skipping track_time_to_first_token: time-to-first-token already recorded on this tracker. " \
                  "Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.time_to_first_token = time_to_first_token
  @ld_client.track(
    '$ld:ai:tokens:ttf',
    @context,
    flag_data,
    time_to_first_token
  )
end

#track_tokens(token_usage) ⇒ Object

Track token usage.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • token_usage (TokenUsage) —

    An object containing token usage details



248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/server/ai/ai_config_tracker.rb', line 248

def track_tokens(token_usage)
  unless @summary.usage.nil?
    @logger&.warn("Skipping track_tokens: token usage already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.usage = token_usage
  if token_usage.total.positive?
    @ld_client.track(
      '$ld:ai:tokens:total',
      @context,
      flag_data,
      token_usage.total
    )
  end
  if token_usage.input.positive?
    @ld_client.track(
      '$ld:ai:tokens:input',
      @context,
      flag_data,
      token_usage.input
    )
  end
  return unless token_usage.output.positive?

  @ld_client.track(
    '$ld:ai:tokens:output',
    @context,
    flag_data,
    token_usage.output
  )
end