Class: Ask::Eval::Configuration

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/eval/configuration.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Configuration

Returns a new instance of Configuration.



16
17
18
19
20
21
 
# File 'lib/ask/eval/configuration.rb', line 16

def initialize
  @default_judge = nil
  @verbose = false
  @track_cost = false
  @cost_tracker = CostTracker.new
end

Instance Attribute Details

#default_judge ⇒ Object?

Returns the default judge model to use for LLM-as-judge assertions. Can be a callable, an Ask::Provider instance, or a model string.

Returns:

  • (Object, nil) —

    the default judge model to use for LLM-as-judge assertions. Can be a callable, an Ask::Provider instance, or a model string.



8
9
10
 
# File 'lib/ask/eval/configuration.rb', line 8

def default_judge
  @default_judge
end

#track_cost ⇒ Boolean

Returns whether to track token usage and costs.

Returns:

  • (Boolean) —

    whether to track token usage and costs



14
15
16
 
# File 'lib/ask/eval/configuration.rb', line 14

def track_cost
  @track_cost
end

#verbose ⇒ Boolean

Returns whether to print detailed output during evaluation.

Returns:

  • (Boolean) —

    whether to print detailed output during evaluation



11
12
13
 
# File 'lib/ask/eval/configuration.rb', line 11

def verbose
  @verbose
end

Instance Method Details

#_accumulate_cost(cost) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Internal: accumulate cost from a judge result.



25
26
27
28
29
30
31
32
33
34
 
# File 'lib/ask/eval/configuration.rb', line 25

def _accumulate_cost(cost)
  return unless @track_cost && cost

  cost_tracker.record(
    model: cost[:model] || "judge",
    input_tokens: cost[:input_tokens] || 0,
    output_tokens: cost[:output_tokens] || 0,
    duration: cost[:duration] || 0
  )
end

#cost_report ⇒ Hash

Returns cost report summary.

Returns:

  • (Hash) —

    cost report summary



42
43
44
 
# File 'lib/ask/eval/configuration.rb', line 42

def cost_report
  cost_tracker.summary
end

#cost_tracker ⇒ Ask::Eval::CostTracker

Returns the cost tracker instance.

Returns:



37
38
39
 
# File 'lib/ask/eval/configuration.rb', line 37

def cost_tracker
  @cost_tracker ||= CostTracker.new
end

#reset! ⇒ Object

Reset all configuration state.



47
48
49
50
51
52
 
# File 'lib/ask/eval/configuration.rb', line 47

def reset!
  @default_judge = nil
  @verbose = false
  @track_cost = false
  @cost_tracker = CostTracker.new
end