Class: Ask::Eval::Judge

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/eval/judge.rb

Overview

Abstract base class for all LLM judges.

A judge evaluates a TestCase and returns a Result with pass/fail, score, reason, and optional cost data.

Examples:

judge = Ask::Eval::Judge::Faithful.new(model: my_model)
result = judge.call(test_case)
result.passed  # => true/false
result.score   # => 0.95
result.reason  # => "The response accurately reflects the context..."

Defined Under Namespace

Classes: Result

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(model: nil, track_cost: false) ⇒ Judge

Returns a new instance of Judge.

Parameters:

  • model (Object, nil) (defaults to: nil) —

    A callable (responds to ‘.call` with messages), an Provider instance, a model string (e.g. “openai/gpt-4o-mini”), or nil to use Configuration#default_judge.

  • track_cost (Boolean) (defaults to: false) —

    whether to track token usage and cost



28
29
30
31
32
 
# File 'lib/ask/eval/judge.rb', line 28

def initialize(model: nil, track_cost: false)
  @model = model || default_model
  @track_cost = track_cost
  @cost_tracker = track_cost ? CostTracker.new : nil
end

Instance Attribute Details

#cost_tracker ⇒ Hash? (readonly)

Returns cost accumulator if tracking is enabled.

Returns:

  • (Hash, nil) —

    cost accumulator if tracking is enabled



22
23
24
 
# File 'lib/ask/eval/judge.rb', line 22

def cost_tracker
  @cost_tracker
end

#model ⇒ Object? (readonly)

Returns the judge model (callable, provider, or nil for auto-detect).

Returns:

  • (Object, nil) —

    the judge model (callable, provider, or nil for auto-detect)



19
20
21
 
# File 'lib/ask/eval/judge.rb', line 19

def model
  @model
end

Instance Method Details

#call(test_case) ⇒ Ask::Eval::Judge::Result

Evaluate a test case and return a verdict.

Parameters:

Returns:

Raises:

  • (NotImplementedError) 


37
38
39
 
# File 'lib/ask/eval/judge.rb', line 37

def call(test_case)
  raise NotImplementedError, "#{self.class} must implement #call"
end