Module: Ask::Eval::Assertions::Judge

Defined in:
lib/ask/eval/assertions/judge.rb

Overview

Wraps an LLM judge call into a standard assertion result. This is the bridge between the judge system and the assertion runner.

Class Method Summary collapse

Class Method Details

.evaluate(judge_class, test_case, model: nil, threshold: 0.7, track_cost: false) ⇒ Ask::Eval::Judge::Result

Run a judge assertion against a test case.

Parameters:

  • judge_class (Class) —

    the judge class (e.g., Judges::Faithful)

  • test_case (Ask::Eval::TestCase)
  • model (Object, nil) (defaults to: nil) —

    the judge model

  • threshold (Float) (defaults to: 0.7) —

    minimum acceptable score

  • track_cost (Boolean) (defaults to: false) —

    whether to track costs

Returns:



18
19
20
21
22
23
24
25
26
27
28
 
# File 'lib/ask/eval/assertions/judge.rb', line 18

def evaluate(judge_class, test_case, model: nil, threshold: 0.7, track_cost: false)
  judge = judge_class.new(model: model, track_cost: track_cost)
  result = judge.call(test_case)
  passed = result.score >= threshold
  Ask::Eval::Judge::Result.new(
    passed: passed,
    score: result.score,
    reason: result.reason,
    cost: result.cost
  )
end