Class: Phronomy::Eval::Scorer::LlmJudge

Inherits:
Base
  • Object
show all
Defined in:
lib/phronomy/eval/scorer/llm_judge.rb

Overview

LLM-as-a-Judge scorer. Sends a structured prompt to an LLM and interprets its numeric reply as a quality score in [0.0, 1.0].

The prompt template accepts three named placeholders: %s — the original input question %s — the ground-truth / reference answer %s — the output being evaluated

The LLM is expected to reply with a single decimal number; any extra text is stripped and the value is clamped to [0.0, 1.0]. If parsing fails the scorer returns 0.0 rather than raising.

Examples:

judge = LlmJudge.new(model: "gpt-4o-mini")
judge.score(actual: "Paris", expected: "Paris", input: "Capital of France?")

Constant Summary collapse

DEFAULT_PROMPT = 
<<~PROMPT
  You are an impartial judge evaluating the quality of an AI assistant response.
  Rate the response on a scale from 0.0 (completely wrong or unhelpful) to 1.0 (perfect).
  Respond with ONLY a single decimal number between 0.0 and 1.0 — no other text.

  Question: %<input>s
  Expected answer: %<expected>s
  Actual response: %<actual>s

  Score:
PROMPT

Instance Method Summary collapse

Constructor Details

#initialize(model:, prompt_template: DEFAULT_PROMPT, raise_on_error: false) ⇒ LlmJudge

Returns a new instance of LlmJudge.

Parameters:

  • model (String) —

    RubyLLM model identifier

  • prompt_template (String) (defaults to: DEFAULT_PROMPT) —

    format string with %s, %s, %s

  • raise_on_error (Boolean) (defaults to: false) —

    when true, re-raises scoring exceptions instead of returning 0.0. Use this in batch eval pipelines where silent failures are unacceptable.



39
40
41
42
43
 
# File 'lib/phronomy/eval/scorer/llm_judge.rb', line 39

def initialize(model:, prompt_template: DEFAULT_PROMPT, raise_on_error: false)
  @model = model
  @prompt_template = prompt_template
  @raise_on_error = raise_on_error
end

Instance Method Details

#score(actual:, expected:, input: nil) ⇒ Float

Returns score in [0.0, 1.0]; 0.0 on error when raise_on_error is false.

Returns:

  • (Float) —

    score in [0.0, 1.0]; 0.0 on error when raise_on_error is false



46
47
48
49
50
51
52
53
54
55
 
# File 'lib/phronomy/eval/scorer/llm_judge.rb', line 46

def score(actual:, expected:, input: nil)
  prompt = format(@prompt_template, input: input.to_s, expected: expected.to_s, actual: actual.to_s)
  response = RubyLLM.chat(model: @model).ask(prompt)
  response.content.to_s.strip.scan(/-?\d+\.?\d*/).first.to_f.clamp(0.0, 1.0)
rescue => e
  raise if @raise_on_error

  warn "[LlmJudge] Scoring failed: #{e.message}"
  0.0
end