Class: RSpec::LLM::Matchers::PassLlmJudge

Inherits:
Object
  • Object
show all
Defined in:
lib/rspec/llm/matchers/pass_llm_judge.rb

Overview

LLM-as-judge matcher. Asks the configured judge model whether the actual response satisfies the given criterion.

When the ruby_llm gem is loaded, the judge prompt is sent with a structured RubyLLM::Schema contract that forces the model to return a machine-readable JSON payload:

{ passed: true/false, reason: "..." }

This eliminates brittle first-token YES/NO parsing and surfaces a rich reason string for failure messages. Adapters (or gem configurations) that do not support structured output fall back automatically to the original YES/NO text-parsing strategy.

Direct Known Subclasses

MatchLlmIntent

Instance Method Summary collapse

Constructor Details

#initialize(criterion) ⇒ PassLlmJudge

Returns a new instance of PassLlmJudge.



20
21
22
23
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 20

def initialize(criterion)
  @criterion = criterion
  @judge = nil
end

Instance Method Details

#description ⇒ Object 



46
47
48
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 46

def description
  "pass LLM judge with criterion: #{@criterion.inspect}"
end

#failure_message ⇒ Object 



50
51
52
53
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 50

def failure_message
  "expected response to pass judge criterion #{@criterion.inspect}, " \
    "but judge said: #{format_reason}\n\nResponse:\n#{@actual}"
end

#failure_message_when_negated ⇒ Object 



55
56
57
58
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 55

def failure_message_when_negated
  "expected response NOT to pass judge criterion #{@criterion.inspect}, " \
    "but judge said: #{format_reason}\n\nResponse:\n#{@actual}"
end

#matches?(actual) ⇒ Boolean

Returns:

  • (Boolean) 


31
32
33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 31

def matches?(actual)
  @actual = actual.to_s
  result = judge_adapter.chat_structured(prompt_for(@actual, @criterion), schema: judge_schema)

  if result.is_a?(Hash)
    @verdict = result[:passed] || result["passed"]
    @reason  = (result[:reason] || result["reason"] || "").to_s
  else
    @verdict_text = result.to_s
    @verdict, @reason = parse_verdict(@verdict_text)
  end

  @verdict == true
end

#using(judge) ⇒ Object

Override the judge for this matcher invocation (optional).



26
27
28
29
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 26

def using(judge)
  @judge = judge
  self
end