Class: RSpec::LLM::Matchers::PassLlmJudge

Inherits:
Object
  • Object
show all
Defined in:
lib/rspec/llm/matchers/pass_llm_judge.rb

Overview

LLM-as-judge matcher. Asks the configured judge model whether the actual response satisfies the given criterion. Parses YES/NO from the first non-whitespace token of the judge’s reply.

Direct Known Subclasses

MatchLlmIntent

Instance Method Summary collapse

Constructor Details

#initialize(criterion) ⇒ PassLlmJudge

Returns a new instance of PassLlmJudge.



10
11
12
13
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 10

def initialize(criterion)
  @criterion = criterion
  @judge = nil
end

Instance Method Details

#description ⇒ Object 



28
29
30
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 28

def description
  "pass LLM judge with criterion: #{@criterion.inspect}"
end

#failure_message ⇒ Object 



32
33
34
35
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 32

def failure_message
  "expected response to pass judge criterion #{@criterion.inspect}, " \
    "but judge said: #{format_reason}\n\nResponse:\n#{@actual}"
end

#failure_message_when_negated ⇒ Object 



37
38
39
40
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 37

def failure_message_when_negated
  "expected response NOT to pass judge criterion #{@criterion.inspect}, " \
    "but judge said: #{format_reason}\n\nResponse:\n#{@actual}"
end

#matches?(actual) ⇒ Boolean

Returns:

  • (Boolean) 


21
22
23
24
25
26
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 21

def matches?(actual)
  @actual = actual.to_s
  @verdict_text = judge_adapter.chat(prompt_for(@actual, @criterion))
  @verdict, @reason = parse_verdict(@verdict_text)
  @verdict == true
end

#using(judge) ⇒ Object

Override the judge for this matcher invocation (optional).



16
17
18
19
 
# File 'lib/rspec/llm/matchers/pass_llm_judge.rb', line 16

def using(judge)
  @judge = judge
  self
end