Class: Insika::Evals::Pairwise

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/evals/pairwise.rb

Overview

PAIRWISE AGAINST THE INCUMBENT — the number that answers "can we replace it". An absolute 0.72 says a reply cleared a bar we invented; it says nothing about whether the system already answering 403,231 chats would have done better with the same customer.

Same opening, two transcripts, one question: which one served the customer better? Three outcomes (better / comparable / worse), plus two the panel can produce and must not hide — split when the judges disagree and unknown when none of them answered in a readable way.

Three honesty rules, each of them the difference between a number people should trust and one they should not:

  1. Anonymous. The judge sees "A" and "B" and is never told which one is Insika. Told, it would have an opinion about the new system rather than about the conversations.
  2. Both orders, every judge. Position bias is THE known failure of pairwise LLM judging, and it is the same species as the kill criterion ("better tracks length, or politeness"). So each judge is asked twice with the transcripts swapped, and a verdict that FLIPS with presentation order is reported as comparable with order_dependent: true — a preference that depends on which one was printed first is not a preference.
  3. A split panel stays split. Averaging "better" and "worse" into "comparable" invents agreement that nobody expressed.

Cost: 2 provider calls per judge per case. This is why it never runs as part of the gate and is opt-in on the CLI.

Defined Under Namespace

Classes: Verdict

Constant Summary collapse

BETTER = 
"better"
COMPARABLE = 
"comparable"
WORSE = 
"worse"
SPLIT = 
"split"
UNKNOWN = 
"unknown"

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(asks:) ⇒ Pairwise

asks: [->(prompt) { "" }] — the SAME panel the rubric judge uses (JudgePanel builds both), because "the judges the operator configured" is one decision, not two.

Raises:

  • (ArgumentError) 


53
54
55
56
 
# File 'lib/insika/evals/pairwise.rb', line 53

def initialize(asks:)
  @asks = Array(asks).compact
  raise ArgumentError, "a pairwise comparison needs at least one `ask`" if @asks.empty?
end

Class Method Details

.reference_transcript(messages) ⇒ Object

The incumbent's half. Human turns are NOT flagged to the judge: what it grades is the conversation as the customer received it, and telling it "a person wrote this one" is an invitation to grade the author instead. The fact is carried to the READER as vs: human-assisted instead, which is where it changes a decision.



84
85
86
87
88
89
 
# File 'lib/insika/evals/pairwise.rb', line 84

def self.reference_transcript(messages)
  Array(messages).map do |m|
    speaker = m["role"].to_s == "user" ? "customer" : "assistant"
    "#{speaker}: #{m['text'].to_s.strip}"
  end.join("\n")
end

.transcript(user_turns, turns) ⇒ Object

The replayed conversation as the judge reads it: the user turns we sent, interleaved with the answers the deployment published. turns may be shorter than user_turns (an errored turn aborts the replay) — zip on what ran.



74
75
76
77
78
 
# File 'lib/insika/evals/pairwise.rb', line 74

def self.transcript(user_turns, turns)
  Array(turns).each_with_index.map do |t, i|
    ["customer: #{user_turns[i]}", "assistant: #{t.output_text.to_s.strip}"]
  end.flatten.join("\n")
end

Instance Method Details

#compare(golden:, turns:) ⇒ Object

golden + the run's [TurnResult] -> Verdict, or nil when the case carries no reference (most of them: a pair is curated by a human, like the case itself).



60
61
62
63
64
65
66
67
68
69
 
# File 'lib/insika/evals/pairwise.rb', line 60

def compare(golden:, turns:)
  return nil unless golden.reference?

  ours = Pairwise.transcript(golden.user_turns, turns)
  theirs = Pairwise.reference_transcript(golden.reference_messages)
  return nil if ours.strip.empty?

  panel = @asks.map { |ask| judge_once(ask, ours, theirs) }
  combine(panel, vs: golden.human_assisted? ? "human-assisted" : "agent")
end