Class: Riffer::Evals::Evaluator

Inherits:
Object
  • Object
show all
Defined in:
lib/riffer/evals/evaluator.rb,
sig/generated/riffer/evals/evaluator.rbs

Overview

Base class for all evaluators. Set instructions and the base class calls the judge automatically; override #evaluate for custom logic. See examples/evaluators/ for reference implementations.

class MyEvaluator < Riffer::Evals::Evaluator
instructions "Assess medical accuracy of the response..."
higher_is_better true
judge_model "anthropic/claude-opus-4-5-20251101"
end

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.higher_is_better(value = nil) ⇒ Boolean

Gets or sets whether higher scores are better.

-- : (?bool?) -> bool

Parameters:

  • (Boolean, nil)

Returns:

  • (Boolean) 


34
35
36
37
38
39
40
41
 
# File 'lib/riffer/evals/evaluator.rb', line 34

def higher_is_better(value = nil)
  if value.nil?
    current = @higher_is_better
    return true if current.nil?
    return current
  end
  @higher_is_better = value
end

.instructions(value = nil) ⇒ String?

Gets or sets the evaluation instructions (criteria and scoring rubric).

-- : (?String?) -> String?

Parameters:

  • (String, nil)

Returns:

  • (String, nil) 


25
26
27
28
 
# File 'lib/riffer/evals/evaluator.rb', line 25

def instructions(value = nil)
  return @instructions if value.nil?
  @instructions = value.to_s
end

.judge_model(value = nil) ⇒ String?

Gets or sets the judge model for LLM-as-judge evaluations.

-- : (?String?) -> String?

Parameters:

  • (String, nil)

Returns:

  • (String, nil) 


47
48
49
50
 
# File 'lib/riffer/evals/evaluator.rb', line 47

def judge_model(value = nil)
  return @judge_model if value.nil?
  @judge_model = value.to_s
end

Instance Method Details

#evaluate(input:, output:, ground_truth: nil, messages: []) ⇒ Riffer::Evals::Result

Evaluates an input/output pair. The default calls the judge with the class-level instructions; override for custom logic (e.g. rule-based evaluators).

: (input: String | Array[Hash[Symbol, untyped] | Riffer::Messages::Base], output: String, ?ground_truth: String?, ?messages: Array) -> Riffer::Evals::Result

Parameters:

Returns:



58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/riffer/evals/evaluator.rb', line 58

def evaluate(input:, output:, ground_truth: nil, messages: [])
  instr = self.class.instructions
  raise NotImplementedError, "#{self.class} must set instructions or implement #evaluate" unless instr

  evaluation = judge.evaluate(
    instructions: instr,
    input: format_input(input),
    output: output,
    ground_truth: ground_truth
  )

  result(score: evaluation[:score], reason: evaluation[:reason])
end

#format_input(input) ⇒ String

-- : (String | Array[Hash[Symbol, untyped] | Riffer::Messages::Base]) -> String

Parameters:

Returns:

  • (String) 


76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/riffer/evals/evaluator.rb', line 76

def format_input(input)
  return input if input.is_a?(String)

  input.map do |msg|
    if msg.is_a?(Hash)
      hash = msg #: Hash[untyped, untyped]
      role = hash[:role] || hash["role"]
      content = hash[:content] || hash["content"]
    else
      role = msg.role
      content = msg.content
    end
    "#{role}: #{content}"
  end.join("\n\n")
end

#judge ⇒ Riffer::Evals::Judge

Returns a Judge instance configured for this evaluator.

-- : () -> Riffer::Evals::Judge

Returns:



98
99
100
101
102
103
104
 
# File 'lib/riffer/evals/evaluator.rb', line 98

def judge
  @judge ||= begin
    model = self.class.judge_model || Riffer.config.evals.judge_model
    raise Riffer::ArgumentError, "No judge model configured. Set judge_model on the evaluator or Riffer.config.evals.judge_model" unless model
    Riffer::Evals::Judge.new(model: model)
  end
end

#result(score:, reason: nil, metadata: {}) ⇒ Riffer::Evals::Result

Builds a Result for this evaluator.

: (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped]) -> Riffer::Evals::Result

Parameters:

  • score: (Float)
  • reason: (String, nil) (defaults to: nil)
  • metadata: (Hash[Symbol, untyped]) (defaults to: {})

Returns:



109
110
111
112
113
114
115
116
117
 
# File 'lib/riffer/evals/evaluator.rb', line 109

def result(score:, reason: nil, metadata: {})
  Riffer::Evals::Result.new(
    evaluator: self.class,
    score: score,
    reason: reason,
    metadata: metadata,
    higher_is_better: self.class.higher_is_better
  )
end