Class: Riffer::Evals::Evaluator

Inherits:
Object
  • Object
show all
Defined in:
lib/riffer/evals/evaluator.rb

Overview

Base class for all evaluators in the Riffer framework.

Provides a DSL for defining evaluator metadata and the evaluate method. Subclasses must implement the evaluate method.

See Riffer::Evals::Evaluators.

class MyEvaluator < Riffer::Evals::Evaluator
  description "Evaluates response quality"
  higher_is_better true
  judge_model "anthropic/claude-opus-4-5-20251101"

  def evaluate(input:, output:, context: nil)
    evaluation = judge.evaluate(
      system_prompt: "...",
      user_prompt: "..."
    )
    result(score: evaluation[:score], reason: evaluation[:reason])
  end
end

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.description(value = nil) ⇒ Object

Gets or sets the evaluator description.

: (?String?) -> String?



30
31
32
33
 
# File 'lib/riffer/evals/evaluator.rb', line 30

def description(value = nil)
  return @description if value.nil?
  @description = value.to_s
end

.higher_is_better(value = nil) ⇒ Object

Gets or sets whether higher scores are better.

: (?bool?) -> bool



38
39
40
41
 
# File 'lib/riffer/evals/evaluator.rb', line 38

def higher_is_better(value = nil)
  return @higher_is_better.nil? || @higher_is_better if value.nil?
  @higher_is_better = value
end

.judge_model(value = nil) ⇒ Object

Gets or sets the judge model for LLM-as-judge evaluations.

: (?String?) -> String?



46
47
48
49
 
# File 'lib/riffer/evals/evaluator.rb', line 46

def judge_model(value = nil)
  return @judge_model if value.nil?
  @judge_model = value.to_s
end

Instance Method Details

#evaluate(input:, output:, context: nil) ⇒ Object

Evaluates an input/output pair.

Raises NotImplementedError if not implemented by subclass.

: (input: String, output: String, ?context: Hash[Symbol, untyped]?) -> Riffer::Evals::Result

Raises:

  • (NotImplementedError) 


57
58
59
 
# File 'lib/riffer/evals/evaluator.rb', line 57

def evaluate(input:, output:, context: nil)
  raise NotImplementedError, "#{self.class} must implement #evaluate"
end