Class: Riffer::Evals::Evaluator

Inherits:
Object
  • Object
show all
Extended by:
Helpers::ClassNameConverter
Defined in:
lib/riffer/evals/evaluator.rb

Overview

Base class for all evaluators in the Riffer framework.

Provides a DSL for defining evaluator metadata and the evaluate method. Subclasses must implement the evaluate method.

See Riffer::Evals::Evaluators.

class MyEvaluator < Riffer::Evals::Evaluator
  identifier "my_evaluator"
  description "Evaluates response quality"
  higher_is_better true
  judge_model "anthropic/claude-opus-4-5-20251101"

  def evaluate(input:, output:, context: nil)
    evaluation = judge.evaluate(
      system_prompt: "...",
      user_prompt: "..."
    )
    result(score: evaluation[:score], reason: evaluation[:reason])
  end
end

Constant Summary

Constants included from Helpers::ClassNameConverter

Helpers::ClassNameConverter::DEFAULT_SEPARATOR

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Helpers::ClassNameConverter

class_name_to_path

Class Method Details

.description(value = nil) ⇒ Object

Gets or sets the evaluator description.

: (?String?) -> String?



41
42
43
44
 
# File 'lib/riffer/evals/evaluator.rb', line 41

def description(value = nil)
  return @description if value.nil?
  @description = value.to_s
end

.higher_is_better(value = nil) ⇒ Object

Gets or sets whether higher scores are better.

: (?bool?) -> bool



49
50
51
52
 
# File 'lib/riffer/evals/evaluator.rb', line 49

def higher_is_better(value = nil)
  return @higher_is_better.nil? || @higher_is_better if value.nil?
  @higher_is_better = value
end

.identifier(value = nil) ⇒ Object

Gets or sets the evaluator identifier.

: (?String?) -> String



33
34
35
36
 
# File 'lib/riffer/evals/evaluator.rb', line 33

def identifier(value = nil)
  return @identifier || class_name_to_identifier(name) if value.nil?
  @identifier = value.to_s
end

.judge_model(value = nil) ⇒ Object

Gets or sets the judge model for LLM-as-judge evaluations.

: (?String?) -> String?



57
58
59
60
 
# File 'lib/riffer/evals/evaluator.rb', line 57

def judge_model(value = nil)
  return @judge_model if value.nil?
  @judge_model = value.to_s
end

Instance Method Details

#evaluate(input:, output:, context: nil) ⇒ Object

Evaluates an input/output pair.

Raises NotImplementedError if not implemented by subclass.

: (input: String, output: String, ?context: Hash[Symbol, untyped]?) -> Riffer::Evals::Result

Raises:

  • (NotImplementedError) 


78
79
80
 
# File 'lib/riffer/evals/evaluator.rb', line 78

def evaluate(input:, output:, context: nil)
  raise NotImplementedError, "#{self.class} must implement #evaluate"
end