Class: Phronomy::Eval::Comparison

Inherits:
Object
  • Object
show all
Defined in:
lib/phronomy/eval/comparison.rb

Overview

Runs the same Dataset against two callables (configuration A and B) and packages the paired results for side-by-side comparison.

Examples:

cmp     = Comparison.new(scorer: Scorer::ExactMatch.new)
pairs   = cmp.compare(dataset, callable_a, callable_b)
pairs.each do |pair|
  puts "A=#{pair.result_a.score} B=#{pair.result_b.score}"
end

# Aggregate each side independently
Metrics.new(pairs.map(&:result_a)).to_h

Defined Under Namespace

Classes: ComparisonPair

Instance Method Summary collapse

Constructor Details

#initialize(scorer: Scorer::ExactMatch.new) ⇒ Comparison

Returns a new instance of Comparison.

Parameters:

  • scorer (Scorer::Base) (defaults to: Scorer::ExactMatch.new) 


22
23
24
 
# File 'lib/phronomy/eval/comparison.rb', line 22

def initialize(scorer: Scorer::ExactMatch.new)
  @scorer = scorer
end

Instance Method Details

#compare(dataset, callable_a, callable_b) ⇒ Array<ComparisonPair>

Evaluates both callables on every case in the dataset.

Parameters:

  • dataset (Dataset)
  • callable_a (#call)
  • callable_b (#call)

Returns:



32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/phronomy/eval/comparison.rb', line 32

def compare(dataset, callable_a, callable_b)
  runner_a = Runner.new(scorer: @scorer)
  runner_b = Runner.new(scorer: @scorer)

  results_a = runner_a.run(dataset, callable_a)
  results_b = runner_b.run(dataset, callable_b)

  results_a.zip(results_b).map do |a, b|
    ComparisonPair.new(eval_case: a.eval_case, result_a: a, result_b: b)
  end
end