Module: Braintrust::Scorer

Included in:
Block
Defined in:
lib/braintrust/scorer.rb

Overview

Scorer wraps a scoring function that evaluates task output against expected values.

Use inline with a block (keyword args):

scorer = Scorer.new("my_scorer") { |expected:, output:| output == expected ? 1.0 : 0.0 }

Or include in a class and define #call with keyword args:

class FuzzyMatch
  include Braintrust::Scorer

  def call(expected:, output:)
    output == expected ? 1.0 : 0.0
  end
end

Legacy callables with 3 or 4 positional params are auto-wrapped for backwards compatibility but emit a deprecation warning.

Defined Under Namespace

Modules: Callable Classes: Block, ID

Constant Summary collapse

DEFAULT_NAME = 
"scorer"

Class Method Summary collapse

Class Method Details

.included(base) ⇒ Object

Parameters:

  • base (Class) —

    the class including Scorer



26
27
28
 
# File 'lib/braintrust/scorer.rb', line 26

def self.included(base)
  base.include(Callable)
end

.new(name = nil, &block) ⇒ Scorer::Block

Create a block-based scorer.

Parameters:

  • name (String, nil) (defaults to: nil) —

    optional name (defaults to “scorer”)

  • block (Proc) —

    the scoring implementation; declare only the keyword args you need. Extra kwargs are filtered out automatically.

    Supported kwargs: input:, expected:, output:, metadata:, trace:

Returns:

Raises:

  • (ArgumentError) —

    if the block has unsupported arity



39
40
41
 
# File 'lib/braintrust/scorer.rb', line 39

def self.new(name = nil, &block)
  Block.new(name: name || DEFAULT_NAME, &block)
end