Class: Braintrust::Eval::Context

Inherits:
Object
  • Object
show all
Defined in:
lib/braintrust/eval/context.rb

Overview

Holds all normalized, ready-to-execute eval components. Use Context.build to construct from raw user inputs.

Defined Under Namespace

Classes: Factory

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(task:, scorers:, cases:, experiment_id: nil, experiment_name: nil, project_id: nil, project_name: nil, state: nil, tracer_provider: nil, on_progress: nil, parent_span_attr: nil, generation: nil, parameters: nil) ⇒ Context

Returns a new instance of Context.

Parameters:

  • task (Task) —

    Normalized task wrapper

  • scorers (Array<Scorer>) —

    Normalized scorer wrappers

  • cases (Cases) —

    Normalized eval cases

  • experiment_id (String, nil) (defaults to: nil) —

    Experiment ID for logging and trace linkage

  • experiment_name (String, nil) (defaults to: nil) —

    Experiment name, included in span attributes

  • project_id (String, nil) (defaults to: nil) —

    Project ID

  • project_name (String, nil) (defaults to: nil) —

    Project name

  • state (Braintrust::State, nil) (defaults to: nil) —

    Authenticated API state; nil for local-only evals

  • tracer_provider (#tracer, nil) (defaults to: nil) —

    OpenTelemetry tracer provider

  • on_progress (Proc, nil) (defaults to: nil) —

    Callback invoked after each case completes, receiving a progress Hash

  • parent_span_attr (String, nil) (defaults to: nil) —

    Formatted parent span identifier (“type:id”), linking spans to a parent context

  • generation (Integer, nil) (defaults to: nil) —

    Generation number from the parent span context, used to link spans in a trace hierarchy

  • parameters (Hash, nil) (defaults to: nil) —

    Runtime parameters passed to task and scorers as a ‘parameters:` keyword argument



27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/braintrust/eval/context.rb', line 27

def initialize(task:, scorers:, cases:, experiment_id: nil, experiment_name: nil,
  project_id: nil, project_name: nil, state: nil, tracer_provider: nil,
  on_progress: nil, parent_span_attr: nil, generation: nil, parameters: nil)
  @task = task
  @scorers = scorers
  @cases = cases
  @experiment_id = experiment_id
  @experiment_name = experiment_name
  @project_id = project_id
  @project_name = project_name
  @state = state
  @tracer_provider = tracer_provider
  @on_progress = on_progress
  @parent_span_attr = parent_span_attr
  @generation = generation
  @parameters = parameters
end

Instance Attribute Details

#cases ⇒ Object (readonly)

Returns the value of attribute cases.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def cases
  @cases
end

#experiment_id ⇒ Object (readonly)

Returns the value of attribute experiment_id.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def experiment_id
  @experiment_id
end

#experiment_name ⇒ Object (readonly)

Returns the value of attribute experiment_name.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def experiment_name
  @experiment_name
end

#generation ⇒ Object (readonly)

Returns the value of attribute generation.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def generation
  @generation
end

#on_progress ⇒ Object (readonly)

Returns the value of attribute on_progress.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def on_progress
  @on_progress
end

#parameters ⇒ Object (readonly)

Returns the value of attribute parameters.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def parameters
  @parameters
end

#parent_span_attr ⇒ Object (readonly)

Returns the value of attribute parent_span_attr.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def parent_span_attr
  @parent_span_attr
end

#project_id ⇒ Object (readonly)

Returns the value of attribute project_id.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def project_id
  @project_id
end

#project_name ⇒ Object (readonly)

Returns the value of attribute project_name.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def project_name
  @project_name
end

#scorers ⇒ Object (readonly)

Returns the value of attribute scorers.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def scorers
  @scorers
end

#state ⇒ Object (readonly)

Returns the value of attribute state.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def state
  @state
end

#task ⇒ Object (readonly)

Returns the value of attribute task.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def task
  @task
end

#tracer_provider ⇒ Object (readonly)

Returns the value of attribute tracer_provider.



10
11
12
 
# File 'lib/braintrust/eval/context.rb', line 10

def tracer_provider
  @tracer_provider
end

Class Method Details

.build(task:, scorers:, cases:, experiment_id: nil, experiment_name: nil, project_id: nil, project_name: nil, state: nil, tracer_provider: nil, on_progress: nil, parent: nil, parameters: nil) ⇒ Context

Build a Context from raw user inputs. Delegates to Factory for normalization.

Parameters:

  • task (Task, Proc, #call) —

    Task to evaluate; wrapped into a Task if needed

  • scorers (Array<Scorer, Proc, String, Scorer::ID, #call>) —

    Scorers; each is normalized into a Scorer

  • cases (Cases, Array, Enumerable) —

    Eval cases; wrapped into Braintrust::Eval::Cases if needed

  • experiment_id (String, nil) (defaults to: nil) —

    Experiment ID for logging

  • experiment_name (String, nil) (defaults to: nil) —

    Experiment name, included in span attributes

  • project_id (String, nil) (defaults to: nil) —

    Project ID

  • project_name (String, nil) (defaults to: nil) —

    Project name; required when resolving scorer slugs

  • state (Braintrust::State, nil) (defaults to: nil) —

    Authenticated API state; nil for local-only evals

  • tracer_provider (#tracer, nil) (defaults to: nil) —

    OpenTelemetry tracer provider; defaults to global provider

  • on_progress (Proc, nil) (defaults to: nil) —

    Callback invoked after each case completes, receiving a progress Hash

  • parent (Hash, nil) (defaults to: nil) —

    Parent span info with keys :object_type, :object_id, and optionally :generation

  • parameters (Hash, nil) (defaults to: nil) —

    Runtime parameters passed to task and scorers as a ‘parameters:` keyword argument

Returns:



60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/braintrust/eval/context.rb', line 60

def self.build(task:, scorers:, cases:, experiment_id: nil, experiment_name: nil,
  project_id: nil, project_name: nil, state: nil, tracer_provider: nil,
  on_progress: nil, parent: nil, parameters: nil)
  Factory.new(
    state: state, tracer_provider: tracer_provider,
    project_id: project_id, project_name: project_name
  ).build(
    task: task, scorers: scorers, cases: cases,
    experiment_id: experiment_id, experiment_name: experiment_name,
    on_progress: on_progress, parent: parent, parameters: parameters
  )
end