Class: Langfuse::ExperimentRunner Private

Inherits:
Object
  • Object
show all
Defined in:
lib/langfuse/experiment_runner.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Orchestrates an experiment run: executes a task against each item, runs evaluators, persists scores, and links dataset run items.

This class is not intended to be instantiated directly. Use Client#run_experiment or DatasetClient#run_experiment instead.

rubocop:disable Metrics/ClassLength

Instance Method Summary collapse

Constructor Details

#initialize(client:, name:, items:, task:, evaluators: [], run_evaluators: [], metadata: nil, description: nil, run_name: nil) ⇒ ExperimentRunner

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

rubocop:disable Metrics/ParameterLists

Parameters:

  • client (Client) —

    Langfuse client for API calls

  • name (String) —

    experiment/run name

  • items (Array<DatasetItemClient, Hash>) —

    items to process

  • task (Proc) —

    callable receiving an item, returning output

  • evaluators (Array<Proc>) (defaults to: []) —

    item-level evaluators returning Langfuse::Evaluation or Array<Langfuse::Evaluation>

  • run_evaluators (Array<Proc>) (defaults to: []) —

    run-level evaluators receiving all item results and returning Langfuse::Evaluation or Array<Langfuse::Evaluation>

  • metadata (Hash, nil) (defaults to: nil) —

    metadata attached to each trace

  • description (String, nil) (defaults to: nil) —

    run description for dataset run items

  • run_name (String, nil) (defaults to: nil) —

    explicit run name (defaults to “name - timestamp”)



24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/langfuse/experiment_runner.rb', line 24

def initialize(client:, name:, items:, task:, evaluators: [], run_evaluators: [],
               metadata: nil, description: nil, run_name: nil)
  @client = client
  @name = name
  @items = items.map { |item| normalize_item(item) }
  @task = task
  @evaluators = evaluators
  @run_evaluators = run_evaluators
  @metadata = metadata || {}
  @description = description
  @run_name = run_name || "#{name} - #{Time.now.utc.iso8601}"
  @logger = Langfuse.configuration.logger
  @dataset_run_id = nil
  @dataset_id = nil
end

Instance Method Details

#execute ⇒ ExperimentResult

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns:



42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/langfuse/experiment_runner.rb', line 42

def execute
  item_results = @items.each_with_index.map { |item, index| process_item(item, index) }

  flush_all

  run_evals = execute_run_evaluators(item_results)
  flush_all if run_evals.any?

  ExperimentResult.new(
    name: @name,
    item_results: item_results,
    run_evaluations: run_evals,
    run_name: @run_name,
    description: @description,
    dataset_run_id: @dataset_run_id,
    dataset_run_url: build_dataset_run_url
  )
end