Class: Ask::Eval::Dataset

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/eval/dataset.rb

Overview

A pinned set of fixed inputs for repeatable agent evaluation.

Like Rails fixtures, a dataset gives you a deterministic baseline — but the items are questions (prompts/tasks fed to a live agent), not database records, and the point is running the same items against multiple variants (a changed prompt, a different model) and comparing the results.

Examples:

dataset = Ask::Eval::Dataset.new("support-cases")
dataset.add(input: "The API returns 401 on stale tokens — how do I fix my auth flow?", tags: ["auth"])
dataset.add(input: "Summarize this thread and suggest next steps", expected: "A summary plus actions")
dataset.save("support-cases.json")

reloaded = Ask::Eval::Dataset.load("support-cases.json")

Defined Under Namespace

Classes: Item

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name, items: []) ⇒ Dataset

Returns a new instance of Dataset.

Parameters:

  • name (String) —

    dataset name (used in reports)

  • items (Array<Hash>) (defaults to: []) —

    initial items with symbol or string keys (expected:, context:, tags:, metadata:)



32
33
34
35
36
 
# File 'lib/ask/eval/dataset.rb', line 32

def initialize(name, items: [])
  @name = name.to_s
  @items = []
  items.each { |item| add(**symbolize(item)) }
end

Instance Attribute Details

#name ⇒ String (readonly)

Returns the dataset name.

Returns:

  • (String) —

    the dataset name



39
40
41
 
# File 'lib/ask/eval/dataset.rb', line 39

def name
  @name
end

Class Method Details

.load(path) ⇒ Ask::Eval::Dataset

Load a dataset from a JSON file written by #save (or any compatible items: shape).

Parameters:

  • path (String)

Returns:



103
104
105
106
 
# File 'lib/ask/eval/dataset.rb', line 103

def self.load(path)
  data = JSON.parse(File.read(path))
  new(data["name"] || File.basename(path, ".json"), items: data["items"] || [])
end

Instance Method Details

#add(input:, id: nil, expected: nil, context: nil, tags: [], metadata: {}) ⇒ Item

Add an item.

Parameters:

  • input (String) —

    the prompt/task fed to the agent (required)

  • expected (String, nil) (defaults to: nil) —

    expected/reference output

  • context (String, nil) (defaults to: nil) —

    source context

  • tags (Array<String>) (defaults to: []) —

    labels for filtering

  • metadata (Hash) (defaults to: {}) —

    free-form

Returns:

Raises:

  • (ArgumentError) —

    on empty input



50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/ask/eval/dataset.rb', line 50

def add(input:, id: nil, expected: nil, context: nil, tags: [], metadata: {})
  raise ArgumentError, "input is required" if input.to_s.strip.empty?

  item = Item.new(
    id: id || "item_#{@items.size + 1}",
    input: input.to_s,
    expected: expected,
    context: context,
    tags: Array(tags),
    metadata: metadata
  )
  @items << item
  item
end

#each(&block) ⇒ Object 



71
 
# File 'lib/ask/eval/dataset.rb', line 71

def each(&block) = @items.each(&block)

#experiment(runner:, scorer: nil) ⇒ Ask::Eval::Experiment

Create an experiment over this dataset (see Experiment).

Parameters:

  • runner (Proc) —

    called with the item's input; returns the agent's output string

  • scorer (Proc, nil) (defaults to: nil) —

    called with (input:, output:, expected:); returns a score (0..1)

Returns:



80
81
82
 
# File 'lib/ask/eval/dataset.rb', line 80

def experiment(runner:, scorer: nil)
  Experiment.new(self, runner: runner, scorer: scorer)
end

#items ⇒ Array<Item>

Returns all items, in order.

Returns:

  • (Array<Item>) —

    all items, in order



66
 
# File 'lib/ask/eval/dataset.rb', line 66

def items = @items.dup

#save(path) ⇒ void

This method returns an undefined value.

Write the dataset to a JSON file.

Parameters:

  • path (String) 


93
94
95
96
 
# File 'lib/ask/eval/dataset.rb', line 93

def save(path)
  File.write(path, JSON.pretty_generate(to_h))
  nil
end

#size ⇒ Integer

Returns number of items.

Returns:

  • (Integer) —

    number of items



69
 
# File 'lib/ask/eval/dataset.rb', line 69

def size = @items.size

#to_h ⇒ Hash

Returns serialized form (for #save).

Returns:

  • (Hash) —

    serialized form (for #save)



85
86
87
 
# File 'lib/ask/eval/dataset.rb', line 85

def to_h
  { name: @name, items: @items.map(&:to_h) }
end