Class: Ask::Agent::Evaluator

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/agent/evaluator.rb

Defined Under Namespace

Classes: Dimension, Result

Constant Summary collapse

DEFAULT_DIMENSIONS =

Default rubric borrowed from the course's evaluator-rubric template.

[
  Dimension.new(name: "correctness",  description: "Does the output match the requested goal?",                               weight: 3),
  Dimension.new(name: "completeness", description: "Are all aspects of the goal addressed?",                                  weight: 2),
  Dimension.new(name: "verification", description: "Is there evidence that the output actually works?",                        weight: 2),
  Dimension.new(name: "scope",        description: "Did it stay within the defined boundaries without overreaching?",          weight: 1),
  Dimension.new(name: "clarity",      description: "Is the output clear, well-structured, and maintainable?",                  weight: 1),
].freeze
MAX_EVAL_RETRIES =

How many times the evaluator may retry on a malformed response.

2

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(model:, rubric: DEFAULT_DIMENSIONS) ⇒ Evaluator

Returns a new instance of Evaluator.

Parameters:

  • model (String) —

    the model id to use for evaluation (should differ from the generator's model)

  • rubric (Array<Dimension>) (defaults to: DEFAULT_DIMENSIONS) —

    the rubric dimensions to evaluate against



42
43
44
45
 
# File 'lib/ask/agent/evaluator.rb', line 42

def initialize(model:, rubric: DEFAULT_DIMENSIONS)
  @model = model
  @rubric = rubric
end

Instance Attribute Details

#model ⇒ Object (readonly)

Returns the value of attribute model.



38
39
40
 
# File 'lib/ask/agent/evaluator.rb', line 38

def model
  @model
end

#rubric ⇒ Object (readonly)

Returns the value of attribute rubric.



38
39
40
 
# File 'lib/ask/agent/evaluator.rb', line 38

def rubric
  @rubric
end

Instance Method Details

#evaluate(goal:, response:, event_emitter: nil) ⇒ Result

Evaluate a response against a goal.

Parameters:

  • goal (String) —

    what the generator was asked to do

  • response (String) —

    what the generator produced

  • event_emitter (#emit, nil) (defaults to: nil) —

    optional event emitter for streaming evaluation

Returns:

  • (Result) —

    structured evaluation result



53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/ask/agent/evaluator.rb', line 53

def evaluate(goal:, response:, event_emitter: nil)
  event_emitter&.emit(Events::EvaluationStart.new(dimensions: @rubric.map(&:name)))

  chat = build_chat
  chat.with_instructions(evaluation_prompt(goal))

  accumulated = +""
  chat.ask(response.to_s) do |chunk|
    if chunk.content.to_s.strip.length > 0
      accumulated << chunk.content.to_s
      event_emitter&.emit(Events::EvaluationDelta.new(content: chunk.content.to_s))
    end
  end

  result = parse_result(accumulated)
  event_emitter&.emit(Events::EvaluationEnd.new(
    decision: result.decision,
    feedback: result.feedback,
    scores: result.scores,
    evidence: result.evidence
  ))

  result
end