Class: Insika::Evals::Golden

Inherits:
Struct
  • Object
show all
Defined in:
lib/insika/evals/golden.rb

Overview

A curated behavior case, loaded from a data file (evals/golden//*.yml). Data, not code — same spirit as tools-as-data. See evals/README.md for the format.

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#agent ⇒ Object

Returns the value of attribute agent

Returns:

  • (Object) —

    the current value of agent



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def agent
  @agent
end

#expect ⇒ Object

Returns the value of attribute expect

Returns:

  • (Object) —

    the current value of expect



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def expect
  @expect
end

#id ⇒ Object

Returns the value of attribute id

Returns:

  • (Object) —

    the current value of id



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def id
  @id
end

#reference ⇒ Object

Returns the value of attribute reference

Returns:

  • (Object) —

    the current value of reference



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def reference
  @reference
end

#requires ⇒ Object

Returns the value of attribute requires

Returns:

  • (Object) —

    the current value of requires



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def requires
  @requires
end

#source ⇒ Object

Returns the value of attribute source

Returns:

  • (Object) —

    the current value of source



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def source
  @source
end

#turns ⇒ Object

Returns the value of attribute turns

Returns:

  • (Object) —

    the current value of turns



14
15
16
 
# File 'lib/insika/evals/golden.rb', line 14

def turns
  @turns
end

Instance Method Details

#human_assisted? ⇒ Boolean

Did a PERSON type part of the reference half? After a handoff the operator's words are stored as role: assistant (P23a), and comparing a model to a human and calling it a win is a lie in both directions — so the pair is LABELLED and the report never prints the outcome without it.

Returns:

  • (Boolean) 


53
54
55
 
# File 'lib/insika/evals/golden.rb', line 53

def human_assisted?
  reference_messages.any? { |m| MessageOrigin.origin_of(m) == MessageOrigin::OPERATOR }
end

#min_score ⇒ Object 



59
 
# File 'lib/insika/evals/golden.rb', line 59

def min_score = expect["min_score"]

#must_not ⇒ Object

Names of the negative assertions to run (e.g. "pii_leak", "tool_error").



29
 
# File 'lib/insika/evals/golden.rb', line 29

def must_not = Array(expect["must_not"]).map(&:to_s)

#policy ⇒ Object

How much the agent should ask before acting (RFC-0014 §3.3). nil = the store has no opinion and only the rubric decides.



33
 
# File 'lib/insika/evals/golden.rb', line 33

def policy = GoldenLoader.presence(expect["policy"])

#reference? ⇒ Boolean

Returns:

  • (Boolean) 


47
 
# File 'lib/insika/evals/golden.rb', line 47

def reference? = !reference_messages.empty?

#reference_messages ⇒ Object

THE INCUMBENT'S CONVERSATION for the same opening (RFC-0014 §3.4) — the other half of a pairwise comparison. Data in the case, not a store read: the eval is a client, and a pair that lives in one reviewable file cannot go stale against a database nobody looked at.



45
 
# File 'lib/insika/evals/golden.rb', line 45

def reference_messages = Array(reference["messages"])

#reference_source ⇒ Object 



46
 
# File 'lib/insika/evals/golden.rb', line 46

def reference_source = GoldenLoader.presence(reference["source"])

#required_capabilities ⇒ Object 



38
 
# File 'lib/insika/evals/golden.rb', line 38

def required_capabilities = Array(requires["capabilities"]).map(&:to_s)

#required_tools ⇒ Object

What the DEPLOYMENT must have for this case to mean anything (RFC-0014 §3.2). Empty = runs everywhere.



37
 
# File 'lib/insika/evals/golden.rb', line 37

def required_tools = Array(requires["tools"]).map(&:to_s)

#requirements? ⇒ Boolean

Returns:

  • (Boolean) 


39
 
# File 'lib/insika/evals/golden.rb', line 39

def requirements? = !(required_tools + required_capabilities).empty?

#rubric ⇒ Object

LLM-judge rubric + threshold (consumed in Fase B — deferred here).



58
 
# File 'lib/insika/evals/golden.rb', line 58

def rubric = expect["rubric"]

#tools_called ⇒ Object

Tool refs the case expects; a trailing "?" marks OPTIONAL (never fails). -> [{ name:, optional: }]



20
21
22
23
24
25
26
 
# File 'lib/insika/evals/golden.rb', line 20

def tools_called
  Array(expect["tools_called"]).map do |ref|
    s = ref.to_s
    optional = s.end_with?("?")
    { name: optional ? s[0..-2] : s, optional: optional }
  end
end

#user_turns ⇒ Object

The user messages to replay, in order.



16
 
# File 'lib/insika/evals/golden.rb', line 16

def user_turns = turns.map { |t| t["user"] }