Class: Insika::Evals::Golden

Inherits:
Struct
  • Object
show all
Defined in:
lib/insika/evals/golden.rb

Overview

A curated behavior case, loaded from a data file (evals/golden//*.yml). Data, not code — same spirit as tools-as-data. See evals/README.md for the format.

A case is ONE of two shapes: turns: (a scripted replay) or persona: (a conversation the Simulator GENERATES). A persona case is simulated? — the replay Runner skips it, and the Simulator drives it.

tenant (C3.1): which tenant authored this case — "platform" (the single-tenant default, like save_artifact's own binding_tenant) unless the case declares one. run_persona_eval uses it to keep a QA agent from ever running (or even seeing) another tenant's persona case in the same store.

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#agent ⇒ Object

Returns the value of attribute agent

Returns:

  • (Object) —

    the current value of agent



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def agent
  @agent
end

#expect ⇒ Object

Returns the value of attribute expect

Returns:

  • (Object) —

    the current value of expect



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def expect
  @expect
end

#id ⇒ Object

Returns the value of attribute id

Returns:

  • (Object) —

    the current value of id



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def id
  @id
end

#persona ⇒ Object

Returns the value of attribute persona

Returns:

  • (Object) —

    the current value of persona



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def persona
  @persona
end

#reference ⇒ Object

Returns the value of attribute reference

Returns:

  • (Object) —

    the current value of reference



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def reference
  @reference
end

#requires ⇒ Object

Returns the value of attribute requires

Returns:

  • (Object) —

    the current value of requires



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def requires
  @requires
end

#source ⇒ Object

Returns the value of attribute source

Returns:

  • (Object) —

    the current value of source



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def source
  @source
end

#tenant ⇒ Object

Returns the value of attribute tenant

Returns:

  • (Object) —

    the current value of tenant



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def tenant
  @tenant
end

#turns ⇒ Object

Returns the value of attribute turns

Returns:

  • (Object) —

    the current value of turns



23
24
25
 
# File 'lib/insika/evals/golden.rb', line 23

def turns
  @turns
end

Instance Method Details

#human_assisted? ⇒ Boolean

Did a PERSON type part of the reference half? After a handoff the operator's words are stored as role: assistant, and comparing a model to a human and calling it a win is a lie in both directions — so the pair is LABELLED and the report never prints the outcome without it.

Returns:

  • (Boolean) 


69
70
71
 
# File 'lib/insika/evals/golden.rb', line 69

def human_assisted?
  reference_messages.any? { |m| MessageOrigin.origin_of(m) == MessageOrigin::OPERATOR }
end

#min_score ⇒ Object 



75
 
# File 'lib/insika/evals/golden.rb', line 75

def min_score = expect["min_score"]

#must_not ⇒ Object

Names of the negative assertions to run (e.g. "pii_leak", "tool_error").



45
 
# File 'lib/insika/evals/golden.rb', line 45

def must_not = Array(expect["must_not"]).map(&:to_s)

#opens_with ⇒ Object 



32
 
# File 'lib/insika/evals/golden.rb', line 32

def opens_with = persona ? persona.opens_with : user_turns.first

#policy ⇒ Object

How much the agent should ask before acting. nil = the store has no opinion and only the rubric decides.



49
 
# File 'lib/insika/evals/golden.rb', line 49

def policy = GoldenLoader.presence(expect["policy"])

#reference? ⇒ Boolean

Returns:

  • (Boolean) 


63
 
# File 'lib/insika/evals/golden.rb', line 63

def reference? = !reference_messages.empty?

#reference_messages ⇒ Object

THE INCUMBENT'S CONVERSATION for the same opening — the other half of a pairwise comparison. Data in the case, not a store read: the eval is a client, and a pair that lives in one reviewable file cannot go stale against a database nobody looked at.



61
 
# File 'lib/insika/evals/golden.rb', line 61

def reference_messages = Array(reference["messages"])

#reference_source ⇒ Object 



62
 
# File 'lib/insika/evals/golden.rb', line 62

def reference_source = GoldenLoader.presence(reference["source"])

#required_capabilities ⇒ Object 



54
 
# File 'lib/insika/evals/golden.rb', line 54

def required_capabilities = Array(requires["capabilities"]).map(&:to_s)

#required_tools ⇒ Object

What the DEPLOYMENT must have for this case to mean anything. Empty = runs everywhere.



53
 
# File 'lib/insika/evals/golden.rb', line 53

def required_tools = Array(requires["tools"]).map(&:to_s)

#requirements? ⇒ Boolean

Returns:

  • (Boolean) 


55
 
# File 'lib/insika/evals/golden.rb', line 55

def requirements? = !(required_tools + required_capabilities).empty?

#rubric ⇒ Object

LLM-judge rubric + threshold (consumed in — deferred here).



74
 
# File 'lib/insika/evals/golden.rb', line 74

def rubric = expect["rubric"]

#simulated? ⇒ Boolean

The simulated customer. nil for a scripted case.

Returns:

  • (Boolean) 


30
 
# File 'lib/insika/evals/golden.rb', line 30

def simulated? = !persona.nil?

#tools_called ⇒ Object

Tool refs the case expects; a trailing "?" marks OPTIONAL (never fails). -> [{ name:, optional: }]



36
37
38
39
40
41
42
 
# File 'lib/insika/evals/golden.rb', line 36

def tools_called
  Array(expect["tools_called"]).map do |ref|
    s = ref.to_s
    optional = s.end_with?("?")
    { name: optional ? s[0..-2] : s, optional: optional }
  end
end

#user_turns ⇒ Object

The user messages to replay, in order. Empty for a persona case: a generated conversation has no scripted turns.



27
 
# File 'lib/insika/evals/golden.rb', line 27

def user_turns = turns.map { |t| t["user"] }