Module: Insika::Evals::GoldenLoader

Defined in:
lib/insika/evals/golden.rb

Overview

Loads + validates golden files. Fails LOUD on a malformed case — a silently dropped golden is a hole in the safety net.

Defined Under Namespace

Classes: InvalidGolden

Class Method Summary collapse

Class Method Details

.build(raw, source: "(inline)") ⇒ Object

hash (string keys) -> validated Golden. source is only for error messages.

Raises:



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/insika/evals/golden.rb', line 99

def build(raw, source: "(inline)")
  raise InvalidGolden, "#{source}: golden must be a mapping" unless raw.is_a?(Hash)

  id = presence(raw["id"]) || (raise InvalidGolden, "#{source}: 'id' is required")
  agent = presence(raw["agent"]) || (raise InvalidGolden, "#{source}: 'agent' is required (case '#{id}')")
  persona = normalize_persona(raw["persona"], id: id, source: source)
  if persona && !raw["turns"].nil?
    raise InvalidGolden, "#{source}: a case is ONE shape — 'turns' or 'persona', not both (case '#{id}')"
  end

  turns = persona ? [] : normalize_turns(raw["turns"], id: id, source: source)
  expect = raw["expect"] || {}
  raise InvalidGolden, "#{source}: 'expect' must be a mapping (case '#{id}')" unless expect.is_a?(Hash)

  validate_policy!(expect["policy"], id: id, source: source)
  requires = raw["requires"] || {}
  unless requires.is_a?(Hash)
    raise InvalidGolden, "#{source}: 'requires' must be a mapping (case '#{id}')"
  end

  reference = normalize_reference(raw["reference"], id: id, source: source)
  tenant = presence(raw["tenant"]) || "platform"

  Golden.new(id: id, agent: agent, turns: turns, expect: expect,
             requires: requires, reference: reference, source: source, persona: persona,
             tenant: tenant)
end

.load_dir(dir) ⇒ Object

Loads every .yml/.yaml under dir (recursive), sorted by path for a stable run order. -> [Golden].



87
88
89
 
# File 'lib/insika/evals/golden.rb', line 87

def load_dir(dir)
  Dir.glob(File.join(dir, "**", "*.{yml,yaml}")).sort.map { |f| load_file(f) }
end

.load_file(path) ⇒ Object 



91
92
93
94
95
96
 
# File 'lib/insika/evals/golden.rb', line 91

def load_file(path)
  raw = YAML.safe_load(File.read(path), permitted_classes: [], aliases: false) || {}
  build(raw, source: path)
rescue Psych::SyntaxError => e
  raise InvalidGolden, "#{path}: invalid YAML — #{e.message}"
end

.normalize_persona(raw, id:, source:) ⇒ Object

persona: is the alternative shape to turns:: the conversation is GENERATED, not replayed. Malformed is REFUSED — a persona without knows or max_turns would simulate nothing. The PersonaLoader already prefixes its messages with the source path; the case id is added ONCE here (the loader is shared by the persona-file CLI, which has no case shape).



133
134
135
136
137
138
139
 
# File 'lib/insika/evals/golden.rb', line 133

def normalize_persona(raw, id:, source:)
  return nil if raw.nil?

  PersonaLoader.build(raw, source: source)
rescue PersonaLoader::InvalidPersona => e
  raise InvalidGolden, "#{e.message} (case '#{id}')"
end

.normalize_reference(raw, id:, source:) ⇒ Object

reference: { "source" => String?, "messages" => [{ "role" =>, "text" =>, "origin" => }] }. Absent -> {}, and the case simply has nothing to compare against. Malformed is REFUSED: a reference that half-loads would produce a pairwise verdict about a transcript nobody wrote.

Raises:



145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/insika/evals/golden.rb', line 145

def normalize_reference(raw, id:, source:)
  return {} if raw.nil?
  raise InvalidGolden, "#{source}: 'reference' must be a mapping (case '#{id}')" unless raw.is_a?(Hash)

  messages = raw["messages"]
  unless messages.is_a?(Array) && !messages.empty?
    raise InvalidGolden, "#{source}: reference needs a non-empty 'messages' array (case '#{id}')"
  end

  { "source" => presence(raw["source"]),
    "messages" => messages.each_with_index.map { |m, i| reference_message(m, i, id: id, source: source) } }.compact
end

.normalize_turns(turns, id:, source:) ⇒ Object

turns: a non-empty array of { "user" => String }. Rejects anything else so a typo (e.g. users:) surfaces at load time, not as an empty replay.



191
192
193
194
195
196
197
198
199
200
201
 
# File 'lib/insika/evals/golden.rb', line 191

def normalize_turns(turns, id:, source:)
  unless turns.is_a?(Array) && !turns.empty?
    raise InvalidGolden, "#{source}: 'turns' must be a non-empty array (case '#{id}')"
  end

  turns.each_with_index.map do |t, i|
    user = t.is_a?(Hash) ? presence(t["user"]) : nil
    user || (raise InvalidGolden, "#{source}: turns[#{i}] needs a non-empty 'user' (case '#{id}')")
    { "user" => user }
  end
end

.presence(v) ⇒ Object 



203
204
205
206
 
# File 'lib/insika/evals/golden.rb', line 203

def presence(v)
  s = v.to_s.strip
  s.empty? ? nil : s
end

.reference_message(raw, index, id:, source:) ⇒ Object

Raises:



158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
 
# File 'lib/insika/evals/golden.rb', line 158

def reference_message(raw, index, id:, source:)
  where = "#{source}: reference.messages[#{index}] (case '#{id}')"
  raise InvalidGolden, "#{where} must be a mapping" unless raw.is_a?(Hash)

  role = presence(raw["role"])
  raise InvalidGolden, "#{where} needs a 'role' of user or assistant" unless %w[user assistant].include?(role)

  text = presence(raw["text"]) || (raise InvalidGolden, "#{where} needs a non-empty 'text'")
  # The SAME closed vocabulary the engine stamps. A typo'd marker would
  # read as "absent" downstream, which is how a human turn gets scored as the
  # incumbent's model.
  origin = begin
    MessageOrigin.parse!(raw["origin"])
  rescue Insika::ValidationError => e
    raise InvalidGolden, "#{where}: #{e.message}"
  end
  { "role" => role, "text" => text }.merge(origin ? { "origin" => origin } : {})
end

.validate_policy!(value, id:, source:) ⇒ Object

A typo'd policy must not silently mean "no policy" — the case would go on passing while the rule it was written for stopped being checked. The Assertions constant is resolved at CALL time (this file loads first, and assertions.rb touches Safety::Detectors at load time).

Raises:



181
182
183
184
185
186
187
 
# File 'lib/insika/evals/golden.rb', line 181

def validate_policy!(value, id:, source:)
  name = presence(value)
  return if name.nil? || Assertions::POLICIES.key?(name)

  raise InvalidGolden, "#{source}: unknown policy #{name.inspect} (case '#{id}') — " \
                       "known: #{Assertions::POLICIES.keys.join(', ')}"
end