Module: Insika::Evals::GoldenLoader

Defined in:
lib/insika/evals/golden.rb

Overview

Loads + validates golden files. Fails LOUD on a malformed case — a silently dropped golden is a hole in the safety net.

Defined Under Namespace

Classes: InvalidGolden

Class Method Summary collapse

Class Method Details

.build(raw, source: "(inline)") ⇒ Object

hash (string keys) -> validated Golden. source is only for error messages.

Raises:



83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
 
# File 'lib/insika/evals/golden.rb', line 83

def build(raw, source: "(inline)")
  raise InvalidGolden, "#{source}: golden must be a mapping" unless raw.is_a?(Hash)

  id = presence(raw["id"]) || (raise InvalidGolden, "#{source}: 'id' is required")
  agent = presence(raw["agent"]) || (raise InvalidGolden, "#{source}: 'agent' is required (case '#{id}')")
  turns = normalize_turns(raw["turns"], id: id, source: source)
  expect = raw["expect"] || {}
  raise InvalidGolden, "#{source}: 'expect' must be a mapping (case '#{id}')" unless expect.is_a?(Hash)

  validate_policy!(expect["policy"], id: id, source: source)
  requires = raw["requires"] || {}
  unless requires.is_a?(Hash)
    raise InvalidGolden, "#{source}: 'requires' must be a mapping (case '#{id}')"
  end

  reference = normalize_reference(raw["reference"], id: id, source: source)

  Golden.new(id: id, agent: agent, turns: turns, expect: expect,
             requires: requires, reference: reference, source: source)
end

.load_dir(dir) ⇒ Object

Loads every .yml/.yaml under dir (recursive), sorted by path for a stable run order. -> [Golden].



71
72
73
 
# File 'lib/insika/evals/golden.rb', line 71

def load_dir(dir)
  Dir.glob(File.join(dir, "**", "*.{yml,yaml}")).sort.map { |f| load_file(f) }
end

.load_file(path) ⇒ Object 



75
76
77
78
79
80
 
# File 'lib/insika/evals/golden.rb', line 75

def load_file(path)
  raw = YAML.safe_load(File.read(path), permitted_classes: [], aliases: false) || {}
  build(raw, source: path)
rescue Psych::SyntaxError => e
  raise InvalidGolden, "#{path}: invalid YAML — #{e.message}"
end

.normalize_reference(raw, id:, source:) ⇒ Object

reference: { "source" => String?, "messages" => [{ "role" =>, "text" =>, "origin" => }] }. Absent -> {}, and the case simply has nothing to compare against. Malformed is REFUSED: a reference that half-loads would produce a pairwise verdict about a transcript nobody wrote.

Raises:



108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'lib/insika/evals/golden.rb', line 108

def normalize_reference(raw, id:, source:)
  return {} if raw.nil?
  raise InvalidGolden, "#{source}: 'reference' must be a mapping (case '#{id}')" unless raw.is_a?(Hash)

  messages = raw["messages"]
  unless messages.is_a?(Array) && !messages.empty?
    raise InvalidGolden, "#{source}: reference needs a non-empty 'messages' array (case '#{id}')"
  end

  { "source" => presence(raw["source"]),
    "messages" => messages.each_with_index.map { |m, i| reference_message(m, i, id: id, source: source) } }.compact
end

.normalize_turns(turns, id:, source:) ⇒ Object

turns: a non-empty array of { "user" => String }. Rejects anything else so a typo (e.g. users:) surfaces at load time, not as an empty replay.



154
155
156
157
158
159
160
161
162
163
164
 
# File 'lib/insika/evals/golden.rb', line 154

def normalize_turns(turns, id:, source:)
  unless turns.is_a?(Array) && !turns.empty?
    raise InvalidGolden, "#{source}: 'turns' must be a non-empty array (case '#{id}')"
  end

  turns.each_with_index.map do |t, i|
    user = t.is_a?(Hash) ? presence(t["user"]) : nil
    user || (raise InvalidGolden, "#{source}: turns[#{i}] needs a non-empty 'user' (case '#{id}')")
    { "user" => user }
  end
end

.presence(v) ⇒ Object 



166
167
168
169
 
# File 'lib/insika/evals/golden.rb', line 166

def presence(v)
  s = v.to_s.strip
  s.empty? ? nil : s
end

.reference_message(raw, index, id:, source:) ⇒ Object

Raises:



121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/insika/evals/golden.rb', line 121

def reference_message(raw, index, id:, source:)
  where = "#{source}: reference.messages[#{index}] (case '#{id}')"
  raise InvalidGolden, "#{where} must be a mapping" unless raw.is_a?(Hash)

  role = presence(raw["role"])
  raise InvalidGolden, "#{where} needs a 'role' of user or assistant" unless %w[user assistant].include?(role)

  text = presence(raw["text"]) || (raise InvalidGolden, "#{where} needs a non-empty 'text'")
  # The SAME closed vocabulary the engine stamps. A typo'd marker would
  # read as "absent" downstream, which is how a human turn gets scored as the
  # incumbent's model.
  origin = begin
    MessageOrigin.parse!(raw["origin"])
  rescue Insika::ValidationError => e
    raise InvalidGolden, "#{where}: #{e.message}"
  end
  { "role" => role, "text" => text }.merge(origin ? { "origin" => origin } : {})
end

.validate_policy!(value, id:, source:) ⇒ Object

A typo'd policy must not silently mean "no policy" — the case would go on passing while the rule it was written for stopped being checked. The Assertions constant is resolved at CALL time (this file loads first, and assertions.rb touches Safety::Detectors at load time).

Raises:



144
145
146
147
148
149
150
 
# File 'lib/insika/evals/golden.rb', line 144

def validate_policy!(value, id:, source:)
  name = presence(value)
  return if name.nil? || Assertions::POLICIES.key?(name)

  raise InvalidGolden, "#{source}: unknown policy #{name.inspect} (case '#{id}') — " \
                       "known: #{Assertions::POLICIES.keys.join(', ')}"
end