Class: Insika::Evals::CaseResult

Inherits:
Struct
  • Object
show all
Defined in:
lib/insika/evals/assertions.rb

Overview

The verdict for one golden case. judge (a Judge::Verdict) is attached AFTER the deterministic pass when the case has a rubric and a judge is configured; until then a rubric'd case is judge_pending? — it reads as "not fully evaluated", never a silent pass. A case passes only if the deterministic checks pass AND (there's no judge verdict OR it passed).

skipped (a reason, nil = it ran) is the THIRD outcome: the deployment lacks something the case declared it needs, so there was nothing to assert. It is never a pass and never a failure — a suite of 40 cases where 12 are skipped says something true, where 40 cases with 12 failures on capability grounds says nothing and gets ignored. pairwise (a Pairwise::Verdict) is attached when the case carries a reference: and a panel is configured. It DELIBERATELY does not enter pass?: "worse than the incumbent" is a judgement about a replacement decision, not a regression in the suite, and letting it fail a case would put an opinion about two conversations in the pre-merge gate.

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#agent ⇒ Object

Returns the value of attribute agent

Returns:

  • (Object) —

    the current value of agent



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def agent
  @agent
end

#checks ⇒ Object

Returns the value of attribute checks

Returns:

  • (Object) —

    the current value of checks



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def checks
  @checks
end

#error ⇒ Object

Returns the value of attribute error

Returns:

  • (Object) —

    the current value of error



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def error
  @error
end

#id ⇒ Object

Returns the value of attribute id

Returns:

  • (Object) —

    the current value of id



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def id
  @id
end

#judge ⇒ Object

Returns the value of attribute judge

Returns:

  • (Object) —

    the current value of judge



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def judge
  @judge
end

#pairwise ⇒ Object

Returns the value of attribute pairwise

Returns:

  • (Object) —

    the current value of pairwise



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def pairwise
  @pairwise
end

#rubric ⇒ Object

Returns the value of attribute rubric

Returns:

  • (Object) —

    the current value of rubric



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def rubric
  @rubric
end

#skipped ⇒ Object

Returns the value of attribute skipped

Returns:

  • (Object) —

    the current value of skipped



46
47
48
 
# File 'lib/insika/evals/assertions.rb', line 46

def skipped
  @skipped
end

Instance Method Details

#failures ⇒ Object 



50
51
52
 
# File 'lib/insika/evals/assertions.rb', line 50

def failures = checks.reject(&:pass)
# Has a rubric to score but no verdict yet (judge disabled / not run). A skipped
# case is not pending anything — nobody is going to judge a turn that never ran.

#judge_pending? ⇒ Boolean

Has a rubric to score but no verdict yet (judge disabled / not run). A skipped case is not pending anything — nobody is going to judge a turn that never ran.

Returns:

  • (Boolean) 


53
 
# File 'lib/insika/evals/assertions.rb', line 53

def judge_pending? = !skipped? && !rubric.to_s.strip.empty? && judge.nil?

#pass? ⇒ Boolean

Returns:

  • (Boolean) 


49
 
# File 'lib/insika/evals/assertions.rb', line 49

def pass? = !skipped? && error.nil? && checks.all?(&:pass) && (judge.nil? || judge.pass)

#skipped? ⇒ Boolean

Returns:

  • (Boolean) 


48
 
# File 'lib/insika/evals/assertions.rb', line 48

def skipped? = !skipped.nil?