Module: LLMExperiment::Stats

Defined in:
lib/llm_experiment/stats.rb

Overview

Exact, not approximate. With five tasks per cell the normal approximation to the Mann-Whitney U is not trustworthy, and the whole permutation set is small enough to enumerate: C(10,5) is 252. So the p-values here are counted, not estimated.

Class Method Summary collapse

Class Method Details

.exact_mann_whitney(a, b) ⇒ Object

Exact two-sided p: enumerate every way to split the pooled values into two samples of the observed sizes and count the splits at least as extreme.



44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/llm_experiment/stats.rb', line 44

def exact_mann_whitney(a, b)
  n1 = a.size
  n2 = b.size
  return { u: nil, p: nil, note: "empty sample" } if n1.zero? || n2.zero?

  total = n1 + n2
  combos = (1..total).reduce(1, :*) / ((1..n1).reduce(1, :*) * (1..n2).reduce(1, :*))
  return { u: u_statistic(a, b), p: nil, note: "#{combos} permutations; too many to enumerate" } if combos > 200_000

  pooled = a + b
  observed = u_statistic(a, b)
  mean_u = n1 * n2 / 2.0
  observed_deviation = (observed - mean_u).abs

  at_least_as_extreme = 0
  (0...total).to_a.combination(n1).each do |idx|
    set = idx.to_h { |i| [i, true] }
    left = idx.map { |i| pooled[i] }
    right = (0...total).reject { |i| set[i] }.map { |i| pooled[i] }
    at_least_as_extreme += 1 if (u_statistic(left, right) - mean_u).abs >= observed_deviation - 1e-9
  end

  { u: observed, p: at_least_as_extreme.to_f / combos, permutations: combos }
end

.median(values) ⇒ Object 



11
12
13
14
15
16
17
 
# File 'lib/llm_experiment/stats.rb', line 11

def median(values)
  return nil if values.empty?

  sorted = values.sort
  mid = sorted.size / 2
  sorted.size.odd? ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2.0
end

.ranks(values) ⇒ Object

Mid-ranks, so ties do not silently inflate the statistic.



20
21
22
23
24
25
26
27
28
29
30
31
32
 
# File 'lib/llm_experiment/stats.rb', line 20

def ranks(values)
  indexed = values.each_with_index.sort_by { |v, _| v }
  result = Array.new(values.size)
  i = 0
  while i < indexed.size
    j = i
    j += 1 while j + 1 < indexed.size && indexed[j + 1][0] == indexed[i][0]
    mid = ((i + 1) + (j + 1)) / 2.0
    (i..j).each { |k| result[indexed[k][1]] = mid }
    i = j + 1
  end
  result
end

.u_statistic(a, b) ⇒ Object

U for sample a against sample b.



35
36
37
38
39
40
 
# File 'lib/llm_experiment/stats.rb', line 35

def u_statistic(a, b)
  all = a + b
  r = ranks(all)
  rank_sum_a = r.first(a.size).sum
  rank_sum_a - (a.size * (a.size + 1) / 2.0)
end