Module: Perfgate::Statistics::MannWhitneyU

Defined in:
lib/perfgate/statistics/mann_whitney_u.rb

Overview

One-sided Mann-Whitney U test: estimates how likely it is that the candidate distribution is drawn from a "worse" (larger-valued) population than the baseline distribution, without assuming normality (spec section 16.3). Uses a normal approximation with a tie correction, adapted from the Milestone 0 spike (spikes/regression_injection.rb) that validated this approach against a seeded +20% duration regression.

Baseline never surfaces this p-value directly to users (spec 16.3: "do not expose p-values alone as user-facing proof") -- it is combined with a practical-significance threshold by Comparison::MetricDecision.

Class Method Summary collapse

Class Method Details

.assign_tie_group(sorted, ranked, start_index) ⇒ Object

Finds the run of tied values starting at start_index, assigns them all the same averaged rank, and returns the index just past the run.



70
71
72
73
74
75
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 70

def assign_tie_group(sorted, ranked, start_index)
  end_index = tie_group_end(sorted, start_index)
  average_rank = ((start_index + 1) + (end_index + 1)) / 2.0
  (start_index..end_index).each { |k| ranked[k] = sorted[k] + [average_rank] }
  end_index + 1
end

.candidate_rank_sum(baseline, candidate) ⇒ Object 



50
51
52
53
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 50

def candidate_rank_sum(baseline, candidate)
  ranks = rank(baseline.map { |v| [v, :baseline] } + candidate.map { |v| [v, :candidate] })
  ranks.each_index.sum { |i| ranks[i][1] == :candidate ? ranks[i][2] : 0 }
end

.one_sided_p(baseline, candidate) ⇒ Object

Returns p, the probability of observing rank sums this extreme (or more) under the null hypothesis of no difference, tested against the one-sided alternative that candidate tends to be larger than baseline. Smaller p is stronger evidence the candidate is worse.



25
26
27
28
29
30
31
32
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 25

def one_sided_p(baseline, candidate)
  return 1.0 if baseline.empty? || candidate.empty?

  z = z_score(baseline, candidate)
  return 0.5 if z.nil?

  0.5 * Math.erfc(z / Math.sqrt(2))
end

.rank(pairs) ⇒ Object

Assigns tied (averaged) ranks to a list of [value, label] pairs, returning [value, label, rank] triples sorted by value.



57
58
59
60
61
62
63
64
65
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 57

def rank(pairs)
  sorted = pairs.sort_by { |value, _label| value }
  ranked = Array.new(sorted.size)

  index = 0
  index = assign_tie_group(sorted, ranked, index) while index < sorted.size

  ranked
end

.standard_deviation_u(baseline_size, candidate_size) ⇒ Object 



46
47
48
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 46

def standard_deviation_u(baseline_size, candidate_size)
  Math.sqrt(baseline_size * candidate_size * (baseline_size + candidate_size + 1) / 12.0)
end

.tie_group_end(sorted, start_index) ⇒ Object 



77
78
79
80
81
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 77

def tie_group_end(sorted, start_index)
  end_index = start_index
  end_index += 1 while end_index + 1 < sorted.size && sorted[end_index + 1][0] == sorted[start_index][0]
  end_index
end

.z_score(baseline, candidate) ⇒ Object

Standardized U statistic for the candidate sample, or nil when the null distribution has zero variance (e.g. every sample tied).



36
37
38
39
40
41
42
43
44
 
# File 'lib/perfgate/statistics/mann_whitney_u.rb', line 36

def z_score(baseline, candidate)
  n1 = baseline.size
  n2 = candidate.size
  u_candidate = candidate_rank_sum(baseline, candidate) - (n2 * (n2 + 1) / 2.0)
  std_u = standard_deviation_u(n1, n2)
  return nil if std_u.zero?

  (u_candidate - (n1 * n2 / 2.0)) / std_u
end