Module: Vangrail::Posterior

Defined in:
lib/vangrail/evidence.rb

Overview

Combines rail evidence into a posterior probability that the text is an attack.

The arithmetic is one line: odds after = odds before times every likelihood ratio. In bits it is addition, which is why the contributions of individual rails can be printed and read.

Three things make this more than a formality, and all three are things the published defences leave on the floor.

The prior is the deployment's, and it dominates. Detector papers evaluate on balanced corpora, where half the traffic is an attack; a documentation desk sees maybe one poisoned page in ten thousand. At that base rate a rail with a one percent false-alarm rate is wrong far more often than it is right when it fires, and no amount of detection rate fixes it. That is not a criticism of the rails: it is the arithmetic every operator inherits and almost none is shown.

Abstention is evidence of nothing, which is different from evidence against. A rail that was off, unreachable, or undecided contributes no term at all, and this gem is unusual in knowing which rails those were: certain? is exactly that fact, and here it finally has arithmetic to feed.

Correlated rails do not each get a vote. Three rails that fire on the same sentence for the same reason are one observation reported three times, and summing them is how naive Bayes talks itself into certainty. Rails measured to agree are grouped, and a group contributes once.

Constant Summary collapse

DEFAULT_CONFIDENCE =

Combines and returns [posterior, contributions].

observations maps a rail name to true (fired), false (ran and did not fire), or nil (did not run). The nils are the point. direct carries rails that computed their own log-likelihood ratio rather than answering yes or no. A rail that can say how sure it is should not be flattened to one bit on the way in, and nothing about the arithmetic changes: bits are bits, whoever produced them. Defensible by default. The point estimate is what a corpus happened to produce and it is unreadable at the edges: a rail that caught none of the published attacks and fired on none of eighteen thousand documents scores +7 bits on the point estimate, from two smoothing constants dividing each other, and -2.7 on the bound. The bound is the number that survives being measured against somebody else's corpus, so it is the one that runs.

0.95

Class Method Summary collapse

Class Method Details

.combine(prior:, observations:, evidence: EvidenceData::TABLE, confidence: DEFAULT_CONFIDENCE, direct: {}) ⇒ Object

Raises:

  • (ArgumentError) 


215
216
217
218
219
220
221
222
 
# File 'lib/vangrail/evidence.rb', line 215

def combine(prior:, observations:, evidence: EvidenceData::TABLE, confidence: DEFAULT_CONFIDENCE,
            direct: {})
  raise ArgumentError, 'prior must be strictly between 0 and 1' unless prior.positive? && prior < 1

  contributions = weigh(observations, evidence, confidence) + quantified(direct)
  total = contributions.sum { |c| c[:bits] }
  [from_odds(to_odds(prior) * (2**total)), contributions]
end

.false_alarm_needed(prior:, detection: 0.75, target: 0.5) ⇒ Object

The false-alarm rate a single rail would need to carry a block on its own.

Rearranged from the same identity: at base rate prior, one rail with detection detection reaches target only if it almost never fires on ordinary text. The answers come out in the region of one in ten thousand, which is below what any hand-built benign corpus can demonstrate: showing a rate that low needs tens of thousands of clean documents on which the rail stayed silent.

That is the practical case for combining rails rather than trusting one, and it is an argument about evidence rather than about taste.



282
283
284
 
# File 'lib/vangrail/evidence.rb', line 282

def false_alarm_needed(prior:, detection: 0.75, target: 0.5)
  detection / (to_odds(target) / to_odds(prior))
end

.from_odds(odds) ⇒ Object 



297
298
299
300
301
 
# File 'lib/vangrail/evidence.rb', line 297

def from_odds(odds)
  return 1.0 if odds.infinite?

  odds / (1 + odds)
end

.quantified(direct) ⇒ Object 



286
287
288
289
290
291
 
# File 'lib/vangrail/evidence.rb', line 286

def quantified(direct)
  direct.map do |rail, bits|
    { group: rail.to_s, rail: rail.to_s, fired: bits.positive?, bits: bits.to_f,
      spoke_for: [rail.to_s], quantified: true }
  end
end

.required_bits(prior:, target: 0.5) ⇒ Object

How many bits it takes to get from a base rate to a target confidence.

This is the number the whole design turns on, and it is worth being able to compute rather than assert. Reaching an even-money posterior from one attack in ten thousand takes about 13.3 bits, and no rail in this gem is worth half that, which is a statement about what a single detector can honestly justify rather than about these particular rails.



267
268
269
 
# File 'lib/vangrail/evidence.rb', line 267

def required_bits(prior:, target: 0.5)
  Math.log2(to_odds(target) / to_odds(prior))
end

.speak_for(group, members, confidence = nil) ⇒ Object 



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
 
# File 'lib/vangrail/evidence.rb', line 243

def speak_for(group, members, confidence = nil)
  fired = members.select { |m| m[:fired] }
  chosen = if fired.empty?
             members.max_by { |m| m[:entry].detection }
           else
             fired.max_by { |m| m[:entry].bits(true, confidence: confidence) }
           end

  {
    group: group,
    rail: chosen[:rail],
    fired: chosen[:fired],
    bits: chosen[:entry].bits(chosen[:fired], confidence: confidence),
    spoke_for: members.map { |m| m[:rail] },
  }
end

.to_odds(probability) ⇒ Object 



293
294
295
 
# File 'lib/vangrail/evidence.rb', line 293

def to_odds(probability)
  probability / (1 - probability)
end

.weigh(observations, evidence, confidence = nil) ⇒ Object

One term per group rather than one per rail.

Within a group, the firing rail with the most evidence speaks for the group; if none fired, the most sensitive member's silence speaks for it. Both rules pick the single most informative member, which is the conservative reading of a set of observations that are not independent.



230
231
232
233
234
235
236
237
238
239
240
241
 
# File 'lib/vangrail/evidence.rb', line 230

def weigh(observations, evidence, confidence = nil)
  seen = observations.filter_map do |rail, fired|
    next if fired.nil?

    entry = evidence[rail.to_s]
    next unless entry&.measured?

    { rail: rail.to_s, group: entry.group || rail.to_s, fired: fired, entry: entry }
  end

  seen.group_by { |o| o[:group] }.map { |group, members| speak_for(group, members, confidence) }
end