Class: Vangrail::LinearModel

Inherits:
Object
  • Object
show all
Defined in:
lib/vangrail/linear_model.rb

Overview

A linear classifier over hashed n-grams, loaded from a file somebody fitted.

No weights ship with this gem, and that is the finding rather than an omission. The classifier that does ship, Rails::Bayes, was fitted on 48 hand-written clauses and catches 15 of 48 held out; the same architecture fitted on 15,140 real prompts catches three quarters of them. The difference is the corpus, and the corpus has to be the deployment's, because a model fitted on somebody else's traffic is the thing this repository spent a long time measuring the cost of.

Weights do not compress into a readable table either. Pruning the fitted model to its 20,000 largest weights costs 26 points of detection, because the signal is spread across two hundred thousand of them rather than concentrated in a vocabulary anyone could read. So the shipped artifact is the trainer and the reader; the model is a file a deployment generates and keeps.

ruby script/train_linear.rb --emit model.json
GUARDRAILS_LINEAR_MODEL=model.json GUARDRAILS_RAILS=input,linear

Features live here rather than in the trainer, so that fitting and scoring cannot drift apart. A classifier whose training features differ from its serving features by one stemmer revision is a classifier that scores well in every test and badly in production, and nothing about the failure looks like a bug.

Constant Summary collapse

LIMIT =

A four-thousand character prefix, hashed into a fixed table. Character four-grams are sampled every STRIDE characters. The bucket count and the stride are written into the file; LIMIT stays a process constant. Change the stride and the character-gram indices move.

4000
BUCKETS = 
2**18
STRIDE = 
2
MAX_BUCKETS =

A hostile file names its own table size. Array.new of that number is the allocation, so the bound has to sit in front of it.

2**20

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(weights:, bias: 0.0, buckets: BUCKETS, stride: STRIDE, threshold: nil, trained_on: nil) ⇒ LinearModel

Returns a new instance of LinearModel.

Raises:

  • (ArgumentError) 


65
66
67
68
69
70
71
72
73
74
 
# File 'lib/vangrail/linear_model.rb', line 65

def initialize(weights:, bias: 0.0, buckets: BUCKETS, stride: STRIDE, threshold: nil, trained_on: nil)
  raise ArgumentError, "weights.size (#{weights.size}) != buckets (#{buckets})" unless weights.size == buckets

  @weights = weights
  @bias = bias
  @buckets = buckets
  @stride = stride
  @threshold = threshold
  @trained_on = trained_on
end

Instance Attribute Details

#bias ⇒ Object (readonly)

Returns the value of attribute bias.



45
46
47
 
# File 'lib/vangrail/linear_model.rb', line 45

def bias
  @bias
end

#buckets ⇒ Object (readonly)

Returns the value of attribute buckets.



45
46
47
 
# File 'lib/vangrail/linear_model.rb', line 45

def buckets
  @buckets
end

#stride ⇒ Object (readonly)

Returns the value of attribute stride.



45
46
47
 
# File 'lib/vangrail/linear_model.rb', line 45

def stride
  @stride
end

#threshold ⇒ Object (readonly)

Returns the value of attribute threshold.



45
46
47
 
# File 'lib/vangrail/linear_model.rb', line 45

def threshold
  @threshold
end

#trained_on ⇒ Object (readonly)

Returns the value of attribute trained_on.



45
46
47
 
# File 'lib/vangrail/linear_model.rb', line 45

def trained_on
  @trained_on
end

Class Method Details

.bucket(feature, buckets = BUCKETS) ⇒ Object

FNV-1a rather than String#hash, which is seeded per process: a model whose feature indices move between runs cannot be saved, and the failure would look like a classifier that trained perfectly and predicts at random.



89
90
91
92
93
 
# File 'lib/vangrail/linear_model.rb', line 89

def self.bucket(feature, buckets = BUCKETS)
  hash = 2_166_136_261
  feature.each_byte { |byte| hash = ((hash ^ byte) * 16_777_619) & 0xFFFFFFFF }
  hash % buckets
end

.features(text, buckets = BUCKETS, stride = STRIDE) ⇒ Object

Word stems, adjacent stem pairs, and character four-grams taken every stride characters, counted and capped. The cap is what stops a page repeating one word from outvoting a page that says something. Train calls this with the process STRIDE; score calls it with the stride the file named, so the two cannot silently disagree.



100
101
102
103
 
# File 'lib/vangrail/linear_model.rb', line 100

def self.features(text, buckets = BUCKETS, stride = STRIDE)
  _body, words, normalised = prepared(text)
  features_from(words, normalised, buckets, stride)
end

.features_from(words, normalised, buckets, stride) ⇒ Object 



132
133
134
135
136
137
138
139
140
141
 
# File 'lib/vangrail/linear_model.rb', line 132

def self.features_from(words, normalised, buckets, stride)
  grams = words + words.each_cons(2).map { |pair| pair.join(' ') }
  chars = if normalised.length > 4
            (0..(normalised.length - 4)).step(stride).map { |i| "c:#{normalised[i, 4]}" }
          else
            []
          end
  (grams + chars).tally.transform_values { |count| [count, 3].min }
                 .each_with_object(Hash.new(0)) { |(feature, count), acc| acc[bucket(feature, buckets)] += count }
end

.load(path) ⇒ Object

Raises:

  • (ArgumentError) 


47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/vangrail/linear_model.rb', line 47

def self.load(path)
  data = JSON.parse(File.read(path))
  buckets = bounded_integer(data['buckets'], name: 'buckets', default: BUCKETS, max: MAX_BUCKETS)
  # Older files have no stride field; they were trained at 2.
  stride = bounded_integer(data['stride'], name: 'stride', default: 2, max: LIMIT)
  weights = Array.new(buckets, 0.0)
  data.fetch('weights').each do |index, value|
    i = index.to_i
    raise ArgumentError, "weight index #{i} is outside #{buckets} buckets" if i.negative? || i >= buckets

    weights[i] = value
  end
  raise ArgumentError, "loaded #{weights.size} weights for #{buckets} buckets" unless weights.size == buckets

  new(weights: weights, bias: data['bias'].to_f, buckets: buckets, stride: stride,
      threshold: data['threshold'], trained_on: data['trained_on'])
end

.prepared(text) ⇒ Object 



105
106
107
108
 
# File 'lib/vangrail/linear_model.rb', line 105

def self.prepared(text)
  body = text.to_s[0, LIMIT]
  [body, NLP.words(body).map { |word| NLP.stem(word) }, NLP.normalize(body)]
end

Instance Method Details

#ruby_score(text) ⇒ Object 



121
122
123
124
 
# File 'lib/vangrail/linear_model.rb', line 121

def ruby_score(text)
  _body, words, normalised = self.class.prepared(text)
  score_ruby(words, normalised)
end

#score(text) ⇒ Object

The log-odds the model assigns, positive towards attack. Stemming and Unicode folding stay in Ruby; the hashed bag and the dot product are the native kernel when vangrail-native is loaded.



113
114
115
116
117
118
119
 
# File 'lib/vangrail/linear_model.rb', line 113

def score(text)
  _body, words, normalised = self.class.prepared(text)
  table = native_table
  return table.score(bias, buckets, stride, words, normalised) if table

  score_ruby(words, normalised)
end

#to_h ⇒ Object 



150
151
152
153
154
 
# File 'lib/vangrail/linear_model.rb', line 150

def to_h
  { 'buckets' => buckets, 'stride' => stride, 'bias' => bias, 'threshold' => threshold,
    'trained_on' => trained_on,
    'weights' => @weights.each_with_index.filter_map { |value, i| [i.to_s, value] unless value.zero? }.to_h }
end