Class: Decompound::Model

Inherits:
Object
  • Object
show all
Defined in:
lib/decompound/model.rb

Overview

Reads the binary model format (decompound-binary-v1):

magic "DCPDv1\n"
count           uint32 LE
bucket_count    uint32 LE
buckets         bucket_count x (key prefix, NUL-padded to 3 bytes + uint32 LE first index)
offsets         (count + 1) x uint32 LE, byte offsets into the keys blob
probabilities   count x 3 bytes (prefix, infix, suffix), 255 = missing
keys            concatenated UTF-8 n-grams, sorted bytewise

The file is held as a single string and searched in place, so loaded memory stays near file size instead of materializing millions of Ruby strings and hash entries. The buckets narrow each binary search to the keys sharing the query's first three bytes.

Constant Summary collapse

MAGIC = 
"DCPDv1\n"
PREFIX = 
0
INFIX = 
1
SUFFIX = 
2
MISSING = 
255
SCALE = 
254.0

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(data) ⇒ Model

Returns a new instance of Model.

Raises:

  • (ArgumentError) 


31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/decompound/model.rb', line 31

def initialize(data)
  raise ArgumentError, "not a decompound binary model" unless data.byteslice(0, MAGIC.bytesize) == MAGIC

  @data = data.freeze
  @count, bucket_count = data.byteslice(MAGIC.bytesize, 8).unpack("VV")
  buckets_at = MAGIC.bytesize + 8
  @offsets_at = buckets_at + (bucket_count * 7)
  @probabilities_at = @offsets_at + ((@count + 1) * 4)
  @keys_at = @probabilities_at + (@count * 3)
  @buckets = read_buckets(data, buckets_at, bucket_count)
end

Class Method Details

.load(path) ⇒ Object 



27
28
29
 
# File 'lib/decompound/model.rb', line 27

def self.load(path)
  new(File.binread(path))
end

Instance Method Details

#probability(ngram, position, default) ⇒ Object 



43
44
45
46
47
48
49
 
# File 'lib/decompound/model.rb', line 43

def probability(ngram, position, default)
  index = find(ngram.b)
  return default unless index

  code = @data.getbyte(@probabilities_at + (index * 3) + position)
  (code == MISSING) ? default : code / SCALE
end