Module: MultiCompress::Database

Defined in:
lib/multi_compress/database.rb

Overview

Long-term, SQL-readable envelopes. MCDB1 is dictionary-free. MCDB2 is an opt-in, dictionary-required format for one homogeneous database column.

Defined Under Namespace

Classes: Dictionary

Constant Summary collapse

MAGIC = 
"MCDB".b
VERSION_V1 = 
1
VERSION_V2 = 
2
VERSION =

MCDB1 compatibility alias

VERSION_V1
CODEC_ZSTD = 
1
FLAGS_NONE = 
0
FLAGS_V1 = 
FLAGS_NONE
V1_HEADER_SIZE =

4 magic + 1 version + 1 codec + 1 flags + 8 size + 4 crc32

19
V2_HEADER_SIZE =

MCDB1 header + 8 byte application dictionary_ref

27
HEADER_SIZE = 
V1_HEADER_SIZE
ORIGINAL_SIZE_OFFSET = 
7
CRC_OFFSET = 
15
DICTIONARY_REF_OFFSET = 
19
MAX_OUTPUT = 
16 * 1024 * 1024
MAX_DICTIONARY_BYTES = 
256 * 1024
MAX_DICTIONARY_REF = 
(2**63) - 1
MAX_ENVELOPE_V1 = 
16_842_771
MAX_ENVELOPE_V2 = 
16_842_779
MAX_ENVELOPE = 
MAX_ENVELOPE_V2
ZSTD_LEVEL = 
6

Class Method Summary collapse

Class Method Details

.compress(text, dictionary: nil) ⇒ Object

Without a dictionary this emits MCDB1. With Database::Dictionary it emits MCDB2. Choose one format per database column; mixed use is deliberately not what the generated dictionary view is optimized for.



93
94
95
96
97
 
# File 'lib/multi_compress/database.rb', line 93

def compress(text, dictionary: nil)
  return compress_v1(text) if dictionary.nil?

  compress_v2(text, dictionary: dictionary)
end

.compress_v1(text) ⇒ Object 



99
100
101
102
103
104
105
106
 
# File 'lib/multi_compress/database.rb', line 99

def compress_v1(text)
  bytes = input_bytes!(text, format: "v1")
  frame = MultiCompress.compress(bytes, algo: :zstd, level: ZSTD_LEVEL)
  validate_canonical_zstd_frame!(frame, bytes.bytesize, expected_dictionary_id: 0)
  envelope = header_v1(bytes.bytesize, MultiCompress.crc32(bytes)) << frame
  check_envelope_size!(envelope, MAX_ENVELOPE_V1, "v1")
  envelope
end

.compress_v2(text, dictionary:) ⇒ Object 



108
109
110
111
112
113
114
115
116
 
# File 'lib/multi_compress/database.rb', line 108

def compress_v2(text, dictionary:)
  dictionary = database_dictionary!(dictionary)
  bytes = input_bytes!(text, format: "v2")
  frame = MultiCompress.compress(bytes, algo: :zstd, level: ZSTD_LEVEL, dictionary: dictionary.native)
  validate_canonical_zstd_frame!(frame, bytes.bytesize, expected_dictionary_id: dictionary.zstd_id)
  envelope = header_v2(bytes.bytesize, MultiCompress.crc32(bytes), dictionary.id) << frame
  check_envelope_size!(envelope, MAX_ENVELOPE_V2, "v2")
  envelope
end

.decompress(blob, dictionary: nil) ⇒ Object 



118
119
120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/multi_compress/database.rb', line 118

def decompress(blob, dictionary: nil)
  header = parse_header!(blob)
  case header.fetch(:version)
  when VERSION_V1
    raise ArgumentError, "MCDB1 does not accept a dictionary" unless dictionary.nil?

    decode_v1(blob.b, header)
  when VERSION_V2
    decode_v2(blob.b, header, dictionary)
  else
    err("unsupported version #{header.fetch(:version)}")
  end
end

.dictionary_ref(blob) ⇒ Object

Parses only the envelope header. It is safe for registry/FK consistency checks and does not decompress the payload.



141
142
143
144
145
146
 
# File 'lib/multi_compress/database.rb', line 141

def dictionary_ref(blob)
  header = parse_header!(blob)
  return nil if header.fetch(:version) == VERSION_V1

  header.fetch(:dictionary_ref)
end

.original_size(blob) ⇒ Object 



148
149
150
 
# File 'lib/multi_compress/database.rb', line 148

def original_size(blob)
  parse_header!(blob).fetch(:original_size)
end

.valid?(blob, dictionary: nil) ⇒ Boolean

Returns:

  • (Boolean) 


132
133
134
135
136
137
 
# File 'lib/multi_compress/database.rb', line 132

def valid?(blob, dictionary: nil)
  decompress(blob, dictionary: dictionary)
  true
rescue StandardError
  false
end

.version_string ⇒ Object 



152
153
154
 
# File 'lib/multi_compress/database.rb', line 152

def version_string
  "MCDB1 + MCDB2 (zstd #{MultiCompress.version(:zstd)})"
end