Module: Omnizip::Algorithms::Zstandard::LiteralsEncoder

Includes:
Constants
Defined in:
lib/omnizip/algorithms/zstandard/literals_encoder.rb

Overview

Literals Section encoder (RFC 8878 ยง3.1.1.3.1): picks Raw, RLE or Huffman-compressed, whichever is smallest.

Constant Summary

Constants included from Constants

Constants::BLOCK_HEADER_SIZE, Constants::BLOCK_MAX_SIZE, Constants::BLOCK_TYPE_COMPRESSED, Constants::BLOCK_TYPE_RAW, Constants::BLOCK_TYPE_RESERVED, Constants::BLOCK_TYPE_RLE, Constants::BUFFER_SIZE, Constants::DEFAULT_LEVEL, Constants::DEFAULT_REPEAT_OFFSETS, Constants::FSE_DEFAULT_TABLELOG, Constants::FSE_MAX_ACCURACY_LOG, Constants::FSE_MIN_ACCURACY_LOG, Constants::HUFFMAN_MAX_BITS, Constants::HUFFMAN_MAX_CODE_LENGTH, Constants::HUFFMAN_MAX_LOG, Constants::HUFFMAN_STANDARD_TABLE_SIZE, Constants::HUF_SYMBOLVALUE_MAX, Constants::LDM_MIN_LEVEL, Constants::LITERALS_BLOCK_COMPRESSED, Constants::LITERALS_BLOCK_RAW, Constants::LITERALS_BLOCK_RLE, Constants::LITERALS_BLOCK_TREELESS, Constants::LITERALS_LENGTH_ACCURACY_LOG, Constants::LITERAL_LENGTH_TABLE, Constants::MAGIC_BYTES, Constants::MAGIC_NUMBER, Constants::MATCH_LENGTH_ACCURACY_LOG, Constants::MATCH_LENGTH_TABLE, Constants::MAX_LEVEL, Constants::MIN_LEVEL, Constants::MODE_FSE, Constants::MODE_PREDEFINED, Constants::MODE_REPEAT, Constants::MODE_RLE, Constants::OFFSET_ACCURACY_LOG, Constants::OF_BASE, Constants::OF_BITS, Constants::PREDEFINED_LL_DISTRIBUTION, Constants::PREDEFINED_ML_DISTRIBUTION, Constants::PREDEFINED_OFFSET_DISTRIBUTION, Constants::REPEAT_OFFSET_1, Constants::REPEAT_OFFSET_2, Constants::REPEAT_OFFSET_3, Constants::SKIPPABLE_MAGIC_BASE, Constants::SKIPPABLE_MAGIC_MASK, Constants::WINDOW_LOG_MAX, Constants::WINDOW_LOG_MIN

Class Method Summary collapse

Methods included from Constants

highbit32

Class Method Details

.encode(literals) ⇒ String

Encode a literals section for literals.

Parameters:

  • literals (String)

Returns:

  • (String) —

    the full section (header + content)



37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/omnizip/algorithms/zstandard/literals_encoder.rb', line 37

def encode(literals)
  return encode_raw(literals) if literals.bytesize < 16

  return encode_rle(literals) if rle?(literals)

  compressed = nil
  begin
    compressed = HuffmanEncoder.encode_literals(literals)
  rescue Omnizip::CompressionError
    compressed = nil
  end

  if compressed && compressed.bytesize < literals.bytesize
    compressed
  else
    encode_raw(literals)
  end
end

.encode_raw(literals) ⇒ Object

Raw literals block with the minimal size-format header.



57
58
59
60
 
# File 'lib/omnizip/algorithms/zstandard/literals_encoder.rb', line 57

def encode_raw(literals)
  encode_raw_rle_header(LITERALS_BLOCK_RAW, literals.bytesize) +
    literals
end

.encode_raw_rle_header(type, size) ⇒ Object

1/2/3-byte Raw/RLE header per the size-format rules. For the 2- and 3-byte forms the Size_Format bits (01 / 11) live in bits 2-3 and the size starts at bit 4.



82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/omnizip/algorithms/zstandard/literals_encoder.rb', line 82

def encode_raw_rle_header(type, size)
  if size < 32
    [type | (size << 3)].pack("C")
  elsif size < 4096
    lhc = type | 0x04 | (size << 4)
    [lhc & 0xFF, (lhc >> 8) & 0xFF].pack("CC")
  else
    lhc = type | 0x0C | (size << 4)
    [lhc & 0xFF, (lhc >> 8) & 0xFF, (lhc >> 16) & 0xFF].pack("CCC")
  end
end

.encode_rle(literals) ⇒ Object

RLE literals block: one byte, repeated.



63
64
65
66
 
# File 'lib/omnizip/algorithms/zstandard/literals_encoder.rb', line 63

def encode_rle(literals)
  encode_raw_rle_header(LITERALS_BLOCK_RLE, literals.bytesize) +
    literals.getbyte(0).chr
end

.rle?(literals) ⇒ Boolean

Returns:

  • (Boolean) 


68
69
70
71
72
73
74
75
76
77
 
# File 'lib/omnizip/algorithms/zstandard/literals_encoder.rb', line 68

def rle?(literals)
  first = literals.getbyte(0)
  idx = 1
  while idx < literals.bytesize
    return false if literals.getbyte(idx) != first

    idx += 1
  end
  true
end