Class: Omnizip::Algorithms::XzUtilsDecoder

Inherits:

Object

Object
Omnizip::Algorithms::XzUtilsDecoder

show all

Includes:: LZMA::Constants

Defined in:: lib/omnizip/algorithms/lzma/xz_utils_decoder.rb

Overview

XZ Utils implementation of LZMA decoder

IMPORTANT: This is NOT the LZMA SDK/7-Zip decoder! XZ Utils modified LZMA significantly - this is XZ format only. Components integrated:

LiteralDecoder: Matched/unmatched literal decoding
StateMachine: 12-state FSM for probability model selection
LengthCoder: 3-level length decoding (low/mid/high)
DistanceCoder: 64-slot distance decoding with aligned bits

The decoder follows XZ Utils' exact decoding sequence:

Read LZMA header (property byte, dict size, uncompressed size)
Initialize range decoder and probability models
Decode loop:
- Decode is_match bit
- If literal: decode byte (matched/unmatched)
- If match: decode length and distance
- Update state machine
- Write to output
Handle EOS marker

Examples:

Basic usage

decoder = Omnizip::Algorithms::XzUtilsDecoder.new(input)
data = decoder.decode_stream

With output stream

decoder = Omnizip::Algorithms::XzUtilsDecoder.new(input)
File.open('output.txt', 'wb') { |f| decoder.decode_stream(f) }

Constant Summary collapse

MAX_DICT_SIZE = Maximum dictionary size to prevent memory exhaustion 64MB is a reasonable practical limit

64 * 1024 * 1024

BitModel = Alias for nested classes for easier access

LZMA::BitModel

LengthCoder =

LZMA::LengthCoder

DistanceCoder =

LZMA::DistanceCoder

LiteralDecoder =

LZMA::LiteralDecoder

RangeDecoder =

LZMA::RangeDecoder

SdkStateMachine =

Implementations::SevenZip::LZMA::StateMachine

LZ_DICT_REPEAT_MAX = XZ Utils dictionary constants (from lz_decoder.h) See: /Users/mulgogi/src/external/xz/src/liblzma/lz/lz_decoder.h

LZ_DICT_INIT_POS = 576

2 * LZ_DICT_REPEAT_MAX

Constants included from LZMA::Constants

Instance Attribute Summary collapse

#dict_size ⇒ Object readonly
Returns the value of attribute dict_size.
#lc ⇒ Object readonly
Returns the value of attribute lc.
#lp ⇒ Object readonly
Returns the value of attribute lp.
#pb ⇒ Object readonly
Returns the value of attribute pb.
#range_decoder ⇒ Object readonly
Returns the value of attribute range_decoder.
#uncompressed_size ⇒ Object readonly
Returns the value of attribute uncompressed_size.

Instance Method Summary collapse

#add_to_dictionary(data) ⇒ void
Add uncompressed data to the dictionary.
#compact_buffer ⇒ Object
No-op: circular buffer has fixed size, no compaction needed.
#decode_stream(output = nil, preserve_dict: false, check_rc_finished: true) ⇒ String, Integer
Decode a compressed stream.
#dict_index(pos) ⇒ Object
Map a monotonic position to a circular buffer index.
#finish_state_reset ⇒ void
Finish state reset - called AFTER setting new input.
#flush_circular_output(output) ⇒ Object
Flush decoded output from the circular dictionary buffer.
#initialize(input, options = {}) ⇒ XzUtilsDecoder constructor
Initialize the SDK-compatible decoder.
#prepare_state_reset ⇒ void
Reset only state machine and rep distances, preserve probability models.
#reset(new_lc: nil, new_lp: nil, new_pb: nil, preserve_dict: false) ⇒ void
Reset the decoder state for reuse with new properties.
#reset_models ⇒ void
Reset all probability models in place (without creating new arrays).
#reset_range_decoder ⇒ void
Reset only the range decoder for next chunk.
#reset_state_machine_only ⇒ void
Reset state machine only - preserves rep distances.
#reset_state_only ⇒ Object
#set_input(new_input) ⇒ void
Set new input stream for chunked decoding.
#set_uncompressed_size(size, allow_eopm: true) ⇒ void
Set uncompressed size for chunked decoding.

Constructor Details

#initialize(input, options = {}) ⇒ `XzUtilsDecoder`

Initialize the SDK-compatible decoder

Parameters:

input (IO) —
Input stream of compressed data
options (Hash) (defaults to: {}) —
Decoding options

Options Hash (options):

:lzma2_mode (Boolean) —
If true, initialize without reading header (for LZMA2 use, requires lc, lp, pb, dict_size, uncompressed_size)
:lc (Integer) —
Literal context bits (required for lzma2_mode)
:lp (Integer) —
Literal position bits (required for lzma2_mode)
:pb (Integer) —
Position bits (required for lzma2_mode)
:dict_size (Integer) —
Dictionary size (required for lzma2_mode)
:uncompressed_size (Integer) —
Uncompressed size (required for lzma2_mode)
:preloaded_data (String) —
Data to preload into dictionary (for LZMA2 uncompressed chunks followed by compressed chunks)
:validate_size (Boolean) —
If true, validate decoded size matches uncompressed_size (default: false, only for .lzma format)

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 156

def initialize(input, options = {})
  @input = input
  @decoder_id = object_id # Track decoder instance ID

  # Check for preloaded data (from LZMA2 uncompressed chunks)
  @preloaded_data = options[:preloaded_data]
  @validate_size = options.fetch(:validate_size, false)
  @allow_eopm = options.fetch(:allow_eopm, nil)

  if options[:lzma2_mode]
    # Direct initialization for LZMA2 use (XZ Utils pattern)
    # The LZMA2 decoder provides parameters directly, no header to read
    # See: /Users/mulgogi/src/external/xz/src/liblzma/lzma/lzma2_decoder.c:140-141
    @lc = options.fetch(:lc)
    @lp = options.fetch(:lp)
    @pb = options.fetch(:pb)
    @dict_size = [[options.fetch(:dict_size), 1].max, MAX_DICT_SIZE].min
    @uncompressed_size = options.fetch(:uncompressed_size)
  else
    # Standalone LZMA file - read header from input
    read_header
  end

  validate_parameters
  init_models
  init_coders

  # Cache computed values for hot loop
  @pb_mask = (1 << @pb) - 1
  @pb_shift = 1 << @pb
  @literal_mask = (0x100 << @lp) - (0x100 >> @lc)
end

Instance Attribute Details

#dict_size ⇒ `Object` (readonly)

Returns the value of attribute dict_size.



89
90
91

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 89

def dict_size
  @dict_size
end

#lc ⇒ `Object` (readonly)

Returns the value of attribute lc.



89
90
91

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 89

def lc
  @lc
end

#lp ⇒ `Object` (readonly)

Returns the value of attribute lp.



89
90
91

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 89

def lp
  @lp
end

#pb ⇒ `Object` (readonly)

Returns the value of attribute pb.



89
90
91

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 89

def pb
  @pb
end

#range_decoder ⇒ `Object` (readonly)

Returns the value of attribute range_decoder.



89
90
91

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 89

def range_decoder
  @range_decoder
end

#uncompressed_size ⇒ `Object` (readonly)

Returns the value of attribute uncompressed_size.



89
90
91

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 89

def uncompressed_size
  @uncompressed_size
end

Instance Method Details

#add_to_dictionary(data) ⇒ `void`

This method returns an undefined value.

Add uncompressed data to the dictionary

XZ Utils pattern (lzma2_decoder.c:195, dict_write):

Copy uncompressed data to the dictionary as-is
Update dict_full to reflect new data
This allows subsequent compressed chunks to reference the data

This is used by LZMA2 decoder for uncompressed chunks (control=0x1 or 0x2)

Parameters:

data (String) —
Uncompressed data to add to dictionary

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 707

def add_to_dictionary(data)
  idx = @dict_pos
  buf_end = @buf_end
  data.each_byte do |byte|
    @dict_buf.setbyte(idx, byte)
    @pos += 1
    idx += 1
    idx = LZ_DICT_INIT_POS if idx >= buf_end
  end
  @dict_pos = idx

  # Update dict_full to reflect new data
  unless @has_wrapped
    @dict_full = @pos - LZ_DICT_INIT_POS
    if @dict_full >= @dict_size
      @has_wrapped = true
      @dict_full = @dict_size
    end
  end
end

#compact_buffer ⇒ `Object`

No-op: circular buffer has fixed size, no compaction needed. Kept for API compatibility — LZMA2 decoder calls this between chunks.

730	# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 730 def compact_buffer; end

#decode_stream(output = nil, preserve_dict: false, check_rc_finished: true) ⇒ `String`, `Integer`

Decode a compressed stream

Main decoding loop following SDK's LzmaDec_DecodeToDic logic:

Initialize range decoder
Process each position: decode literals/matches
Detect EOS marker
Return decompressed data

XZ Utils dictionary system (from lz_decoder.h):

pos starts at LZ_DICT_INIT_POS (576)
full = pos - LZ_DICT_INIT_POS (count of valid bytes)
has_wrapped = false until dictionary buffer wraps
Distance validation: full > distance

Parameters:

output (IO, nil) (defaults to: nil) —
Optional output stream (if nil, returns String)
preserve_dict (Boolean) (defaults to: false) —
Whether to preserve dictionary from previous decode
check_rc_finished (Boolean) (defaults to: true) —
Whether to check if range decoder is finished

Returns:

(String, Integer) —
Decompressed data or bytes written

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 207

def decode_stream(output = nil, preserve_dict: false,
check_rc_finished: true)
  @decode_stream_call_count ||= 0
  @decode_stream_call_count += 1

  # Create range decoder if it doesn't exist (first chunk)
  # This happens when the decoder is created directly for LZMA (not LZMA2)
  @range_decoder ||= RangeDecoder.new(@input)

  # Special case: empty input (uncompressed_size == 0)
  # Return immediately without trying to decode anything
  if @uncompressed_size != 0xFFFFFFFFFFFFFFFF && @uncompressed_size.zero?
    return "" # Empty output
  end

  @debug_iter = 0

  # Track bytes decoded in this chunk (for multi-chunk streams)
  # This is needed to limit match lengths correctly when @uncompressed_size
  # represents only the current chunk's size, not the total size
  @chunk_bytes_decoded = 0

  # Initialize state and dictionary (XZ Utils system from lz_decoder.c)
  # See: /Users/mulgogi/src/external/xz/src/liblzma/lz/lz_decoder.c:56
  # For LZMA2 multi-chunk streams, state machine persists across chunks
  # Only reset when not preserving dictionary (first chunk)
  #
  # IMPORTANT: Initialize @state if it's nil (first call) OR if not preserving dict
  if @state.nil? || !preserve_dict
    @state = SdkStateMachine.new
  end

  # For LZMA2 multi-chunk streams, preserve dictionary across chunks
  # when preserve_dict is true (control >= 0x80 but < 0xA0)
  # For subsequent chunks, the reset() method handles dictionary reset
  # For the first chunk (when @dict_buf is nil), we need to init it here
  if @dict_buf.nil?
    @buf_end = LZ_DICT_INIT_POS + @dict_size
    buf_size = @buf_end
    @dict_buf = ("\0" * buf_size).b
    @pos = LZ_DICT_INIT_POS
    @dict_pos = LZ_DICT_INIT_POS # Circular buffer write position (tracks @pos)
    @dict_full = 0
    @has_wrapped = false

    # Add preloaded data to dictionary (from LZMA2 uncompressed chunks)
    # This must be done before decoding so matches can reference this data
    if @preloaded_data && !@preloaded_data.empty?
      idx = @dict_pos
      buf_end = @buf_end
      @preloaded_data.each_byte do |byte|
        @dict_buf.setbyte(idx, byte)
        @pos += 1
        idx += 1
        idx = LZ_DICT_INIT_POS if idx >= buf_end
      end
      @dict_pos = idx
      # Update dict_full to reflect preloaded data
      @dict_full = @pos - LZ_DICT_INIT_POS
      @preloaded_data = nil # Clear after loading
    end
  end

  # Track starting position for multi-chunk streams
  # IMPORTANT: Calculate start_pos AFTER dictionary initialization!
  # This ensures that preloaded data (from LZMA2 uncompressed chunks) is
  # properly reflected in start_pos, so we only return NEW bytes.
  # For LZMA2, we need to return only the NEW bytes, not all bytes from LZ_DICT_INIT_POS
  start_pos = @pos || LZ_DICT_INIT_POS

  # Initialize rep distances (XZ Utils initializes to 0)
  # See: /Users/mulgogi/src/external/xz/src/liblzma/lzma/lzma_decoder.c:1054-1055
  # For LZMA2 multi-chunk streams, rep distances persist across chunks
  # Only reset when not preserving dictionary (first chunk)
  #
  # IMPORTANT: Initialize rep distances if they're nil OR not preserving dict
  if @rep0.nil? || @rep1.nil? || @rep2.nil? || @rep3.nil? || !preserve_dict
    @rep0 = 0
    @rep1 = 0
    @rep2 = 0
    @rep3 = 0
  end

  # Read range decoder init bytes (must happen after set_input sets correct stream)
  if @range_decoder.respond_to?(:read_init_bytes) && @range_decoder.init_bytes_remaining.positive?
    @range_decoder.read_init_bytes
  end

  # Main decoding loop
  # XZ Utils pattern (lzma_decoder.c:305-306):
  # Set dict.limit = dict.pos + (size_t)(coder->uncompressed_size)
  # Then check dict.pos < dict.limit
  # Since our @pos starts at LZ_DICT_INIT_POS, we set limit accordingly
  # IMPORTANT: For multi-chunk streams, calculate limit from start_pos, not LZ_DICT_INIT_POS!
  # XZ Utils uses dict->pos (current position) + uncompressed_size
  # We use start_pos (current position) + @uncompressed_size
  limit = if @uncompressed_size == 0xFFFFFFFFFFFFFFFF
            nil # No limit for unknown size
          else
            start_pos + @uncompressed_size
          end

  # Circular buffer: no growth needed. Buffer is fixed at dict_size + LZ_DICT_INIT_POS.
  # Output is flushed incrementally when the circular position nears wrap point.
  # Matches C++ 7-Zip SDK: dic is pre-allocated to dicBufSize and never resized.
  @flush_pos = @pos
  @output_accumulator = output.nil? ? StringIO.new("".b) : nil

  iteration = 0
  loop do
    iteration += 1
    # Check if we've reached the expected size (if known)
    # XZ Utils: checks dict.pos < dict.limit

    # Handle nil @pos or limit gracefully
    if limit && (@pos.nil? || limit.nil?)
      raise "Invalid state: @pos=#{@pos.inspect}, limit=#{limit.inspect}"
    end

    # Circular buffer: flush output before it can be overwritten.
    # LZMA matches are at most 273 bytes, so flush when within 273 of wrapping.
    if @pos - @flush_pos > @dict_size - 273
      flush_circular_output(output)
    end

    # Decode is_match bit
    pos_state = @pos & @pb_mask
    # XZ Utils: is_match[state][pos_state] where the array is NUM_STATES * (1 << pb)
    # The array stride changes with pb value
    model_index = (@state.value * @pb_shift) + pos_state

    @debug_iter += 1

    is_match = @range_decoder.decode_bit(@is_match_models[model_index])

    if is_match.zero?
      # Decode literal
      decode_literal
    elsif decode_match
      # Decode match - returns true if EOS detected
      break
    end

    # XZ Utils: Check if we've reached the limit (known uncompressed size)
    # Reference: lzma_decoder.c:347, 680-692
    # When dict.pos == dict.limit, the decoder should stop
    # IMPORTANT: Must verify range decoder is finished (code == 0)
    # If code != 0, there's leftover data in the compressed stream (corruption)
    if limit && @pos >= limit

      # XZ Utils pattern (lzma_decoder.c:689-700):
      # Check if range decoder is finished (code == 0)
      # - If finished → STREAM_END (success)
      # - If NOT finished AND allow_eopm is false → DATA_ERROR (corruption)
      # - If NOT finished AND allow_eopm is true → continue (expect EOPM)
      # Reference: /Users/mulgogi/src/external/xz/src/liblzma/lzma/lzma_decoder.c:689-700
      #
      # For LZMA2: @allow_eopm is false, so range decoder MUST be finished
      # For .lzma format: @allow_eopm may be true, so we continue decoding to find EOPM
      # Reference: /Users/mulgogi/src/external/xz/src/liblzma/rangecoder/range_decoder.h:138-139
      # rc_is_finished(range_decoder) = ((range_decoder).code == 0)
      #
      # NOTE: The check_rc_finished parameter is a legacy override for .lzma format
      # If explicitly set to false, it allows EOPM even when uncompressed size is known
      # Reference: alone_decoder.c:127 (LZMA_LZMA1EXT_ALLOW_EOPM)
      should_check = if @allow_eopm == true
                       # EOPM is explicitly allowed, skip the check
                       false
                     elsif @allow_eopm == false
                       # LZMA2 mode: always check (EOPM is not allowed)
                       true
                     else
                       # @allow_eopm is nil (not set, first chunk or legacy mode)
                       # Use check_rc_finished parameter as default
                       check_rc_finished
                     end

      if should_check
        # If EOPM is not allowed, range decoder MUST be finished
        unless @range_decoder.code.zero?
          raise Omnizip::DecompressionError,
                "LZMA stream finished with leftover compressed data (range_decoder.code=#{@range_decoder.code}, expected 0). This indicates corruption in the compressed stream or an invalid EOPM for LZMA2."
        end
        # XZ Utils pattern (lzma_decoder.c): when STREAM_END is reached,
        # reset the range coder for the next chunk. This happens even for
        # no-reset chunks (control 0x80-0x9F) - the range coder is ALWAYS
        # re-initialized between chunks, only state/dict/models are preserved.
        @range_decoder.reset
        break
      elsif @range_decoder.code.zero?
        # EOPM is allowed (e.g., LZMA_Alone format)
        # If range decoder is finished, we're done
        # XZ Utils: rc_reset at STREAM_END
        @range_decoder.reset
        break
        # Otherwise, continue decoding to find EOPM marker
        # XZ Utils sets eopm_is_valid = true and continues
        # Reference: lzma_decoder.c:704
      end
    end
  end

  # Validate decoded size against expected uncompressed_size
  # Only for .lzma (LZMA_Alone) format where validate_size=true
  # For .lzma format with known uncompressed_size, verify we decoded the right amount
  # This catches "too_small_size-without-eopm" files where the header says 1 byte
  # but the compressed data produces more output
  # XZ format does NOT validate size at the LZMA decoder level - it's handled at block level
  if @validate_size && @uncompressed_size && @uncompressed_size != 0xFFFFFFFFFFFFFFFF
    # Calculate actual decoded size (from start of data, not LZ_DICT_INIT_POS)
    actual_decoded_size = @pos - LZ_DICT_INIT_POS

    if actual_decoded_size != @uncompressed_size
      raise Omnizip::DecompressionError,
            "LZMA stream size mismatch: expected #{@uncompressed_size} bytes, decoded #{actual_decoded_size} bytes. The file may be corrupted or have an invalid uncompressed size field."
    end

    # IMPORTANT: Check for leftover compressed data after EOPM
    # If EOPM was encountered (range_decoder.code == 0) but there's still data
    # in the input stream, the file is corrupted.
    # Reference: /Users/mulgogi/src/external/xz/src/liblzma/common/alone_decoder.c
    #
    # We only check for leftover data when:
    # 1. EOPM was encountered (code == 0) AND
    # 2. There's more data in the input stream
    #
    # If EOPM was NOT encountered (code != 0), leftover data is expected
    # (it's part of the compressed stream that we haven't read yet).
    if @allow_eopm && @range_decoder&.code&.zero? && @range_decoder&.stream
      stream = @range_decoder.stream
      # Try to peek at the next byte - if available, there's data AFTER EOPM
      begin
        next_byte = stream.getbyte
        if next_byte
          # Put the byte back
          stream.ungetbyte(next_byte) if stream.respond_to?(:ungetbyte)
          raise Omnizip::DecompressionError,
                "LZMA_Alone file has data after the end-of-payload marker. The file may be corrupted or contain concatenated streams."
        end
      rescue IOError, EOFError
        # Stream doesn't support peeking or is exhausted, that's fine
      end
    elsif !@allow_eopm && @range_decoder&.stream
      # For LZMA2 mode (EOPM not allowed): check for leftover data
      stream = @range_decoder.stream
      begin
        next_byte = stream.getbyte
        if next_byte
          stream.ungetbyte(next_byte) if stream.respond_to?(:ungetbyte)
          raise Omnizip::DecompressionError,
                "LZMA_Alone file has more compressed data than expected. The uncompressed size field (#{@uncompressed_size} bytes) appears to be too small."
        end
      rescue IOError, EOFError
        # Stream doesn't support peeking or is exhausted, that's fine
      end
    end
  end

  # Flush remaining output from the circular buffer
  flush_circular_output(output)

  # Return output
  if output
    @pos - start_pos
  else
    @output_accumulator.string.force_encoding(Encoding::BINARY)
  end
end

#dict_index(pos) ⇒ `Object`

Map a monotonic position to a circular buffer index. Buffer layout: [LZ_DICT_INIT_POS zero bytes] + [dict_size circular region] Matches C++ 7-Zip SDK: dic with dicPos wrapping at dicBufSize.



101
102
103

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 101

def dict_index(pos)
  LZ_DICT_INIT_POS + ((pos - LZ_DICT_INIT_POS) % @dict_size)
end

#finish_state_reset ⇒ `void`

This method returns an undefined value.

Finish state reset - called AFTER setting new input

Resets the range decoder to read from the new input stream. This completes the state reset process started by prepare_state_reset.

XZ Utils pattern (lzma_decoder.c:1034-1083):

rc_reset is called as part of lzma_decoder_reset
rc_reset sets range = UINT32_MAX, code = 0, init_bytes_left = 5
The 5 initialization bytes are read during the first normalize calls

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 646

def finish_state_reset
  # Reset range decoder (XZ Utils rc_reset)
  # This reinitializes the range decoder for the new chunk
  # The reset will read 5 bytes from the input when decode_stream starts
  @range_decoder&.reset
end

#flush_circular_output(output) ⇒ `Object`

Flush decoded output from the circular dictionary buffer. Copies bytes from @flush_pos to @pos into the output stream or accumulator. Handles wrap-around when the data spans the circular buffer boundary.

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 108

def flush_circular_output(output)
  return if @pos == @flush_pos

  length = @pos - @flush_pos

  if length > @dict_size
    raise Omnizip::DecompressionError,
          "Internal error: flush lag (#{length}) exceeds dict_size (#{@dict_size})"
  end

  start_idx = dict_index(@flush_pos)
  buf_end = @buf_end

  if start_idx + length <= buf_end
    # No wrap
    data = @dict_buf.byteslice(start_idx, length)
  else
    # Wraps around the circular boundary
    first_len = buf_end - start_idx
    data = @dict_buf.byteslice(start_idx, first_len)
    data << @dict_buf.byteslice(LZ_DICT_INIT_POS, length - first_len)
  end

  if output
    output.write(data.force_encoding(Encoding::BINARY))
  else
    @output_accumulator ||= StringIO.new("".b)
    @output_accumulator.write(data)
  end

  @flush_pos = @pos
end

#prepare_state_reset ⇒ `void`

Reset only state machine and rep distances, preserve probability models

XZ Utils pattern for state reset only (control >= 0xA0):

Reset state machine
Reset rep distances
Reset probability models (via reset_models)
Reset range decoder (rc_reset + rc_read_init)
PRESERVE dictionary content (no dict_reset)

XZ Utils source (lzma2_decoder.c):

For control >= 0xA0: calls lzma_lzma_decoder_reset(decoder, NULL)
lzma_lzma_decoder_reset always calls init_temporals which resets probability models

Prepare state reset - called BEFORE setting new input

Resets state machine, rep distances, and probability models. The range decoder will be reset in finish_state_reset AFTER the new input is set (to match XZ Utils lzma_decoder_reset behavior).

For LZMA2 control >= 0xC0, this is called before set_input to reset everything except the range decoder for the new chunk.

Returns:

(void)
(void)

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 602

def prepare_state_reset
  # Reset state machine (XZ Utils line 1053)
  @state = SdkStateMachine.new

  # Reset rep distances (XZ Utils lines 1054-1057)
  @rep0 = 0
  @rep1 = 0
  @rep2 = 0
  @rep3 = 0

  # Reset probability models (XZ Utils init_temporals for control >= 0xA0)
  reset_models

  nil
end

#reset(new_lc: nil, new_lp: nil, new_pb: nil, preserve_dict: false) ⇒ `void`

This method returns an undefined value.

Reset the decoder state for reuse with new properties

XZ Utils pattern (lzma_decoder.c:1034-1083):

Resets state machine and rep distances
Resets range decoder
Reinitializes all probability models
Preserves dictionary (managed externally by LZMA2 decoder)

Parameters:

new_lc (Integer, nil) (defaults to: nil) —
New lc value (if nil, keeps current)
new_lp (Integer, nil) (defaults to: nil) —
New lp value (if nil, keeps current)
new_pb (Integer, nil) (defaults to: nil) —
New pb value (if nil, keeps current)
preserve_dict (Boolean) (defaults to: false) —
If true, preserve dictionary state (pos, dict_full)

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 489

def reset(new_lc: nil, new_lp: nil, new_pb: nil, preserve_dict: false)
  # Update properties if provided
  properties_changed = !!(new_lc || new_lp || new_pb)
  @lc = new_lc if new_lc
  @lp = new_lp if new_lp
  @pb = new_pb if new_pb

  # Recompute cached values when properties change
  if properties_changed
    @pb_mask = (1 << @pb) - 1
    @pb_shift = 1 << @pb
    @literal_mask = (0x100 << @lp) - (0x100 >> @lc)
  end

  # Reset state machine (XZ Utils line 1053)
  # Always create a new state machine when resetting
  @state = SdkStateMachine.new

  # Reset rep distances (XZ Utils lines 1071-1074)
  # IMPORTANT: ALWAYS reset rep distances to 0 when state is reset
  # This happens for both control=0xE0 (dict reset) and control=0xC0 (state reset)
  # Reference: /Users/mulgogi/src/external/xz/src/liblzma/lzma/lzma_decoder.c:1071-1074
  @rep0 = 0
  @rep1 = 0
  @rep2 = 0
  @rep3 = 0

  # Reset range decoder for next chunk
  # XZ Utils pattern (lzma_decoder.c:1061):
  # - rc_reset sets range=UINT32_MAX, code=0, init_bytes_left=5
  # - This MUST happen during reset, not deferred to decode_stream
  # Reference: /Users/mulgogi/src/external/xz/src/liblzma/lzma/lzma_decoder.c:1061
  @range_decoder&.reset

  # Reinitialize probability models (XZ Utils lines 1064-1082)
  # IMPORTANT: Use reset_models (reset in place) instead of init_models (create new)
  # for state reset only. Only create new models when properties change.
  if properties_changed
    init_models
  else
    reset_models
  end

  # Reinitialize coders (needed for pb changes)
  # Only recreate coders when properties have changed
  init_coders if properties_changed

  # Reset dictionary position and full count (XZ Utils pattern)
  # Only reset if preserve_dict is false
  unless preserve_dict
    # Reinitialize dictionary buffer
    # XZ Utils allocates a new buffer for each dictionary reset
    @dict_buf = ("\0" * (@dict_size + LZ_DICT_INIT_POS)).b
    @pos = LZ_DICT_INIT_POS
    @dict_pos = LZ_DICT_INIT_POS
    @dict_full = 0
    @has_wrapped = false
  end

  nil
end

#reset_models ⇒ `void`

This method returns an undefined value.

Reset all probability models in place (without creating new arrays)

This matches XZ Utils init_temporals behavior for control >= 0xA0. Unlike init_models which creates new arrays, this resets existing BitModels in place to preserve object identity for any references.

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 558

def reset_models
  # Reset literal models
  @literal_models.each(&:reset)

  # Reset match/rep models
  @is_match_models.each(&:reset)
  @is_rep_models.each(&:reset)
  @is_rep0_models.each(&:reset)
  @is_rep1_models.each(&:reset)
  @is_rep2_models.each(&:reset)
  @is_rep0_long_models.each(&:reset)

  # Reset length coders
  @length_coder.reset_models
  @rep_length_coder.reset_models

  # Reset distance coder
  @distance_coder.reset_models
end

#reset_range_decoder ⇒ `void`

This method returns an undefined value.

Reset only the range decoder for next chunk

XZ Utils pattern (lzma_decoder.c:1014-1017): When LZMA chunk ends (LZMA_STREAM_END), reset range decoder for next LZMA2 chunk, but preserve state and probability models.

Note: This method is a no-op in our implementation because decode_stream creates a fresh RangeDecoder for each chunk. The range decoder initialization happens automatically when the new RangeDecoder is created with the new input.



671
672
673

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 671

def reset_range_decoder
  # No-op: RangeDecoder is created fresh in decode_stream
end

#reset_state_machine_only ⇒ `void`

This method returns an undefined value.

Reset state machine only - preserves rep distances

This is used for control >= 0xA0 but < 0xC0 where we want to reset the state machine but preserve rep distances from the previous chunk.

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 625

def reset_state_machine_only
  # Reset state machine only (XZ Utils line 1053)
  @state = SdkStateMachine.new

  # Reset probability models (XZ Utils init_temporals for control >= 0xA0)
  reset_models

  nil
end

#reset_state_only ⇒ `Object`

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 653

def reset_state_only
  # Complete state reset requires both prepare and finish phases
  prepare_state_reset
  finish_state_reset
end

#set_input(new_input) ⇒ `void`

This method returns an undefined value.

Set new input stream for chunked decoding

For LZMA2, the range decoder is persistent across chunks and is reset separately via prepare_state_reset + finish_state_reset. This method just updates the input stream reference.

Parameters:

new_input (IO) —
New input stream

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 683

def set_input(new_input)
  @input = new_input

  # Create range decoder if it doesn't exist (first chunk)
  if @range_decoder.nil?
    @range_decoder = RangeDecoder.new(@input)
  else
    # Update the range decoder's input stream to the new input
    # This is needed because RangeDecoder holds a reference to the stream
    @range_decoder.update_stream(@input)
  end
end

#set_uncompressed_size(size, allow_eopm: true) ⇒ `void`

This method returns an undefined value.

Set uncompressed size for chunked decoding

XZ Utils pattern (lzma2_decoder.c:140-141): Pass the chunk's uncompressed_size to the LZMA decoder for each LZMA2 chunk.

Parameters:

size (Integer) —
Uncompressed size for current chunk
allow_eopm (Boolean) (defaults to: true) —
Whether to allow end-of-payload marker

# File 'lib/omnizip/algorithms/lzma/xz_utils_decoder.rb', line 741

def set_uncompressed_size(size, allow_eopm: true)
  @uncompressed_size = size
  @allow_eopm = allow_eopm
end

Class: Omnizip::Algorithms::XzUtilsDecoder

Overview

Examples:

Basic usage

With output stream

Constant Summary collapse

576

Constants included from LZMA::Constants

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(input, options = {}) ⇒ XzUtilsDecoder

Instance Attribute Details

#dict_size ⇒ Object (readonly)

#lc ⇒ Object (readonly)

#lp ⇒ Object (readonly)

#pb ⇒ Object (readonly)

#range_decoder ⇒ Object (readonly)

#uncompressed_size ⇒ Object (readonly)

Instance Method Details

#add_to_dictionary(data) ⇒ void

#compact_buffer ⇒ Object

#decode_stream(output = nil, preserve_dict: false, check_rc_finished: true) ⇒ String, Integer

#dict_index(pos) ⇒ Object

#finish_state_reset ⇒ void

#flush_circular_output(output) ⇒ Object

#prepare_state_reset ⇒ void

#reset(new_lc: nil, new_lp: nil, new_pb: nil, preserve_dict: false) ⇒ void

#reset_models ⇒ void

#reset_range_decoder ⇒ void

#reset_state_machine_only ⇒ void

#reset_state_only ⇒ Object

#set_input(new_input) ⇒ void

#set_uncompressed_size(size, allow_eopm: true) ⇒ void

#initialize(input, options = {}) ⇒ `XzUtilsDecoder`

#dict_size ⇒ `Object` (readonly)

#lc ⇒ `Object` (readonly)

#lp ⇒ `Object` (readonly)

#pb ⇒ `Object` (readonly)

#range_decoder ⇒ `Object` (readonly)

#uncompressed_size ⇒ `Object` (readonly)

#add_to_dictionary(data) ⇒ `void`

#compact_buffer ⇒ `Object`

#decode_stream(output = nil, preserve_dict: false, check_rc_finished: true) ⇒ `String`, `Integer`

#dict_index(pos) ⇒ `Object`

#finish_state_reset ⇒ `void`

#flush_circular_output(output) ⇒ `Object`

#prepare_state_reset ⇒ `void`

#reset(new_lc: nil, new_lp: nil, new_pb: nil, preserve_dict: false) ⇒ `void`

#reset_models ⇒ `void`

#reset_range_decoder ⇒ `void`

#reset_state_machine_only ⇒ `void`

#reset_state_only ⇒ `Object`

#set_input(new_input) ⇒ `void`

#set_uncompressed_size(size, allow_eopm: true) ⇒ `void`