Class: Oxygene::CARArchive

Inherits:
Object
  • Object
show all
Defined in:
lib/oxygene/car_archive.rb

Overview

Parses a Content Addressable Archive (CAR) bundle loaded e.g. from an ATProto firehose message or from a repo .car exported from a PDS. The header part is decoded immediately, while the subsequent body/data sections are lazily decoded only as requested.

Only CAR version v1 is supported, and only limited to the features or variants used in ATProto (e.g. CIDs only in the v1 version as defined in DASL).

Related specifications:

Direct Known Subclasses

CARRepo

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(data) ⇒ CARArchive

Creates a CAR archive reader from an in-memory CAR file.

Parameters:

  • data (String) —

    CAR archive file as a binary string

Raises:

  • (DecodeError) —

    if the archive header has missing or invalid fields

  • (UnsupportedError) —

    if the archive uses an unsupported CAR version



46
47
48
49
50
51
52
53
 
# File 'lib/oxygene/car_archive.rb', line 46

def initialize(data)
  @sections = []
  @section_map = {}
  @buffer = StringIO.new(data)
  @map_needs_update = false

  read_header(@buffer)
end

Instance Attribute Details

#roots ⇒ Array<CID> (readonly)

Returns array of root CIDs listed in the archive header.

Returns:

  • (Array<CID>) —

    array of root CIDs listed in the archive header



37
38
39
 
# File 'lib/oxygene/car_archive.rb', line 37

def roots
  @roots
end

Class Method Details

.convert_data(object) ⇒ Hash, Array

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Converts decoded CBOR values to their ATProto-specific JSON representation.

The passed hash or array is converted recursively in place. The conversion involves:

  • replacing binary strings with $bytes objects with Base64-encoded data
  • converting CBOR CID tags to $link objects holding an Oxygene::CID

Parameters:

  • object (Hash, Array) —

    decoded CBOR object to convert

Returns:

  • (Hash, Array) —

    the same object, updated in place

Raises:

  • (DecodeError) —

    if the top-level value is not a hash or array



134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
 
# File 'lib/oxygene/car_archive.rb', line 134

def self.convert_data(object)
  if object.is_a?(Hash)
    object.each do |k, v|
      if v.is_a?(Hash) || v.is_a?(Array)
        convert_data(v)
      elsif v.is_a?(CBOR::Tagged)
        object[k] = make_cid_link(v)
      elsif v.is_a?(String) && v.encoding == Encoding::ASCII_8BIT
        object[k] = make_bytes(v)
      end
    end
  elsif object.is_a?(Array)
    object.each_with_index do |v, i|
      if v.is_a?(Hash) || v.is_a?(Array)
        convert_data(v)
      elsif v.is_a?(CBOR::Tagged)
        object[i] = make_cid_link(v)
      elsif v.is_a?(String) && v.encoding == Encoding::ASCII_8BIT
        object[i] = make_bytes(v)
      end
    end
  else
    raise DecodeError, "Unexpected value type in record: #{object}"
  end
end

.make_bytes(data) ⇒ Hash{String => String}

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Converts a binary string to an ATProto JSON $bytes object.

Parameters:

  • data (String) —

    binary data to encode

Returns:

  • (Hash{String => String}) —

    a $bytes JSON object



181
182
183
184
185
186
 
# File 'lib/oxygene/car_archive.rb', line 181

def self.make_bytes(data)
  string = Base64.strict_encode64(data)
  string.chomp!('=') while string.getbyte(-1) == 61

  { '$bytes' => string }
end

.make_cid_link(cid) ⇒ Hash{String => CID}

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Converts a CBOR CID tag to an ATProto JSON $link object.

Parameters:

  • cid (CBOR::Tagged) —

    CBOR tag containing the CID binary data

Returns:

  • (Hash{String => CID}) —

    a $link JSON object

Raises:

  • (DecodeError) —

    if the value is not a supported CID



169
170
171
 
# File 'lib/oxygene/car_archive.rb', line 169

def self.make_cid_link(cid)
  { '$link' => CID.from_cbor_tag(cid) }
end

Instance Method Details

#inspect ⇒ String

Returns a string with a representation of the object for debugging purposes.

Returns:

  • (String) 


191
192
193
194
195
196
197
198
199
200
201
 
# File 'lib/oxygene/car_archive.rb', line 191

def inspect
  vars = (instance_variables - [:@section_map]).map { |v|
    if v == :@sections && @buffer
      "#{v}=[...]"
    else
      "#{v}=#{instance_variable_get(v).inspect}"
    end
  }

  "#<#{self.class}:0x#{object_id} #{vars.join(", ")}>"
end

#parsed_sections ⇒ Array<CARSection>

Returns the list of archive sections that have been parsed so far, without parsing any more sections.

Returns:

  • (Array<CARSection>) —

    sections parsed so far, in archive order



95
96
97
 
# File 'lib/oxygene/car_archive.rb', line 95

def parsed_sections
  @sections.dup.freeze
end

#section_with_cid(cid, use_map: false, return_body: true) ⇒ Hash, ...

Looks up a section with a given CID in the archive, decoding sections as needed.

Sections are parsed lazily – if the requested section has already been loaded, it's returned without parsing any more parts of the archive, otherwise unread sections are parsed only until a match is found.

When making repeated lookups for sections in one archive, e.g. when walking the MST tree of a CAR repo, pass the use_map: true option, which tells CARArchive to build and use an index that maps CIDs to sections for quicker lookup (otherwise, parsed sections are searched sequentially). For performance, this isn't enabled by default, since the common case of processing firehose commit messages only does a single lookup to find the record data.

The CID may be passed as either an Oxygene::CID object, or its Oxygene::CID#cbor_form string.

Parameters:

  • cid (CID, String) —

    a CID object or its binary CBOR representation, including the leading null byte

  • use_map (Boolean) (defaults to: false) —

    whether to use and update the lookup index that maps CIDs to sections

  • return_body (Boolean) (defaults to: true) —

    whether to return the section's JSON body directly instead of an Oxygene::CARSection object

Returns:

  • (Hash, Array, CARSection, nil) —

    the requested section or its decoded body converted to ATProto JSON, or nil if not found

Raises:



78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/oxygene/car_archive.rb', line 78

def section_with_cid(cid, use_map: false, return_body: true)
  if found_section = find_parsed_section(cid, use_map)
    return (return_body ? found_section.json_body : found_section)
  end

  if found_section = parse_sections_until_match(cid, use_map)
    return (return_body ? found_section.json_body : found_section)
  end

  nil
end

#sections ⇒ Array<CARSection>

Parses all sections in the archive if they haven't been loaded yet, and returns the list of all sections in the order as listed in the archive.

Once all sections have been read, the original archive data buffer is released so it can be garbage-collected.

Returns:

  • (Array<CARSection>) —

    all sections in the archive

Raises:



109
110
111
112
113
114
115
116
117
118
119
120
 
# File 'lib/oxygene/car_archive.rb', line 109

def sections
  if @buffer
    if !@buffer.eof?
      read_section(@buffer) while !@buffer.eof?
      @map_needs_update = true
    end

    @buffer = nil
  end

  @sections
end