Class: CArray::Serializer

Inherits:

Object

• Object
• CArray::Serializer

Defined in:

lib/carray/serialize.rb

Overview

:nodoc:

Constant Summary

PEP_TO_SYMBOL =

Canonical primitive-type notation for the trailer data_class schema. The .ca format owns this mapping: it is the same bare PEP 3118 notation CArray's MemoryView layer emits via ca_mv_format_for, frozen here as the format's own copy so a future MemoryView producer flip does not silently change the on-disk spec. Non-primitive member types (:fixlen / :object / nested class / CArray template / :bitfield) are absent, so SYMBOL_TO_PEP[type] is nil for them, which enforces the flat-primitive-only rule for free (a non-expressible member raises on save).

{
  "?"  => :boolean,
  "b"  => :int8,   "B" => :uint8,
  "h"  => :int16,  "H" => :uint16,
  "i"  => :int32,  "I" => :uint32,
  "q"  => :int64,  "Q" => :uint64,
  "f"  => :float32, "d" => :float64,
  "Zf" => :cmplx64, "Zd" => :cmplx128,
}.freeze

Class Method Summary

• .pack_header(h, file_endian) ⇒ Object

  Pack a header field Hash into a HEADER_BYTES buffer, integers in file_endian byte order.

• .unpack_header(buf, file_endian) ⇒ Object

  Parse a HEADER_BYTES buffer (integers in file_endian) into a field Hash.

Instance Method Summary

• #initialize(io) ⇒ Serializer constructor

  Allocates a Serializer around io.

• #load(data_type: nil) ⇒ CArray

  Reads a _CARRAY3 payload from the wrapped IO and returns the reconstructed array, re-wrapping it through its recorded data_class when the trailer carries one.

• #save(ca, endian: CArray.endian) ⇒ CArray

  Writes ca to the wrapped IO in the _CARRAY3 portable format.

Constructor Details

#initialize(io) ⇒ Serializer

Returns a new instance of Serializer.

Allocates a Serializer around io. A String is wrapped in a StringIO; anything else is used directly.

Parameters:

• io (IO, String) —

  destination or source.

# File 'lib/carray/serialize.rb', line 101

def initialize (io)
  case io
  when String
    @io = StringIO.new(io)
  else
    @io = io
  end
end

Class Method Details

.pack_header(h, file_endian) ⇒ Object

Pack a header field Hash into a HEADER_BYTES buffer, integers in file_endian byte order.

# File 'lib/carray/serialize.rb', line 386

def pack_header (h, file_endian)
  e     = (file_endian == CA_LITTLE_ENDIAN) ? "<" : ">"
  shape = h[:shape]
  body = [
    MAGIC,               # a8   0
    ENDIAN_MARKER,       # L    8
    VERSION_MAJOR,       # C    12
    VERSION_MINOR,       # C    13
    HEADER_BYTES,        # S    14
    h[:has_mask],        # C    16
    h[:has_trailer],     # C    17
    h[:data_type_code],  # C    18
    h[:ndim],            # C    19
    0,                   # L    20  reserved0
    *shape,              # q16  24
    h[:element_bytes],   # L    152
    h[:flags],           # l    156
    h[:elements],        # Q    160
    h[:data_offset],     # Q    168
    h[:data_bytes],      # Q    176
    h[:mask_offset],     # Q    184
    h[:mask_bytes],      # Q    192
    h[:trailer_offset],  # Q    200
    h[:trailer_bytes],   # Q    208
    0,                   # Q    216  data_checksum
    0,                   # C    224  checksum_algo
  ].pack("a8L#{e}CCS#{e}CCCCL#{e}q#{e}16L#{e}l#{e}Q#{e}Q#{e}Q#{e}Q#{e}Q#{e}Q#{e}Q#{e}Q#{e}C")
  body.ljust(HEADER_BYTES, "\x00")
end

.unpack_header(buf, file_endian) ⇒ Object

Parse a HEADER_BYTES buffer (integers in file_endian) into a field Hash. Byte-slice unpack mirrors the C struct offsets.

# File 'lib/carray/serialize.rb', line 418

def unpack_header (buf, file_endian)
  e = (file_endian == CA_LITTLE_ENDIAN) ? "<" : ">"
  {
    endian_marker:  buf[8, 4].unpack1("L#{e}"),
    version_major:  buf[12].ord,
    version_minor:  buf[13].ord,
    header_bytes:   buf[14, 2].unpack1("S#{e}"),
    has_mask:       buf[16].ord,
    has_trailer:    buf[17].ord,
    data_type_code: buf[18].ord,
    ndim:           buf[19].ord,
    shape:          buf[24, 128].unpack("q#{e}16"),
    element_bytes:  buf[152, 4].unpack1("L#{e}"),
    flags:          buf[156, 4].unpack1("l#{e}"),
    elements:       buf[160, 8].unpack1("Q#{e}"),
    data_offset:    buf[168, 8].unpack1("Q#{e}"),
    data_bytes:     buf[176, 8].unpack1("Q#{e}"),
    mask_offset:    buf[184, 8].unpack1("Q#{e}"),
    mask_bytes:     buf[192, 8].unpack1("Q#{e}"),
    trailer_offset: buf[200, 8].unpack1("Q#{e}"),
    trailer_bytes:  buf[208, 8].unpack1("Q#{e}"),
    data_checksum:  buf[216, 8].unpack1("Q#{e}"),
    checksum_algo:  buf[224].ord,
  }
end

Instance Method Details

#load(data_type: nil) ⇒ CArray

Reads a _CARRAY3 payload from the wrapped IO and returns the reconstructed array, re-wrapping it through its recorded data_class when the trailer carries one.

Parameters:

• opt (Hash) —

  :data_type overrides the element type for a bare CA_FIXLEN payload.

Returns:

• (CArray)

Raises:

• (RuntimeError) —

  on a bad magic string, corrupt header, or unsupported version.

# File 'lib/carray/serialize.rb', line 194

def load (**opt)
  buf = @io.read(HEADER_BYTES)
  unless buf && buf.bytesize == HEADER_BYTES
    raise "not a CArray binary data (truncated header)"
  end
  if buf[0, 8] != MAGIC
    raise "not a CArray binary data (bad magic)"
  end

  # The byte order is self-describing via endian_marker at offset 8:
  # the writer stored 0x01020304 in file order, so the raw bytes are
  # 01 02 03 04 on a big-endian file and 04 03 02 01 on a little-endian
  # one.  The leading byte alone decides; anything else is corrupt.
  case buf[8].ord
  when 0x01 then file_endian = CA_BIG_ENDIAN
  when 0x04 then file_endian = CA_LITTLE_ENDIAN
  else
    raise "corrupt CArray binary data (endian marker mismatch)"
  end
  h = self.class.unpack_header(buf, file_endian)

  if h[:endian_marker] != ENDIAN_MARKER
    raise "corrupt CArray binary data (endian marker mismatch)"
  end
  if h[:version_major] != VERSION_MAJOR
    raise "unsupported CArray binary version #{h[:version_major]}.#{h[:version_minor]}"
  end
  if h[:header_bytes] != HEADER_BYTES
    raise "unsupported CArray binary header size #{h[:header_bytes]}"
  end
  if h[:data_bytes] != h[:elements] * h[:element_bytes]
    raise "corrupt CArray binary data (data_bytes cross-check failed)"
  end

  swap      = (file_endian != CArray.endian)
  data_type = h[:data_type_code]
  ndim      = h[:ndim]
  dim       = h[:shape][0, ndim]
  bytes     = h[:element_bytes]

  if opt[:data_type] and data_type == CArray.data_type_code(CA_FIXLEN)
    data_type = opt[:data_type]
  end

  ca = CArray.new(data_type, dim, :bytes => bytes)
  ca.load_binary(@io)
  ca[] = ca.swap_bytes if swap

  if h[:has_mask] != 0
    ca.mask = 0
    ca.mask.load_binary(@io)
  end

  if h[:trailer_bytes] > 0
    trailer_raw = @io.read(h[:trailer_bytes])
    trailer = decode_trailer(trailer_raw)
    ca = apply_trailer(ca, trailer)
  end

  return ca
end

#save(ca, endian: CArray.endian) ⇒ CArray

Writes ca to the wrapped IO in the _CARRAY3 portable format. The data region is raw contiguous bytes; attribute Hash and data_class schema (if any) ride in a YAML trailer at the tail.

Parameters:

• ca (CArray) —

  array to write.

• opt (Hash) —

  :endian forces the output byte order (defaults to the host's; a differing value swaps data + header).

Returns:

• (CArray) —

  ca.

Raises:

• (ArgumentError) —

  when ca is a CA_OBJECT array (use Marshal.dump(ca) instead), or carries a data_class the v1.0 flat-primitive schema cannot express.

# File 'lib/carray/serialize.rb', line 121

def save (ca, **opt)
  if ca.data_type == :object
    raise ArgumentError,
          "CArray.save cannot serialise a CA_OBJECT array " \
          "(arbitrary Ruby objects have no portable representation); " \
          "use Marshal.dump(ca) for a Ruby-only round-trip"
  end

  file_endian = opt[:endian] || CArray.endian
  swap        = (file_endian != CArray.endian)

  elements     = ca.elements
  element_bytes = ca.bytes
  data_bytes   = elements * element_bytes
  has_mask     = ca.has_mask?

  # Trailer is present only when there is semantic content: an
  # attribute Hash or a data_class schema.  A plain numeric array
  # writes no trailer (both trailer fields zero), keeping the
  # C / FORTRAN reader path a header + raw data.
  trailer = build_trailer(ca)
  trailer_str = trailer.empty? ? nil : encode_trailer(trailer)

  data_offset    = HEADER_BYTES
  mask_offset    = has_mask ? data_offset + data_bytes : 0
  mask_bytes     = has_mask ? elements : 0
  tail_offset    = has_mask ? mask_offset + mask_bytes : data_offset + data_bytes
  trailer_offset = trailer_str ? tail_offset : 0
  trailer_bytes  = trailer_str ? trailer_str.bytesize : 0

  dim = ca.shape
  dim = dim + Array.new(CA_RANK_MAX - dim.size, 0)

  header = {
    has_mask:       has_mask ? 1 : 0,
    has_trailer:    trailer_str ? 1 : 0,
    data_type_code: CArray.data_type_code(ca.data_type),
    ndim:           ca.ndim,
    shape:          dim,
    element_bytes:  element_bytes,
    flags:          ca.flags,
    elements:       elements,
    data_offset:    data_offset,
    data_bytes:     data_bytes,
    mask_offset:    mask_offset,
    mask_bytes:     mask_bytes,
    trailer_offset: trailer_offset,
    trailer_bytes:  trailer_bytes,
  }

  @io.write(self.class.pack_header(header, file_endian))

  data_ca = swap ? ca.swap_bytes : ca
  data_ca.dump_binary(@io)

  if has_mask
    ca.mask.dump_binary(@io)   # int8, endian-neutral
  end

  @io.write(trailer_str) if trailer_str

  return ca
end