Module: CArray::ArrowTensor

Defined in:
lib/carray/arrow_tensor.rb

Overview

Reader / writer for Arrow's encapsulated tensor IPC message, for exchanging a dense numeric array with the Arrow ecosystem (pyarrow.ipc.write_tensor / read_tensor and equivalents).

Experimental: the method names, this namespace and the packaging are not frozen, and the layer may later move to a standalone, array-library-neutral gem. It is an interoperability format, not an archival one — an Arrow tensor carries no mask, attributes or data_class, so CArray.save / CArray.load remain the way to persist a CArray faithfully.

Only the numeric types an Arrow tensor admits are supported (int8..int64, uint8..uint64, float32, float64). Boolean, complex, fixlen and object arrays are rejected, as is a masked array — Arrow tensors have no validity concept, so the caller must be explicit (ca.value to drop the mask, ca.unmask_copy(fill) to bake it in, or save the mask separately).

Constant Summary collapse

DTYPE_TO_ARROW =

CArray data_type symbol -> Arrow element-type descriptor. [:float, precision] precision: 1 = single, 2 = double [:int, bit_width, signed?]

{
  float32: [:float, 1], float64: [:float, 2],
  int8:  [:int, 8,  true],  int16: [:int, 16, true],
  int32: [:int, 32, true],  int64: [:int, 64, true],
  uint8: [:int, 8,  false], uint16:[:int, 16, false],
  uint32:[:int, 32, false], uint64:[:int, 64, false],
}.freeze

Class Method Summary collapse

Class Method Details

.read(io) ⇒ Object

Read an Arrow tensor IPC message from +io+ and return a new CArray.



95
96
97
98
99
100
101
102
103
104
105
106
 
# File 'lib/carray/arrow_tensor.rb', line 95

def read (io)
  b = io.respond_to?(:read) ? io.read : io.to_s
  b = b.b
  meta = parse_metadata(b)
  type = meta[:data_type]
  unless CArray.respond_to?(:data_type_code) && DTYPE_TO_ARROW.key?(type)
    raise ArgumentError,
          "Arrow tensor element type #{type.inspect} has no CArray " \
          "counterpart"
  end
  build_carray(type, meta[:shape], meta[:strides], meta[:body])
end

.write(ca, io) ⇒ Object

Write +ca+ to +io+ as an Arrow tensor IPC message.

Raises:

  • (ArgumentError) 


58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/carray/arrow_tensor.rb', line 58

def write (ca, io)
  if ca.has_mask?
    raise ArgumentError,
          "cannot write a masked CArray as an Arrow tensor " \
          "(Arrow tensors have no validity concept); pass ca.value to " \
          "drop the mask, ca.unmask_copy(fill) to fill it, or use " \
          "CArray.save for a mask-preserving round-trip"
  end
  arrow = DTYPE_TO_ARROW[ca.data_type]
  unless arrow
    raise ArgumentError,
          "cannot write a #{ca.data_type} CArray as an Arrow tensor " \
          "(Arrow tensors are numeric only: int8..int64, uint8..uint64, " \
          "float32, float64)"
  end
  raise ArgumentError, "cannot write a 0-dimensional array" if ca.ndim < 1

  shape   = ca.shape
  elt     = ca.bytes
  strides = row_major_strides(shape, elt)

  body = StringIO.new("".b)
  ca.dump_binary(body)          # raw contiguous element bytes, row-major
  body = body.string

  meta = build_metadata(arrow, shape, strides, body.bytesize)
  pad  = (8 - ((8 + meta.bytesize) % 8)) % 8
  meta += "\0".b * pad

  io.write([0xFFFFFFFF].pack("V"))
  io.write([meta.bytesize].pack("l<"))
  io.write(meta)
  io.write(body)
  ca
end