NDAV - N-Dimensional Array View

A thin wrapper around MemoryView ("buffer protocol" for Ruby).

It provides an interoperability layer for multi-dimensional arrays which can be shared between libraries.

SYNOPSIS

waveform, sample_rate = TorchAudio.load("path/to/audio.wav")

# Convert Torch::Tensor to NDAV
# so that you can convert it to OrtValue,
# a data format for ONNX Runtime
input = waveform
          .to_ndav
          .to_ort_value

# Make ONNX Runtime return result as OrtValue
outputs = OnnxRuntime::Session.new("path/to/model.onnx")
            .run(
              [:output_name],
              {input_name: input},
              output_type: :ort_value
            )

# You may convert OrtValue to Torch::Tensor via NDAV
output_tensor = outputs[0]
                  .to_ndav
                  .to_torch_tensor # converts back to Torch::Tensor

TorchAudio.save("path/to/output.wav", output_tensor, sample_rate)

ABSTRACT

NDAV acts as an interoperability layer between multi-dimensional arrays including images, audio and tensors such as Numo::NArray, Torch.rb's Torch::Tensor, ONNX Runtime Ruby's OnnxRuntime::OrtValue, Red Arrow's Arrow::Array and so on.

It allows data to be shared without copying.

BACKGROUND

In the modern Ruby community, Numo::NArray is often used for data conversion. But, data are copied when converting to and from Numo::NArray. In addition, Numo::NArray neither exports nor accepts MemoryView.

Red Arrow is also used and it can export MemoryView from Arrow::Array (not from Arrow::Tensor, though). It also can be converted to and from Numo::NArray using Red Arrow Numo::NArray. But, in real-world usage, we need, for example, to convert data with many hops:

Torch::Tensor -> Numo::NArray -> Red Arrow -> MemoryView -> some process...

It might not be difficult, but a little bit cumbersome. Additionally, Red Arrow doesn't accept MemoryView.

USAGE

ndav gem is just a base library. You need to install bridges as well. Say, assume you want to make conversions between Numo::NArray each other.

require "numo/narray"
require "ndav"
require "ndav/numo/narray"

numo = Numo::SFloat.new(3, 5).seq # => Numo::SFloat
ndav = numo.to_ndav               # => NDAV

numo = Numo::SFloat.from_ndav(ndav) # => Numo::SFloat
ndav = NDAV.from_numo_narray(numo)  # => NDAV

include NDAV::Converter
numo = NumoNArray(ndav) # => Numo::SFloat
ndav = NDAV(numo)       # => NDAV

For Torch::Tensor and OnnxRuntime::OrtValue, you can do the same operation, therefore you may convert them to each other, like this:

numo
  .to_ndav
  .to_torch_tensor
  .then {|torch_tensor| some_process(torch_tensor)}
  .to_ndav
  .to_ort_value
  .then {|ort_value|
    OnnxRuntime::Session.new("model.onnx")
      .run(
        [:output],
        {input: ort_value},
        output_type: :ort_value
      )[0]
  }
  .to_ndav
  .to_torch_tensor
  .then {|torch_tensor| TorchAudio.save(torch_tensor, sample_rate)}

Working With MemoryView

NDAV can be initialized directly from libraries which export MemoryView such as Red Arrow, without any bridge library:

arrow = Arrow::Int16Array.new([1, 2, 3])
ndav = NDAV.new(arrow)

On the other hand, it also exports MemoryView. You can pass NDAV arrays directly to methods which accept MemoryView such as whispercpp without bridge libraries:

waveform, sample_rate = TorchAudio.load("path/to/audio.wav")
samples = waveform.to_ndav
whisper.full(params, samples)

Notice On Memory Sharing

Notice that NDAV is just a memory view and libraries share a memory address. If you change source data destructively, it affects converted data.

Additionally, you potentially encounter odd data corruption or segmentation fault. These might be bugs in bridge libraries such as ndav-numo-narray. As a user, you don't need to worry about this kind of memory management, but it's worth knowing such situations may occur.

INSTALLATION

% gem install ndav

or,

% bundle add ndav

But, you need bridges for real-world use. See each bridge's documentation for individual requirements.

BRIDGES

There are some bridges using NDAV:

• ndav/ffi: FFI::MemoryPointer, FFI::Pointer <-> NDAV
• ndav-numo-narray: Numo::NArray <-> NDAV
• ndav-ort_value: OnnxRuntime::OrtValue <-> NDAV
• ndav-torch-tensor: Torch::Tensor <-> NDAV
• onnxruntime-torch-tensor: Torch::Tensor <-> OnnxRuntime::OrtValue via NDAV

APPLICATIONS AND LIBRARIES WHICH USE NDAV

• GTCRN - An audio speech enhancement (noise reduction) library. It uses TorchAudio for loading and preprocessing audio before passing it to ONNX Runtime, then performs post-processing on the result and writes it back to a file.
• Itak - An audio processing tool for podcasters. After reducing noise with the GTCRN mentioned above, it uses the VAD function of whispercpp to remove silent periods. Although whispercpp does not accept existing tensor libraries, it does accept MemoryView, so it can be passed via NDAV.

CREATING BRIDGES

Refer to existing bridge implementations listed above to create your bridge.

The points are:

• Implement FromNDAV#from_ndav, ToNDAV#to_ndav, register them, and NDAV.from_your_data and NDAV#to_your_data are automatically derived
• When initializing NDAV object from your object, use lifetime keyword argument for NDAV#initialize effectively to prevent Ruby from GCing your object, which would lead to a dangling pointer
• When initializing your object from NDAV object, keep NDAV object alive to prevent Ruby from GCing NDAV object, which would lead to a dangling pointer, ndav-numo-narray, for instance, embeds the NDAV object in an instance variable

An advantage of NDAV over raw MemoryView is that you can write bridges in pure Ruby in most cases. It helps prototyping and experimentation. As an exception, I had to write C code for ndav-numo-narray because Numo::NArray only provides methods that access data by copying and does not directly expose its raw data pointer to Ruby API. However, in even such case, pure Ruby bridge remains a viable option for prototyping and experiments where a single initial copy is acceptable.

If you are a library author, I want you to consider making your library work with MemoryView instead of creating an NDAV bridge.

FUTURE

If MemoryView gets popular enough in the Ruby ecosystem, this library will end its role and will no longer be needed. I hope such future.

LICENSE

BSD-2-Clause license. See LICENSE file.