Class: VectorAmp::DatasetsResource

Inherits:
Object
  • Object
show all
Defined in:
lib/vector_amp/datasets.rb

Overview

Dataset API resource for create/list/get/search/insert operations.

Instance Method Summary collapse

Constructor Details

#initialize(transport, client: nil) ⇒ DatasetsResource

Parameters:

  • transport (#request) —

    API transport.

  • client (Client, nil) (defaults to: nil) —

    optional client attached to returned Dataset objects.



14
15
16
17
 
# File 'lib/vector_amp/datasets.rb', line 14

def initialize(transport, client: nil)
  @transport = transport
  @client = client
end

Instance Method Details

#add_texts(dataset_id, texts_arg = nil, texts: nil, ids: nil, metadata: nil) ⇒ Hash

Embed and insert texts into a dataset.

Accepts a single string or a list of strings. Ids are auto-generated when omitted; supplied ids may be strings or integers, and integer ids are preserved as JSON numbers. The source text is copied into metadata.text.

Parameters:

  • dataset_id (String) —

    dataset id.

  • texts_arg (String, Array<String>, nil) (defaults to: nil) —

    positional text(s) for convenience.

  • texts (String, Array<String>, nil) (defaults to: nil) —

    keyword text(s).

  • ids (Array, nil) (defaults to: nil) —

    optional ids; generated UUIDs when omitted.

  • metadata (Hash, Array<Hash>, nil) (defaults to: nil) —

    metadata applied to all texts or per-text.

Returns:

  • (Hash) —

    insert response.

Raises:

  • (ArgumentError) 


201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
 
# File 'lib/vector_amp/datasets.rb', line 201

def add_texts(dataset_id, texts_arg = nil, texts: nil, ids: nil, metadata: nil)
  texts = Array(texts || texts_arg)
  raise ArgumentError, "texts must not be empty" if texts.empty?
  raise ArgumentError, "ids length must match texts length" if ids && ids.length != texts.length
  if metadata.is_a?(Array) && metadata.length != texts.length
    raise ArgumentError, "metadata length must match texts length"
  end

  response = embed(dataset_id, texts: texts)
  embeddings = response.fetch("embeddings")
  vectors = texts.each_with_index.map do |text, index|
    item_metadata = metadata.is_a?(Array) ? metadata[index] : metadata
    {
      id: ids ? Utils.coerce_vector_id(ids[index]) : SecureRandom.uuid,
      values: embeddings[index],
      metadata: Utils.compact_hash((item_metadata || {}).merge(text: text))
    }
  end
  insert(dataset_id, vectors: vectors)
end

#create(name:, dim: nil, embedding: nil, metric: "cosine", hybrid: nil, filters: nil, metadata_schema: nil, tuning: nil, metadata: nil, **unknown) ⇒ Dataset

Create a SABLE dataset. index_type is managed by the SDK and always sent as sable.

Only name is required. The embedding defaults to the managed vectoramp/VectorAmp-Embedding-4B model and the dimension is inferred (2560 for the default model). Pass embedding: (a Hash or model String, or the result of Embedding.openai) to use a different model; if the model's dimension is not known to the SDK you must also pass dim:.

Parameters:

  • name (String) —

    dataset name.

  • dim (Integer, nil) (defaults to: nil) —

    embedding/vector dimension; inferred when omitted for known models.

  • embedding (Hash, String, nil) (defaults to: nil) —

    embedding configuration; defaults to the managed VectorAmp model.

  • metric (String) (defaults to: "cosine") —

    distance metric; defaults to cosine.

  • hybrid (Boolean, nil) (defaults to: nil) —

    enable hybrid (dense + sparse) search; sends hybrid: true.

  • filters (Hash, nil) (defaults to: nil) —

    optional filter schema/config.

  • metadata_schema (Hash, nil) (defaults to: nil) —

    optional metadata schema.

  • tuning (Hash, nil) (defaults to: nil) —

    optional SABLE tuning parameters.

  • metadata (Hash, nil) (defaults to: nil) —

    optional dataset metadata.

Returns:

Raises:

  • (ArgumentError) —

    when index_type is supplied, dim cannot be inferred, or unknown options are passed.



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/vector_amp/datasets.rb', line 67

def create(name:, dim: nil, embedding: nil, metric: "cosine", hybrid: nil, filters: nil, metadata_schema: nil, tuning: nil, metadata: nil, **unknown)
  if unknown.key?(:index_type) || unknown.key?("index_type")
    raise ArgumentError, "index_type is managed by VectorAmp Ruby SDK and is always 'sable'"
  end
  Utils.ensure_no_unknown!(unknown, "create")

  resolved_embedding = Embedding.normalize(embedding)
  resolved_dim = dim || Embedding.infer_dim(resolved_embedding)
  if resolved_dim.nil?
    model = resolved_embedding[:model] || resolved_embedding["model"]
    raise ArgumentError, "dim is required for embedding model #{model.inspect}; pass dim: explicitly"
  end

  body = Utils.compact_hash(
    name: name,
    dim: resolved_dim,
    metric: metric,
    embedding: resolved_embedding,
    index_type: "sable",
    hybrid: hybrid,
    filters: filters,
    metadata_schema: metadata_schema,
    tuning: tuning,
    metadata: metadata
  )
  wrap_dataset(@transport.request(:post, "/datasets", body: body))
end

#delete(dataset_id) ⇒ Hash

Delete a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:

  • (Hash) —

    delete response.



37
38
39
 
# File 'lib/vector_amp/datasets.rb', line 37

def delete(dataset_id)
  @transport.request(:delete, "/datasets/#{dataset_id}")
end

#download_document(dataset_id, document_id) ⇒ String

Download the retained original bytes for a dataset document. The HTTP transport follows redirects so this returns the final raw object bytes.

Parameters:

  • dataset_id (String) —

    dataset id.

  • document_id (String) —

    document id returned by #list_documents.

Returns:

  • (String) —

    raw document bytes.



111
112
113
 
# File 'lib/vector_amp/datasets.rb', line 111

def download_document(dataset_id, document_id)
  @transport.request(:get, "/datasets/#{dataset_id}/documents/#{document_id}/download", raw: true, headers: { Accept: "*/*" })
end

#embed(dataset_id, text: nil, texts: nil) ⇒ Hash

Generate embeddings using the dataset embedding configuration.

Parameters:

  • dataset_id (String) —

    dataset id.

  • text (String, nil) (defaults to: nil) —

    single text to embed.

  • texts (Array<String>, nil) (defaults to: nil) —

    multiple texts to embed.

Returns:

  • (Hash) —

    embed response.

Raises:

  • (ArgumentError) 


184
185
186
187
188
 
# File 'lib/vector_amp/datasets.rb', line 184

def embed(dataset_id, text: nil, texts: nil)
  raise ArgumentError, "provide text or texts" if text.nil? && texts.nil?

  @transport.request(:post, "/datasets/#{dataset_id}/embed", body: Utils.compact_hash(text: text, texts: texts))
end

#get(dataset_id) ⇒ Dataset

Fetch a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:



30
31
32
 
# File 'lib/vector_amp/datasets.rb', line 30

def get(dataset_id)
  wrap_dataset(@transport.request(:get, "/datasets/#{dataset_id}"))
end

#insert(dataset_id, vectors:) ⇒ Hash

Insert vectors into a dataset.

Vector record ids may be strings or integers. Integer ids are preserved as JSON numbers (not coerced to strings) so the API does not rewrite them.

Parameters:

  • dataset_id (String) —

    dataset id.

  • vectors (Array<Hash>) —

    vector records with ids, values, and optional metadata.

Returns:

  • (Hash) —

    insert response.



175
176
177
 
# File 'lib/vector_amp/datasets.rb', line 175

def insert(dataset_id, vectors:)
  @transport.request(:post, "/datasets/#{dataset_id}/insert", body: { vectors: Utils.normalize_vectors(vectors) })
end

#list(limit: 50, offset: 0) ⇒ Hash

List datasets.

Parameters:

  • limit (Integer) (defaults to: 50) —

    page size; defaults to 50.

  • offset (Integer) (defaults to: 0) —

    page offset; defaults to 0.

Returns:

  • (Hash) —

    response envelope with datasets wrapped as Dataset objects when present.



23
24
25
 
# File 'lib/vector_amp/datasets.rb', line 23

def list(limit: 50, offset: 0)
  wrap_list(@transport.request(:get, "/datasets", query: { limit: limit, offset: offset }))
end

#list_documents(dataset_id, limit: 50, cursor: nil, status: nil) ⇒ Hash

List retained source documents for a dataset using cursor pagination.

Parameters:

  • dataset_id (String) —

    dataset id.

  • limit (Integer, nil) (defaults to: 50) —

    maximum documents to return.

  • cursor (String, nil) (defaults to: nil) —

    cursor from a previous response's next_cursor.

  • status (String, nil) (defaults to: nil) —

    optional document status filter, e.g. ready.

Returns:

  • (Hash) —

    response envelope with documents and next_cursor.



102
103
104
 
# File 'lib/vector_amp/datasets.rb', line 102

def list_documents(dataset_id, limit: 50, cursor: nil, status: nil)
  @transport.request(:get, "/datasets/#{dataset_id}/documents", query: Utils.compact_hash(limit: limit, cursor: cursor, status: status))
end

#search(dataset_id, query_text_or_options = nil, query: nil, query_text: nil, search_text: nil, top_k: 10, filters: nil, advanced_filters: nil, embedding_model: nil, embedding_provider: nil, nprobe_override: nil, rerank_depth_override: nil, hybrid: nil, sparse_query: nil, alpha: nil, include_embeddings: nil, include_documents: nil, include_metadata: nil, rerank: nil, **unknown) ⇒ Hash

Search a dataset by text or vector query.

Parameters:

  • dataset_id (String) —

    dataset id.

  • query_text_or_options (String, Hash, nil) (defaults to: nil) —

    text query or legacy options hash.

  • query (Array<Numeric>, nil) (defaults to: nil) —

    vector query.

  • query_text (String, nil) (defaults to: nil) —

    explicit text query; overrides positional text when provided.

  • search_text (String, nil) (defaults to: nil) —

    alias for query_text for one-field hybrid/BM25 search.

  • top_k (Integer) (defaults to: 10) —

    number of results; defaults to 10.

  • filters (Hash, nil) (defaults to: nil) —

    metadata filters.

  • advanced_filters (Hash, nil) (defaults to: nil) —

    advanced filter expression.

  • embedding_model (String, nil) (defaults to: nil) —

    optional embedding model override.

  • embedding_provider (String, nil) (defaults to: nil) —

    optional embedding provider override.

  • nprobe_override (Integer, nil) (defaults to: nil) —

    optional SABLE search probe override.

  • rerank_depth_override (Integer, nil) (defaults to: nil) —

    optional rerank depth override.

  • hybrid (Boolean, nil) (defaults to: nil) —

    enable hybrid search when supported.

  • sparse_query (Hash, nil) (defaults to: nil) —

    sparse query for hybrid search.

  • alpha (Numeric, nil) (defaults to: nil) —

    dense/sparse weighting for hybrid search.

  • include_embeddings (Boolean, nil) (defaults to: nil) —

    include vector values in results.

  • include_documents (Boolean, nil) (defaults to: nil) —

    include document fields in results.

  • include_metadata (Boolean, nil) (defaults to: nil) —

    include metadata; API default is true.

  • rerank (Boolean, Hash, nil) (defaults to: nil) —

    enable semantic reranking. true or { enabled: true } uses vectoramp / VectorAmp-Rerank-v1.

Returns:

  • (Hash) —

    search response.

Raises:

  • (ArgumentError) 


136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/vector_amp/datasets.rb', line 136

def search(dataset_id, query_text_or_options = nil, query: nil, query_text: nil, search_text: nil, top_k: 10, filters: nil, advanced_filters: nil,
           embedding_model: nil, embedding_provider: nil, nprobe_override: nil, rerank_depth_override: nil,
           hybrid: nil, sparse_query: nil, alpha: nil, include_embeddings: nil, include_documents: nil,
           include_metadata: nil, rerank: nil, **unknown)
  if query_text_or_options.is_a?(Hash)
    unknown = query_text_or_options.merge(unknown)
    query_text_or_options = nil
  end
  Utils.ensure_no_unknown!(unknown, "search")
  raise ArgumentError, "provide query_text or search_text, not both" if query_text && search_text
  resolved_query_text = query_text || search_text || query_text_or_options
  body = Utils.compact_hash(
    query: query,
    query_text: resolved_query_text,
    top_k: top_k,
    filters: filters,
    advanced_filters: advanced_filters,
    embedding_model: embedding_model,
    embedding_provider: embedding_provider,
    nprobe_override: nprobe_override,
    rerank_depth_override: rerank_depth_override,
    hybrid: hybrid,
    sparse_query: sparse_query,
    alpha: alpha,
    include_embeddings: include_embeddings,
    include_documents: include_documents,
    include_metadata: include_metadata,
    rerank: rerank
  )
  @transport.request(:post, "/datasets/#{dataset_id}/search", body: body)
end

#stats(dataset_id) ⇒ Hash

Fetch dataset statistics.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:

  • (Hash) —

    stats response.



44
45
46
 
# File 'lib/vector_amp/datasets.rb', line 44

def stats(dataset_id)
  @transport.request(:get, "/datasets/#{dataset_id}/stats")
end