Class: VectorAmp::DatasetsResource

Inherits:
Object
  • Object
show all
Defined in:
lib/vector_amp/datasets.rb

Overview

Dataset API resource for create/list/get/search/insert operations.

Instance Method Summary collapse

Constructor Details

#initialize(transport, client: nil) ⇒ DatasetsResource

Parameters:

  • transport (#request) —

    API transport.

  • client (Client, nil) (defaults to: nil) —

    optional client attached to returned Dataset objects.



14
15
16
17
 
# File 'lib/vector_amp/datasets.rb', line 14

def initialize(transport, client: nil)
  @transport = transport
  @client = client
end

Instance Method Details

#add_texts(dataset_id, texts_arg = nil, texts: nil, ids: nil, metadata: nil) ⇒ Hash

Embed and insert texts into a dataset.

Accepts a single string or a list of strings. Ids are auto-generated when omitted; supplied ids may be strings or integers, and integer ids are preserved as JSON numbers. The source text is copied into metadata.text.

Parameters:

  • dataset_id (String) —

    dataset id.

  • texts_arg (String, Array<String>, nil) (defaults to: nil) —

    positional text(s) for convenience.

  • texts (String, Array<String>, nil) (defaults to: nil) —

    keyword text(s).

  • ids (Array, nil) (defaults to: nil) —

    optional ids; generated UUIDs when omitted.

  • metadata (Hash, Array<Hash>, nil) (defaults to: nil) —

    metadata applied to all texts or per-text.

Returns:

  • (Hash) —

    insert response.

Raises:

  • (ArgumentError) 


254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/vector_amp/datasets.rb', line 254

def add_texts(dataset_id, texts_arg = nil, texts: nil, ids: nil, metadata: nil)
  texts = Array(texts || texts_arg)
  raise ArgumentError, "texts must not be empty" if texts.empty?
  raise ArgumentError, "ids length must match texts length" if ids && ids.length != texts.length
  if metadata.is_a?(Array) && metadata.length != texts.length
    raise ArgumentError, "metadata length must match texts length"
  end

  response = embed(dataset_id, texts: texts)
  embeddings = response.fetch("embeddings")
  vectors = texts.each_with_index.map do |text, index|
    item_metadata = metadata.is_a?(Array) ? metadata[index] : metadata
    {
      id: ids ? Utils.coerce_vector_id(ids[index]) : SecureRandom.uuid,
      values: embeddings[index],
      metadata: Utils.compact_hash((item_metadata || {}).merge(text: text))
    }
  end
  insert(dataset_id, vectors: vectors)
end

#create(name:, dim: nil, embedding: nil, metric: "cosine", hybrid: nil, filters: nil, metadata_schema: nil, tuning: nil, metadata: nil, **unknown) ⇒ Dataset

Create a SABLE dataset. index_type is managed by the SDK and always sent as sable.

Only name is required. The embedding defaults to the managed vectoramp/VectorAmp-Embedding-4B model and the dimension is inferred (2560 for the default model). Pass embedding: (a Hash or model String, or the result of Embedding.openai) to use a different model; if the model's dimension is not known to the SDK you must also pass dim:.

Parameters:

  • name (String) —

    dataset name.

  • dim (Integer, nil) (defaults to: nil) —

    embedding/vector dimension; inferred when omitted for known models.

  • embedding (Hash, String, nil) (defaults to: nil) —

    embedding configuration; defaults to the managed VectorAmp model.

  • metric (String) (defaults to: "cosine") —

    distance metric; defaults to cosine.

  • hybrid (Boolean, nil) (defaults to: nil) —

    enable hybrid (dense + sparse) search; sends hybrid: true.

  • filters (Hash, nil) (defaults to: nil) —

    optional filter schema/config.

  • metadata_schema (Array<Hash>, Hash, nil) (defaults to: nil) —

    optional metadata schema.

  • tuning (Hash, nil) (defaults to: nil) —

    optional SABLE tuning parameters.

  • metadata (Hash, nil) (defaults to: nil) —

    optional dataset metadata.

Returns:

Raises:

  • (ArgumentError) —

    when index_type is supplied, dim cannot be inferred, or unknown options are passed.



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/vector_amp/datasets.rb', line 67

def create(name:, dim: nil, embedding: nil, metric: "cosine", hybrid: nil, filters: nil, metadata_schema: nil, tuning: nil, metadata: nil, **unknown)
  if unknown.key?(:index_type) || unknown.key?("index_type")
    raise ArgumentError, "index_type is managed by VectorAmp Ruby SDK and is always 'sable'"
  end
  Utils.ensure_no_unknown!(unknown, "create")

  resolved_embedding = Embedding.normalize(embedding)
  resolved_dim = dim || Embedding.infer_dim(resolved_embedding)
  if resolved_dim.nil?
    model = resolved_embedding[:model] || resolved_embedding["model"]
    raise ArgumentError, "dim is required for embedding model #{model.inspect}; pass dim: explicitly"
  end

  body = Utils.compact_hash(
    name: name,
    dim: resolved_dim,
    metric: metric,
    embedding: resolved_embedding,
    index_type: "sable",
    hybrid: hybrid,
    filters: filters,
    schema: normalize_metadata_schema(metadata_schema),
    tuning: tuning,
    metadata: metadata
  )
  wrap_dataset(@transport.request(:post, "/datasets", body: body))
end

#create_with_openai_api_key(name:, api_key:, size: "small", secret_ref: "emb:openai:api_key", validate: false, **options) ⇒ Dataset

Store/update an OpenAI API key in org secrets, then create an OpenAI-backed dataset whose embedding config references that secret.

Parameters:

  • name (String) —

    dataset name.

  • api_key (String) —

    OpenAI API key to store server-side.

  • size (String, Symbol) (defaults to: "small") —

    "small" or "large".

  • secret_ref (String) (defaults to: "emb:openai:api_key") —

    org secret reference to write and attach to the dataset.

  • validate (Boolean) (defaults to: false) —

    validate the key before storing.

  • options (Hash) —

    forwarded to #create, e.g. hybrid: true or metadata:.

Returns:



104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/vector_amp/datasets.rb', line 104

def create_with_openai_api_key(name:, api_key:, size: "small", secret_ref: "emb:openai:api_key", validate: false, **options)
  secrets = @client&.org_secrets || begin
    require_relative "org_secrets"
    OrgSecretsResource.new(@transport)
  end
  secrets.put_openai_api_key(
    api_key: api_key,
    secret_ref: secret_ref,
    validate: validate,
    model: openai_model(size)
  )
  create(name: name, embedding: Embedding.openai(size, secret_ref: secret_ref), **options)
end

#delete(dataset_id) ⇒ Hash

Delete a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:

  • (Hash) —

    delete response.



37
38
39
 
# File 'lib/vector_amp/datasets.rb', line 37

def delete(dataset_id)
  @transport.request(:delete, "/datasets/#{dataset_id}")
end

#delete_vectors(dataset_id, ids:, write_concern: nil) ⇒ Hash

Delete one or more vectors from a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

  • ids (Array<String,Integer>) —

    vector ids to delete.

  • write_concern (String, nil) (defaults to: nil) —

    optional API write concern.

Returns:

  • (Hash) —

    delete response with deleted and dataset_id.

Raises:

  • (ArgumentError) 


222
223
224
225
226
227
228
229
230
 
# File 'lib/vector_amp/datasets.rb', line 222

def delete_vectors(dataset_id, ids:, write_concern: nil)
  raise ArgumentError, "ids must not be empty" if ids.nil? || ids.empty?

  @transport.request(
    :delete,
    "/datasets/#{dataset_id}/vectors",
    body: Utils.compact_hash(ids: ids, write_concern: write_concern)
  )
end

#download_document(dataset_id, document_id) ⇒ String

Download the retained original bytes for a dataset document. The HTTP transport follows redirects so this returns the final raw object bytes.

Parameters:

  • dataset_id (String) —

    dataset id.

  • document_id (String) —

    document id returned by #list_documents.

Returns:

  • (String) —

    raw document bytes.



149
150
151
 
# File 'lib/vector_amp/datasets.rb', line 149

def download_document(dataset_id, document_id)
  @transport.request(:get, "/datasets/#{dataset_id}/documents/#{document_id}/download", raw: true, headers: { Accept: "*/*" })
end

#embed(dataset_id, text: nil, texts: nil) ⇒ Hash

Generate embeddings using the dataset embedding configuration.

Parameters:

  • dataset_id (String) —

    dataset id.

  • text (String, nil) (defaults to: nil) —

    single text to embed.

  • texts (Array<String>, nil) (defaults to: nil) —

    multiple texts to embed.

Returns:

  • (Hash) —

    embed response.

Raises:

  • (ArgumentError) 


237
238
239
240
241
 
# File 'lib/vector_amp/datasets.rb', line 237

def embed(dataset_id, text: nil, texts: nil)
  raise ArgumentError, "provide text or texts" if text.nil? && texts.nil?

  @transport.request(:post, "/datasets/#{dataset_id}/embed", body: Utils.compact_hash(text: text, texts: texts))
end

#get(dataset_id) ⇒ Dataset

Fetch a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:



30
31
32
 
# File 'lib/vector_amp/datasets.rb', line 30

def get(dataset_id)
  wrap_dataset(@transport.request(:get, "/datasets/#{dataset_id}"))
end

#insert(dataset_id, vectors:) ⇒ Hash

Insert vectors into a dataset.

Vector record ids may be strings or integers. Integer ids are preserved as JSON numbers (not coerced to strings) so the API does not rewrite them.

Parameters:

  • dataset_id (String) —

    dataset id.

  • vectors (Array<Hash>) —

    vector records with ids, values, and optional metadata.

Returns:

  • (Hash) —

    insert response.



213
214
215
 
# File 'lib/vector_amp/datasets.rb', line 213

def insert(dataset_id, vectors:)
  @transport.request(:post, "/datasets/#{dataset_id}/insert", body: { vectors: Utils.normalize_vectors(vectors) })
end

#list(limit: 50, offset: 0) ⇒ Hash

List datasets.

Parameters:

  • limit (Integer) (defaults to: 50) —

    page size; defaults to 50.

  • offset (Integer) (defaults to: 0) —

    page offset; defaults to 0.

Returns:

  • (Hash) —

    response envelope with datasets wrapped as Dataset objects when present.



23
24
25
 
# File 'lib/vector_amp/datasets.rb', line 23

def list(limit: 50, offset: 0)
  wrap_list(@transport.request(:get, "/datasets", query: { limit: limit, offset: offset }))
end

#list_documents(dataset_id, limit: 50, cursor: nil, status: nil) ⇒ Hash

List retained source documents for a dataset using cursor pagination.

Parameters:

  • dataset_id (String) —

    dataset id.

  • limit (Integer, nil) (defaults to: 50) —

    maximum documents to return.

  • cursor (String, nil) (defaults to: nil) —

    cursor from a previous response's next_cursor.

  • status (String, nil) (defaults to: nil) —

    optional document status filter, e.g. ready.

Returns:

  • (Hash) —

    response envelope with documents and next_cursor.



140
141
142
 
# File 'lib/vector_amp/datasets.rb', line 140

def list_documents(dataset_id, limit: 50, cursor: nil, status: nil)
  @transport.request(:get, "/datasets/#{dataset_id}/documents", query: Utils.compact_hash(limit: limit, cursor: cursor, status: status))
end

#patch_metadata_schema(dataset_id, schema) ⇒ Dataset

Add or update typed metadata fields, retaining omitted fields.

Parameters:

  • dataset_id (String) —

    dataset id.

  • schema (Array<Hash>) —

    fields with name and canonical type.

Returns:



122
123
124
 
# File 'lib/vector_amp/datasets.rb', line 122

def patch_metadata_schema(dataset_id, schema)
  update_metadata_schema(dataset_id, schema, "merge")
end

#replace_metadata_schema(dataset_id, schema) ⇒ Dataset

Replace the complete typed metadata schema.

Parameters:

  • dataset_id (String) —

    dataset id.

  • schema (Array<Hash>) —

    complete field list; may be empty.

Returns:



130
131
132
 
# File 'lib/vector_amp/datasets.rb', line 130

def replace_metadata_schema(dataset_id, schema)
  update_metadata_schema(dataset_id, schema, "replace")
end

#search(dataset_id, query_text_or_options = nil, query: nil, query_text: nil, search_text: nil, top_k: 10, filters: nil, advanced_filters: nil, embedding_model: nil, embedding_provider: nil, nprobe_override: nil, rerank_depth_override: nil, hybrid: nil, sparse_query: nil, alpha: nil, include_embeddings: nil, include_documents: nil, include_metadata: nil, rerank: nil, **unknown) ⇒ Hash

Search a dataset by text or vector query.

Parameters:

  • dataset_id (String) —

    dataset id.

  • query_text_or_options (String, Hash, nil) (defaults to: nil) —

    text query or legacy options hash.

  • query (Array<Numeric>, nil) (defaults to: nil) —

    vector query.

  • query_text (String, nil) (defaults to: nil) —

    explicit text query; overrides positional text when provided.

  • search_text (String, nil) (defaults to: nil) —

    alias for query_text for one-field hybrid/BM25 search.

  • top_k (Integer) (defaults to: 10) —

    number of results; defaults to 10.

  • filters (Hash, nil) (defaults to: nil) —

    metadata filters.

  • advanced_filters (Hash, nil) (defaults to: nil) —

    advanced filter expression.

  • embedding_model (String, nil) (defaults to: nil) —

    optional embedding model override.

  • embedding_provider (String, nil) (defaults to: nil) —

    optional embedding provider override.

  • nprobe_override (Integer, nil) (defaults to: nil) —

    optional SABLE search probe override.

  • rerank_depth_override (Integer, nil) (defaults to: nil) —

    optional rerank depth override.

  • hybrid (Boolean, nil) (defaults to: nil) —

    enable hybrid search when supported.

  • sparse_query (Hash, nil) (defaults to: nil) —

    sparse query for hybrid search.

  • alpha (Numeric, nil) (defaults to: nil) —

    dense/sparse weighting for hybrid search.

  • include_embeddings (Boolean, nil) (defaults to: nil) —

    include vector values in results.

  • include_documents (Boolean, nil) (defaults to: nil) —

    include document fields in results.

  • include_metadata (Boolean, nil) (defaults to: nil) —

    include metadata; API default is true.

  • rerank (Boolean, Hash, nil) (defaults to: nil) —

    enable semantic reranking. true or { enabled: true } uses vectoramp / VectorAmp-Rerank-v1.

Returns:

  • (Hash) —

    search response.

Raises:

  • (ArgumentError) 


174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
 
# File 'lib/vector_amp/datasets.rb', line 174

def search(dataset_id, query_text_or_options = nil, query: nil, query_text: nil, search_text: nil, top_k: 10, filters: nil, advanced_filters: nil,
           embedding_model: nil, embedding_provider: nil, nprobe_override: nil, rerank_depth_override: nil,
           hybrid: nil, sparse_query: nil, alpha: nil, include_embeddings: nil, include_documents: nil,
           include_metadata: nil, rerank: nil, **unknown)
  if query_text_or_options.is_a?(Hash)
    unknown = query_text_or_options.merge(unknown)
    query_text_or_options = nil
  end
  Utils.ensure_no_unknown!(unknown, "search")
  raise ArgumentError, "provide query_text or search_text, not both" if query_text && search_text
  resolved_query_text = query_text || search_text || query_text_or_options
  body = Utils.compact_hash(
    query: query,
    query_text: resolved_query_text,
    top_k: top_k,
    filters: filters,
    advanced_filters: advanced_filters,
    embedding_model: embedding_model,
    embedding_provider: embedding_provider,
    nprobe_override: nprobe_override,
    rerank_depth_override: rerank_depth_override,
    hybrid: hybrid,
    sparse_query: sparse_query,
    alpha: alpha,
    include_embeddings: include_embeddings,
    include_documents: include_documents,
    include_metadata: include_metadata,
    rerank: rerank
  )
  @transport.request(:post, "/datasets/#{dataset_id}/search", body: body)
end

#stats(dataset_id) ⇒ Hash

Fetch dataset statistics.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:

  • (Hash) —

    stats response.



44
45
46
 
# File 'lib/vector_amp/datasets.rb', line 44

def stats(dataset_id)
  @transport.request(:get, "/datasets/#{dataset_id}/stats")
end