Class: VectorAmp::DatasetsResource

Inherits:
Object
  • Object
show all
Defined in:
lib/vector_amp/datasets.rb

Overview

Dataset API resource for create/list/get/search/insert operations.

Instance Method Summary collapse

Constructor Details

#initialize(transport, client: nil) ⇒ DatasetsResource

Parameters:

  • transport (#request) —

    API transport.

  • client (Client, nil) (defaults to: nil) —

    optional client attached to returned Dataset objects.



14
15
16
17
 
# File 'lib/vector_amp/datasets.rb', line 14

def initialize(transport, client: nil)
  @transport = transport
  @client = client
end

Instance Method Details

#add_texts(dataset_id, texts_arg = nil, texts: nil, ids: nil, metadata: nil) ⇒ Hash

Embed and insert texts into a dataset.

Accepts a single string or a list of strings. Ids are auto-generated when omitted; supplied ids may be strings or integers, and integer ids are preserved as JSON numbers. The source text is copied into metadata.text.

Parameters:

  • dataset_id (String) —

    dataset id.

  • texts_arg (String, Array<String>, nil) (defaults to: nil) —

    positional text(s) for convenience.

  • texts (String, Array<String>, nil) (defaults to: nil) —

    keyword text(s).

  • ids (Array, nil) (defaults to: nil) —

    optional ids; generated UUIDs when omitted.

  • metadata (Hash, Array<Hash>, nil) (defaults to: nil) —

    metadata applied to all texts or per-text.

Returns:

  • (Hash) —

    insert response.

Raises:

  • (ArgumentError) 


239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
 
# File 'lib/vector_amp/datasets.rb', line 239

def add_texts(dataset_id, texts_arg = nil, texts: nil, ids: nil, metadata: nil)
  texts = Array(texts || texts_arg)
  raise ArgumentError, "texts must not be empty" if texts.empty?
  raise ArgumentError, "ids length must match texts length" if ids && ids.length != texts.length
  if metadata.is_a?(Array) && metadata.length != texts.length
    raise ArgumentError, "metadata length must match texts length"
  end

  response = embed(dataset_id, texts: texts)
  embeddings = response.fetch("embeddings")
  vectors = texts.each_with_index.map do |text, index|
    item_metadata = metadata.is_a?(Array) ? metadata[index] : metadata
    {
      id: ids ? Utils.coerce_vector_id(ids[index]) : SecureRandom.uuid,
      values: embeddings[index],
      metadata: Utils.compact_hash((item_metadata || {}).merge(text: text))
    }
  end
  insert(dataset_id, vectors: vectors)
end

#create(name:, dim: nil, embedding: nil, metric: "cosine", hybrid: nil, filters: nil, metadata_schema: nil, tuning: nil, metadata: nil, **unknown) ⇒ Dataset

Create a SABLE dataset. index_type is managed by the SDK and always sent as sable.

Only name is required. The embedding defaults to the managed vectoramp/VectorAmp-Embedding-4B model and the dimension is inferred (2560 for the default model). Pass embedding: (a Hash or model String, or the result of Embedding.openai) to use a different model; if the model's dimension is not known to the SDK you must also pass dim:.

Parameters:

  • name (String) —

    dataset name.

  • dim (Integer, nil) (defaults to: nil) —

    embedding/vector dimension; inferred when omitted for known models.

  • embedding (Hash, String, nil) (defaults to: nil) —

    embedding configuration; defaults to the managed VectorAmp model.

  • metric (String) (defaults to: "cosine") —

    distance metric; defaults to cosine.

  • hybrid (Boolean, nil) (defaults to: nil) —

    enable hybrid (dense + sparse) search; sends hybrid: true.

  • filters (Hash, nil) (defaults to: nil) —

    optional filter schema/config.

  • metadata_schema (Hash, nil) (defaults to: nil) —

    optional metadata schema.

  • tuning (Hash, nil) (defaults to: nil) —

    optional SABLE tuning parameters.

  • metadata (Hash, nil) (defaults to: nil) —

    optional dataset metadata.

Returns:

Raises:

  • (ArgumentError) —

    when index_type is supplied, dim cannot be inferred, or unknown options are passed.



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/vector_amp/datasets.rb', line 67

def create(name:, dim: nil, embedding: nil, metric: "cosine", hybrid: nil, filters: nil, metadata_schema: nil, tuning: nil, metadata: nil, **unknown)
  if unknown.key?(:index_type) || unknown.key?("index_type")
    raise ArgumentError, "index_type is managed by VectorAmp Ruby SDK and is always 'sable'"
  end
  Utils.ensure_no_unknown!(unknown, "create")

  resolved_embedding = Embedding.normalize(embedding)
  resolved_dim = dim || Embedding.infer_dim(resolved_embedding)
  if resolved_dim.nil?
    model = resolved_embedding[:model] || resolved_embedding["model"]
    raise ArgumentError, "dim is required for embedding model #{model.inspect}; pass dim: explicitly"
  end

  body = Utils.compact_hash(
    name: name,
    dim: resolved_dim,
    metric: metric,
    embedding: resolved_embedding,
    index_type: "sable",
    hybrid: hybrid,
    filters: filters,
    metadata_schema: metadata_schema,
    tuning: tuning,
    metadata: metadata
  )
  wrap_dataset(@transport.request(:post, "/datasets", body: body))
end

#create_with_openai_api_key(name:, api_key:, size: "small", secret_ref: "emb:openai:api_key", validate: false, **options) ⇒ Dataset

Store/update an OpenAI API key in org secrets, then create an OpenAI-backed dataset whose embedding config references that secret.

Parameters:

  • name (String) —

    dataset name.

  • api_key (String) —

    OpenAI API key to store server-side.

  • size (String, Symbol) (defaults to: "small") —

    "small" or "large".

  • secret_ref (String) (defaults to: "emb:openai:api_key") —

    org secret reference to write and attach to the dataset.

  • validate (Boolean) (defaults to: false) —

    validate the key before storing.

  • options (Hash) —

    forwarded to #create, e.g. hybrid: true or metadata:.

Returns:



104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/vector_amp/datasets.rb', line 104

def create_with_openai_api_key(name:, api_key:, size: "small", secret_ref: "emb:openai:api_key", validate: false, **options)
  secrets = @client&.org_secrets || begin
    require_relative "org_secrets"
    OrgSecretsResource.new(@transport)
  end
  secrets.put_openai_api_key(
    api_key: api_key,
    secret_ref: secret_ref,
    validate: validate,
    model: openai_model(size)
  )
  create(name: name, embedding: Embedding.openai(size, secret_ref: secret_ref), **options)
end

#delete(dataset_id) ⇒ Hash

Delete a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:

  • (Hash) —

    delete response.



37
38
39
 
# File 'lib/vector_amp/datasets.rb', line 37

def delete(dataset_id)
  @transport.request(:delete, "/datasets/#{dataset_id}")
end

#delete_vectors(dataset_id, ids:, write_concern: nil) ⇒ Hash

Delete one or more vectors from a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

  • ids (Array<String,Integer>) —

    vector ids to delete.

  • write_concern (String, nil) (defaults to: nil) —

    optional API write concern.

Returns:

  • (Hash) —

    delete response with deleted and dataset_id.

Raises:

  • (ArgumentError) 


207
208
209
210
211
212
213
214
215
 
# File 'lib/vector_amp/datasets.rb', line 207

def delete_vectors(dataset_id, ids:, write_concern: nil)
  raise ArgumentError, "ids must not be empty" if ids.nil? || ids.empty?

  @transport.request(
    :delete,
    "/datasets/#{dataset_id}/vectors",
    body: Utils.compact_hash(ids: ids, write_concern: write_concern)
  )
end

#download_document(dataset_id, document_id) ⇒ String

Download the retained original bytes for a dataset document. The HTTP transport follows redirects so this returns the final raw object bytes.

Parameters:

  • dataset_id (String) —

    dataset id.

  • document_id (String) —

    document id returned by #list_documents.

Returns:

  • (String) —

    raw document bytes.



134
135
136
 
# File 'lib/vector_amp/datasets.rb', line 134

def download_document(dataset_id, document_id)
  @transport.request(:get, "/datasets/#{dataset_id}/documents/#{document_id}/download", raw: true, headers: { Accept: "*/*" })
end

#embed(dataset_id, text: nil, texts: nil) ⇒ Hash

Generate embeddings using the dataset embedding configuration.

Parameters:

  • dataset_id (String) —

    dataset id.

  • text (String, nil) (defaults to: nil) —

    single text to embed.

  • texts (Array<String>, nil) (defaults to: nil) —

    multiple texts to embed.

Returns:

  • (Hash) —

    embed response.

Raises:

  • (ArgumentError) 


222
223
224
225
226
 
# File 'lib/vector_amp/datasets.rb', line 222

def embed(dataset_id, text: nil, texts: nil)
  raise ArgumentError, "provide text or texts" if text.nil? && texts.nil?

  @transport.request(:post, "/datasets/#{dataset_id}/embed", body: Utils.compact_hash(text: text, texts: texts))
end

#get(dataset_id) ⇒ Dataset

Fetch a dataset by id.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:



30
31
32
 
# File 'lib/vector_amp/datasets.rb', line 30

def get(dataset_id)
  wrap_dataset(@transport.request(:get, "/datasets/#{dataset_id}"))
end

#insert(dataset_id, vectors:) ⇒ Hash

Insert vectors into a dataset.

Vector record ids may be strings or integers. Integer ids are preserved as JSON numbers (not coerced to strings) so the API does not rewrite them.

Parameters:

  • dataset_id (String) —

    dataset id.

  • vectors (Array<Hash>) —

    vector records with ids, values, and optional metadata.

Returns:

  • (Hash) —

    insert response.



198
199
200
 
# File 'lib/vector_amp/datasets.rb', line 198

def insert(dataset_id, vectors:)
  @transport.request(:post, "/datasets/#{dataset_id}/insert", body: { vectors: Utils.normalize_vectors(vectors) })
end

#list(limit: 50, offset: 0) ⇒ Hash

List datasets.

Parameters:

  • limit (Integer) (defaults to: 50) —

    page size; defaults to 50.

  • offset (Integer) (defaults to: 0) —

    page offset; defaults to 0.

Returns:

  • (Hash) —

    response envelope with datasets wrapped as Dataset objects when present.



23
24
25
 
# File 'lib/vector_amp/datasets.rb', line 23

def list(limit: 50, offset: 0)
  wrap_list(@transport.request(:get, "/datasets", query: { limit: limit, offset: offset }))
end

#list_documents(dataset_id, limit: 50, cursor: nil, status: nil) ⇒ Hash

List retained source documents for a dataset using cursor pagination.

Parameters:

  • dataset_id (String) —

    dataset id.

  • limit (Integer, nil) (defaults to: 50) —

    maximum documents to return.

  • cursor (String, nil) (defaults to: nil) —

    cursor from a previous response's next_cursor.

  • status (String, nil) (defaults to: nil) —

    optional document status filter, e.g. ready.

Returns:

  • (Hash) —

    response envelope with documents and next_cursor.



125
126
127
 
# File 'lib/vector_amp/datasets.rb', line 125

def list_documents(dataset_id, limit: 50, cursor: nil, status: nil)
  @transport.request(:get, "/datasets/#{dataset_id}/documents", query: Utils.compact_hash(limit: limit, cursor: cursor, status: status))
end

#search(dataset_id, query_text_or_options = nil, query: nil, query_text: nil, search_text: nil, top_k: 10, filters: nil, advanced_filters: nil, embedding_model: nil, embedding_provider: nil, nprobe_override: nil, rerank_depth_override: nil, hybrid: nil, sparse_query: nil, alpha: nil, include_embeddings: nil, include_documents: nil, include_metadata: nil, rerank: nil, **unknown) ⇒ Hash

Search a dataset by text or vector query.

Parameters:

  • dataset_id (String) —

    dataset id.

  • query_text_or_options (String, Hash, nil) (defaults to: nil) —

    text query or legacy options hash.

  • query (Array<Numeric>, nil) (defaults to: nil) —

    vector query.

  • query_text (String, nil) (defaults to: nil) —

    explicit text query; overrides positional text when provided.

  • search_text (String, nil) (defaults to: nil) —

    alias for query_text for one-field hybrid/BM25 search.

  • top_k (Integer) (defaults to: 10) —

    number of results; defaults to 10.

  • filters (Hash, nil) (defaults to: nil) —

    metadata filters.

  • advanced_filters (Hash, nil) (defaults to: nil) —

    advanced filter expression.

  • embedding_model (String, nil) (defaults to: nil) —

    optional embedding model override.

  • embedding_provider (String, nil) (defaults to: nil) —

    optional embedding provider override.

  • nprobe_override (Integer, nil) (defaults to: nil) —

    optional SABLE search probe override.

  • rerank_depth_override (Integer, nil) (defaults to: nil) —

    optional rerank depth override.

  • hybrid (Boolean, nil) (defaults to: nil) —

    enable hybrid search when supported.

  • sparse_query (Hash, nil) (defaults to: nil) —

    sparse query for hybrid search.

  • alpha (Numeric, nil) (defaults to: nil) —

    dense/sparse weighting for hybrid search.

  • include_embeddings (Boolean, nil) (defaults to: nil) —

    include vector values in results.

  • include_documents (Boolean, nil) (defaults to: nil) —

    include document fields in results.

  • include_metadata (Boolean, nil) (defaults to: nil) —

    include metadata; API default is true.

  • rerank (Boolean, Hash, nil) (defaults to: nil) —

    enable semantic reranking. true or { enabled: true } uses vectoramp / VectorAmp-Rerank-v1.

Returns:

  • (Hash) —

    search response.

Raises:

  • (ArgumentError) 


159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/vector_amp/datasets.rb', line 159

def search(dataset_id, query_text_or_options = nil, query: nil, query_text: nil, search_text: nil, top_k: 10, filters: nil, advanced_filters: nil,
           embedding_model: nil, embedding_provider: nil, nprobe_override: nil, rerank_depth_override: nil,
           hybrid: nil, sparse_query: nil, alpha: nil, include_embeddings: nil, include_documents: nil,
           include_metadata: nil, rerank: nil, **unknown)
  if query_text_or_options.is_a?(Hash)
    unknown = query_text_or_options.merge(unknown)
    query_text_or_options = nil
  end
  Utils.ensure_no_unknown!(unknown, "search")
  raise ArgumentError, "provide query_text or search_text, not both" if query_text && search_text
  resolved_query_text = query_text || search_text || query_text_or_options
  body = Utils.compact_hash(
    query: query,
    query_text: resolved_query_text,
    top_k: top_k,
    filters: filters,
    advanced_filters: advanced_filters,
    embedding_model: embedding_model,
    embedding_provider: embedding_provider,
    nprobe_override: nprobe_override,
    rerank_depth_override: rerank_depth_override,
    hybrid: hybrid,
    sparse_query: sparse_query,
    alpha: alpha,
    include_embeddings: include_embeddings,
    include_documents: include_documents,
    include_metadata: include_metadata,
    rerank: rerank
  )
  @transport.request(:post, "/datasets/#{dataset_id}/search", body: body)
end

#stats(dataset_id) ⇒ Hash

Fetch dataset statistics.

Parameters:

  • dataset_id (String) —

    dataset id.

Returns:

  • (Hash) —

    stats response.



44
45
46
 
# File 'lib/vector_amp/datasets.rb', line 44

def stats(dataset_id)
  @transport.request(:get, "/datasets/#{dataset_id}/stats")
end