Class: SpreeMenuChat::EmbeddingClient

Inherits:
Object
  • Object
show all
Defined in:
app/services/spree_menu_chat/embedding_client.rb

Overview

Thin wrapper around Voyage AI's embeddings REST endpoint. All embedding calls in this extension go through here — same "no official Ruby SDK, call the REST API directly" situation as SpreeMenuChat::LlmClient (for Gemini) and SpreeDoordash::Client (for DoorDash's Drive API).

Self-throttles to the real limit (see #throttle!) rather than firing requests as fast as the caller loops and letting a generic error bubble up to ReembedCatalogJob's job-level retry_on — found live, running reembed_all against the real synced catalog: Voyage accounts with no payment method on file (true even for accounts still fully covered by the 200M-token free grant — the reduced limit is separate from the free-token allowance) are throttled to a real 3 requests/minute, 10K tokens/minute.

First attempt at this fix (reactive: catch a 429, sleep ~20s, retry) undershot in practice — confirmed live: after the initial 3-request burst, retries kept landing inside the same still-open rate window (a 3-req/60s window doesn't fully clear just because 20s passed since the failure), so most retries 429'd again and the run silently bottomed out at 21/43 records. Self-throttling — tracking this process's own last N request timestamps and sleeping proactively before the (N+1)th falls inside the window — avoids firing requests doomed to 429 in the first place, rather than reacting after the fact. The reactive 429-retry (#request's rescue) stays as a safety net for anything the proactive throttle doesn't account for (clock skew, another process/job sharing this store's key), not as the primary mechanism anymore.

Class-level (not per-instance) because the rate limit is scoped to the Voyage account/API key, not to any one EmbeddingClient instance — and this client is deliberately non-memoized (see .for_store below), so per-instance state would track nothing real. In-process/in-memory is sufficient here, not a Redis-backed limiter: this stack runs a single Puma+Solid-Queue process (no Redis anywhere in this project, same rationale as SpreeMenuChat::RateLimiter), and the account-level limit is already generous enough (3 RPM) that cross-process undercounting in a multi-dyno deployment would need to be revisited before that kind of scale, not before this.

Defined Under Namespace

Classes: MissingCredentialsError

Constant Summary collapse

BASE_URL = 
'https://api.voyageai.com'.freeze
MAX_REQUESTS_PER_WINDOW =

The real, live-confirmed no-payment-method limit (see class comment).

3
WINDOW_SECONDS = 
60
WINDOW_MARGIN_SECONDS =

clears clock-precision/latency edge cases

2
RATE_LIMIT_RETRY_WAIT =

Reactive safety net for a 429 the proactive throttle didn't prevent.

WINDOW_SECONDS + WINDOW_MARGIN_SECONDS
RATE_LIMIT_MAX_ATTEMPTS = 
3

Class Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(credential: nil) ⇒ EmbeddingClient

Returns a new instance of EmbeddingClient.

Raises:



82
83
84
85
86
 
# File 'app/services/spree_menu_chat/embedding_client.rb', line 82

def initialize(credential: nil)
  @credential = credential
  raise MissingCredentialsError, 'No Menu Chat credential connected for this store' unless @credential
  raise MissingCredentialsError, 'No Voyage AI API key set for this store' if @credential.voyage_api_key.blank?
end

Class Attribute Details

.request_times ⇒ Object

Exposed for specs (reset between examples) — not meant to be read or written from application code.



62
63
64
 
# File 'app/services/spree_menu_chat/embedding_client.rb', line 62

def request_times
  @request_times
end

Class Method Details

.for_store(store = Spree::Store.default) ⇒ Object 



78
79
80
 
# File 'app/services/spree_menu_chat/embedding_client.rb', line 78

def self.for_store(store = Spree::Store.default)
  new(credential: SpreeMenuChat::Credential.find_by(store: store))
end

.instance ⇒ Object

Deliberately NOT memoized — same rationale as every other client in this project (SpreeSquare::Client, SpreeDoordash::Client, SpreeMenuChat::LlmClient): a store's credential can be created/edited by an admin mid-process, and a Faraday connection is cheap enough to not be worth caching against that risk.



74
75
76
 
# File 'app/services/spree_menu_chat/embedding_client.rb', line 74

def self.instance
  for_store
end

.request_times_mutex ⇒ Object 



64
65
66
 
# File 'app/services/spree_menu_chat/embedding_client.rb', line 64

def request_times_mutex
  @request_times_mutex
end

Instance Method Details

#embed(input, input_type:, model: SpreeMenuChat::Config.embedding_model) ⇒ Object

input is a single string or an array of strings — Voyage's own accepted shape supports batching several documents into one request, though every current caller (SpreeMenuChat::Embedder, one record at a time) passes a single string; batching multiple products per call would cut the request count for ReembedCatalogJob's full-catalog pass and is worth doing as a follow-up given the real 3 RPM limit documented above, but isn't required for correctness now that every call is self-throttled to that limit. input_type is Voyage's own asymmetric-embedding hint: "document" for content being indexed (what this extension always embeds), "query" for the customer's question at retrieval time (SpreeMenuChat::Retriever, M3) — using the right one measurably improves retrieval quality per Voyage's own docs, it's not optional boilerplate.



101
102
103
104
105
 
# File 'app/services/spree_menu_chat/embedding_client.rb', line 101

def embed(input, input_type:, model: SpreeMenuChat::Config.embedding_model)
  body = { input: Array(input), model: model, input_type: input_type }
  result = request(:post, '/v1/embeddings', body)
  result.fetch('data', []).map { |row| row['embedding'] }
end