Class: LittleGhost::Providers::OpenAICompatible

Inherits:
Base
  • Object
show all
Defined in:
lib/little_ghost/providers/openai_compatible.rb

Overview

OpenAICompatible brings OpenAI-style Responses or Chat Completions endpoints into LittleGhost. Agents receive the same streaming events whether the endpoint is OpenAI, a hosted model service, or an application gateway.

provider = LittleGhost::Providers::OpenAICompatible.new(
api_key: ENV.fetch("MODEL_API_KEY"),
model: "example-model",
base_url: "https://models.example.test/v1/"
)

The client translates ModelRequest values to the selected wire API and translates responses back to StreamEvent objects. It also sends Embeddings::Request values to the compatible embeddings endpoint.

Retries and streaming output

Transient HTTP and stream failures retry with limited exponential backoff. A :model_retry event reports each retry and whether text had already been emitted. Partial text may repeat after a retry, so consumers that assemble streams must use that event to discard or replace superseded output.

Direct Known Subclasses

LMStudio, OpenAI, OpenRouter

Defined Under Namespace

Classes: ChatNormalizer, Normalizer, ResponsesNormalizer, StreamError

Constant Summary collapse

DEFAULT_BASE_URL =

The OpenAI API endpoint used when base_url is omitted.

"https://api.openai.com/v1/"
DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES =

:nodoc:

8 * 1024 * 1024
INITIAL_RETRY_DELAY =

:nodoc:

1
MAX_RETRY_DELAY =

:nodoc:

16
TRANSIENT_STREAM_ERROR_TYPES = 
%w[
  provider_overloaded provider_unavailable rate_limit_exceeded server timeout
].freeze
CONTEXT_OVERFLOW_MARKERS =

:nodoc:

[
  "context_length_exceeded", "context window", "maximum context length",
  "max context length", "input is too long", "too many input tokens"
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#prepare_request

Constructor Details

#initialize(api_key:, model:, base_url: DEFAULT_BASE_URL, api: :responses, headers: {}, open_timeout: 10, read_timeout: 120, allow_insecure_http: false, max_response_bytes: Support::HTTPClient::DEFAULT_MAX_RESPONSE_BYTES, max_embedding_response_bytes: DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES, max_retries: 2, max_retry_delay: MAX_RETRY_DELAY, transport: nil, sleeper: nil, on_retry: ->(*) {}) ⇒ OpenAICompatible

Configures an OpenAI-compatible client.

api is :responses or :chat_completions. headers adds trusted endpoint-specific headers. max_retries controls retries before the original error is raised, and on_retry receives the attempt, error, and delay. Pass a custom transport for alternate HTTP execution.

Raises:



90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 90

def initialize(
  api_key:,
  model:,
  base_url: DEFAULT_BASE_URL,
  api: :responses,
  headers: {},
  open_timeout: 10,
  read_timeout: 120,
  allow_insecure_http: false,
  max_response_bytes: Support::HTTPClient::DEFAULT_MAX_RESPONSE_BYTES,
  max_embedding_response_bytes: DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES,
  max_retries: 2,
  max_retry_delay: MAX_RETRY_DELAY,
  transport: nil,
  sleeper: nil,
  on_retry: ->(*) {}
)
  @api_key = api_key
  @model = model
  @api = api.to_sym
  raise ConfigurationError, "api must be :responses or :chat_completions" unless %i[responses chat_completions].include?(@api)

  @headers = headers.transform_keys(&:to_s).freeze
  @max_embedding_response_bytes = Integer(max_embedding_response_bytes)
  raise ArgumentError, "max_embedding_response_bytes must be positive" unless @max_embedding_response_bytes.positive?

  @max_retries = Integer(max_retries)
  @max_retry_delay = Integer(max_retry_delay)
  @transport = transport || Support::HTTPClient.new(
    base_url:,
    open_timeout:,
    read_timeout:,
    allow_insecure_http:,
    max_response_bytes:
  )
  @sleeper = sleeper
  @on_retry = on_retry
end

Instance Attribute Details

#api ⇒ Object (readonly)

Provider model identifier and selected OpenAI-compatible wire API.



82
83
84
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 82

def api
  @api
end

#model ⇒ Object (readonly)

Provider model identifier and selected OpenAI-compatible wire API.



82
83
84
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 82

def model
  @model
end

Class Method Details

.request_options ⇒ Object

Request policy supported by OpenAI-compatible HTTP clients.



31
32
33
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 31

def self.request_options
  %i[max_response_bytes max_retries max_retry_delay open_timeout read_timeout].freeze
end

Instance Method Details

#capabilities(metadata: {}) ⇒ Object

Returns the permissive capability contract expected from compatible APIs. Subclasses can override this when the endpoint advertises precise support.



212
213
214
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 212

def capabilities(metadata: {})
  ModelCapabilities.permissive
end

#embed(request) ⇒ Object

Embeds one or more strings with the configured compatible model.

The optional :dimensions request setting selects a supported output size. The response preserves input order and raises ProtocolError when the endpoint returns an incomplete or invalid batch.



174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 174

def embed(request)
  attempts = 0
  begin
    request.cancellation_token.raise_if_cancelled!
    payload = {
      model:,
      input: request.inputs,
      encoding_format: "float"
    }
    dimensions = request.settings[:dimensions]
    payload[:dimensions] = Integer(dimensions) if dimensions
    body = +""
    @transport.stream(
      path: "embeddings",
      headers: {"Authorization" => "Bearer #{@api_key}", "Content-Type" => "application/json"}.merge(@headers),
      body: JSON.generate(payload),
      cancellation_token: request.cancellation_token,
      deadline: request.deadline
    ) do |chunk|
      if body.bytesize + chunk.bytesize > @max_embedding_response_bytes
        raise ProtocolError, "#{embedding_provider_name} embedding response exceeded #{@max_embedding_response_bytes} bytes"
      end
      body << chunk
    end
    normalize_embedding_response(body, request.inputs.length, dimensions && Integer(dimensions))
  rescue HTTPError => error
    raise unless error.retryable? && attempts < @max_retries

    attempts += 1
    delay = capped_retry_delay(request, retry_delay(attempts))
    @on_retry.call(attempts, error, delay)
    wait_before_retry(request, delay)
    retry
  end
end

#stream(request) ⇒ Object

Streams LittleGhost StreamEvent objects for request.

Without a block, returns an Enumerator. Context-window errors normalize to ContextWindowOverflowError, and malformed tool calls normalize to MalformedToolCallError.



134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 134

def stream(request)
  return enum_for(__method__, request) unless block_given?

  attempts = 0

  begin
    partial_text = false
    request.cancellation_token.raise_if_cancelled!
    stream_once(request) do |event|
      partial_text ||= event.type == :text_delta && !event.data[:text].to_s.empty?
      yield event
    end
  rescue HTTPError, StreamError => error
    if context_window_overflow?(error)
      raise ContextWindowOverflowError, "The model context window was exceeded"
    end
    raise if !error.retryable? || attempts >= @max_retries

    attempts += 1
    request.cancellation_token.raise_if_cancelled!
    delay = capped_retry_delay(request, retry_delay(attempts))
    @on_retry.call(attempts, error, delay)
    yield StreamEvent.build(
      :model_retry,
      attempt: attempts,
      delay:,
      error_class: error.class.name,
      partial_text:,
      **retry_error_metadata(error)
    )
    wait_before_retry(request, delay)
    retry
  end
end