Class: LittleGhost::Providers::OpenAICompatible

Inherits:
Object
  • Object
show all
Defined in:
lib/little_ghost/providers/openai_compatible.rb

Overview

OpenAICompatible brings OpenAI-style Responses or Chat Completions endpoints into LittleGhost. Agents receive the same streaming events whether the endpoint is OpenAI, a hosted model service, or an application gateway.

provider = LittleGhost::Providers::OpenAICompatible.new(
api_key: ENV.fetch("MODEL_API_KEY"),
model: "example-model",
base_url: "https://models.example.test/v1/"
)

The client translates ModelRequest values to the selected wire API and translates responses back to StreamEvent objects.

Retries and streaming output

Transient HTTP and stream failures retry with limited exponential backoff. A :model_retry event reports each retry and whether text had already been emitted. Partial text may repeat after a retry, so consumers that assemble streams must use that event to discard or replace superseded output.

Direct Known Subclasses

OpenAI, OpenRouter

Defined Under Namespace

Classes: ChatNormalizer, Normalizer, ResponsesNormalizer, StreamError

Constant Summary collapse

DEFAULT_BASE_URL =

The OpenAI API endpoint used when base_url is omitted.

"https://api.openai.com/v1/"
INITIAL_RETRY_DELAY =

:nodoc:

1
MAX_RETRY_DELAY =

:nodoc:

16
TRANSIENT_STREAM_ERROR_TYPES = 
%w[
  provider_overloaded provider_unavailable rate_limit_exceeded server timeout
].freeze
CONTEXT_OVERFLOW_MARKERS =

:nodoc:

[
  "context_length_exceeded", "context window", "maximum context length",
  "max context length", "input is too long", "too many input tokens"
].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(api_key:, model:, base_url: DEFAULT_BASE_URL, api: :responses, headers: {}, open_timeout: 10, read_timeout: 120, allow_insecure_http: false, max_response_bytes: HTTPTransport::DEFAULT_MAX_RESPONSE_BYTES, max_retries: 2, max_retry_delay: MAX_RETRY_DELAY, transport: nil, sleeper: nil, on_retry: ->(*) {}) ⇒ OpenAICompatible

Configures an OpenAI-compatible client.

api is :responses or :chat_completions. headers adds trusted endpoint-specific headers. max_retries controls retries before the original error is raised, and on_retry receives the attempt, error, and delay. Pass a custom transport for alternate HTTP execution.

Raises:



83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 83

def initialize(
  api_key:,
  model:,
  base_url: DEFAULT_BASE_URL,
  api: :responses,
  headers: {},
  open_timeout: 10,
  read_timeout: 120,
  allow_insecure_http: false,
  max_response_bytes: HTTPTransport::DEFAULT_MAX_RESPONSE_BYTES,
  max_retries: 2,
  max_retry_delay: MAX_RETRY_DELAY,
  transport: nil,
  sleeper: nil,
  on_retry: ->(*) {}
)
  @api_key = api_key
  @model = model
  @api = api.to_sym
  raise ConfigurationError, "api must be :responses or :chat_completions" unless %i[responses chat_completions].include?(@api)

  @headers = headers.transform_keys(&:to_s).freeze
  @max_retries = Integer(max_retries)
  @max_retry_delay = Integer(max_retry_delay)
  @transport = transport || HTTPTransport.new(
    base_url:,
    open_timeout:,
    read_timeout:,
    allow_insecure_http:,
    max_response_bytes:
  )
  @sleeper = sleeper
  @on_retry = on_retry
end

Instance Attribute Details

#api ⇒ Object (readonly)

Provider model identifier and selected OpenAI-compatible wire API.



75
76
77
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 75

def api
  @api
end

#model ⇒ Object (readonly)

Provider model identifier and selected OpenAI-compatible wire API.



75
76
77
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 75

def model
  @model
end

Instance Method Details

#capabilities(metadata: {}) ⇒ Object

Returns the legacy capability contract expected from compatible APIs. Subclasses can override this when the endpoint advertises precise support.



160
161
162
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 160

def capabilities(metadata: {})
  ModelCapabilities.legacy
end

#stream(request) ⇒ Object

Streams LittleGhost StreamEvent objects for request.

Without a block, returns an Enumerator. Context-window errors normalize to ContextWindowOverflowError, and malformed tool calls normalize to MalformedToolCallError.



123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/little_ghost/providers/openai_compatible.rb', line 123

def stream(request)
  return enum_for(__method__, request) unless block_given?

  attempts = 0

  begin
    partial_text = false
    request.cancellation_token.raise_if_cancelled!
    stream_once(request) do |event|
      partial_text ||= event.type == :text_delta && !event.data[:text].to_s.empty?
      yield event
    end
  rescue HTTPError, StreamError => error
    if context_window_overflow?(error)
      raise ContextWindowOverflowError, "The model context window was exceeded"
    end
    raise if !error.retryable? || attempts >= @max_retries

    attempts += 1
    request.cancellation_token.raise_if_cancelled!
    delay = capped_retry_delay(request, retry_delay(attempts))
    @on_retry.call(attempts, error, delay)
    yield StreamEvent.build(
      :model_retry,
      attempt: attempts,
      delay:,
      error_class: error.class.name,
      partial_text:,
      **retry_error_metadata(error)
    )
    wait_before_retry(request, delay)
    retry
  end
end