Class: LittleGhost::Providers::Bedrock

Inherits:
Base
  • Object
show all
Defined in:
lib/little_ghost/providers/bedrock.rb,
lib/little_ghost/providers/bedrock/http_client.rb,
lib/little_ghost/providers/bedrock/aws_protocol.rb,
lib/little_ghost/providers/bedrock/catalog_source.rb,
lib/little_ghost/providers/bedrock/credential_resolver.rb

Overview

Bedrock lets LittleGhost features use models available through Amazon Bedrock. Generation uses Converse and follows the same streaming events as every other LittleGhost provider.

provider = LittleGhost::Providers::Bedrock.new(
model: ENV.fetch("BEDROCK_MODEL_ID"),
region: ENV.fetch("AWS_REGION")
)

The default client uses LittleGhost's standard-library SigV4 and AWS EventStream implementations. Applications may inject client instead.

Transient service and stream failures retry with exponential backoff. Each retry emits :model_retry and reports whether partial text was already emitted, allowing stream consumers to handle repeated output deliberately.

Defined Under Namespace

Classes: AwsSigV4, CatalogSource, CredentialResolver, Credentials, EventStreamDecoder, HTTPClient, StreamError, StreamNormalizer

Constant Summary collapse

INITIAL_RETRY_DELAY =

:nodoc:

1
MAX_RETRY_DELAY =

:nodoc:

16
DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES =

:nodoc:

8 * 1024 * 1024
TRANSIENT_STREAM_ERRORS = 
%w[
  internal_server_exception model_stream_error_exception service_unavailable_exception throttling_exception
].freeze
CONTEXT_OVERFLOW_MARKERS =

:nodoc:

[
  "context window", "maximum context length", "max context length",
  "input is too long", "too many input tokens"
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#prepare_request

Constructor Details

#initialize(model:, region: nil, client: nil, max_retries: 2, sleeper: nil, on_retry: ->(*) {}, max_embedding_response_bytes: DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES, **client_options) ⇒ Bedrock

Configures Bedrock for model.

region and remaining client_options configure the built-in HTTP client. max_retries, sleeper, and on_retry control retry behavior. max_embedding_response_bytes bounds each embedding response retained in memory. Injecting client bypasses creation of the built-in HTTP client.



70
71
72
73
74
75
76
77
78
 
# File 'lib/little_ghost/providers/bedrock.rb', line 70

def initialize(model:, region: nil, client: nil, max_retries: 2, sleeper: nil,
  on_retry: ->(*) {}, max_embedding_response_bytes: DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES, **client_options)
  @model = model
  @client = client || build_client(region:, **client_options)
  @max_retries = Integer(max_retries)
  @sleeper = sleeper
  @on_retry = on_retry
  @max_embedding_response_bytes = positive_integer(max_embedding_response_bytes, :max_embedding_response_bytes)
end

Instance Attribute Details

#model ⇒ Object (readonly)

Bedrock model identifier used for requests.



62
63
64
 
# File 'lib/little_ghost/providers/bedrock.rb', line 62

def model
  @model
end

Class Method Details

.request_options ⇒ Object

Request policy supported by Bedrock retries and its built-in HTTP client.



31
32
33
 
# File 'lib/little_ghost/providers/bedrock.rb', line 31

def self.request_options
  %i[max_response_bytes max_retries open_timeout read_timeout].freeze
end

Instance Method Details

#capabilities(metadata: {}) ⇒ Object

Reads capabilities from Bedrock supported_parameters metadata. Missing metadata produces ModelCapabilities.unknown.



183
184
185
186
187
188
189
190
191
192
193
194
 
# File 'lib/little_ghost/providers/bedrock.rb', line 183

def capabilities(metadata: {})
  parameters = metadata[:supported_parameters] || metadata["supported_parameters"]
  return ModelCapabilities.unknown unless parameters.is_a?(Array)

  values = parameters.map(&:to_s)
  ModelCapabilities.new(
    native_structured_output: values.include?("structured_outputs"),
    tools: values.include?("tools"),
    tool_choice: values.include?("tool_choice"),
    supported_parameters: values
  )
end

#embed(request) ⇒ Object

Embeds text with Amazon Titan Text Embeddings V2.

The :dimensions request setting accepts 256, 512, or 1024 and defaults to 1024. :normalize controls vector normalization and defaults to true. Inputs are requested sequentially, and a failure raises without returning a partial batch.

Raises:



144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/little_ghost/providers/bedrock.rb', line 144

def embed(request)
  unless model == "amazon.titan-embed-text-v2:0"
    raise UnsupportedModelOperationError, "Bedrock embeddings require amazon.titan-embed-text-v2:0"
  end

  dimensions = Integer(request.settings.fetch(:dimensions, 1024))
  raise ConfigurationError, "dimensions must be 256, 512, or 1024" unless [256, 512, 1024].include?(dimensions)
  normalize = request.settings.fetch(:normalize, true)
  unless normalize == true || normalize == false
    raise ConfigurationError, "normalize must be true or false"
  end

  vectors = []
  usage = Usage.new
  request.inputs.each do |input|
    response = with_retries(request) do
      @client.invoke_model(
        model_id: model,
        body: {input_text: input, dimensions:, normalize:},
        cancellation_token: request.cancellation_token,
        deadline: request.deadline,
        max_response_bytes: @max_embedding_response_bytes
      )
    end
    payload = JSON.parse(response.body)
    vector = payload.fetch("embedding")
    unless vector.is_a?(Array) && vector.length == dimensions
      raise ProtocolError, "Bedrock returned an embedding with unexpected dimensions"
    end
    vectors << vector
    usage += Usage.new(input_tokens: payload["inputTextTokenCount"])
  rescue JSON::ParserError, KeyError
    raise ProtocolError, "Bedrock returned an invalid embedding response"
  end
  Embeddings::Response.new(vectors:, usage:, metadata: {model:})
end

#stream(request) ⇒ Object

Streams LittleGhost StreamEvent objects for request.

Without a block, returns an Enumerator. Context-window failures normalize to ContextWindowOverflowError and malformed tool calls normalize to MalformedToolCallError.



85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/little_ghost/providers/bedrock.rb', line 85

def stream(request)
  return enum_for(__method__, request) unless block_given?

  attempts = 0

  begin
    partial_text = false
    request.cancellation_token.raise_if_cancelled!
    normalizer = StreamNormalizer.new(model:)
    stream = Support::InterruptibleStream.new(
      cancellation_token: request.cancellation_token,
      deadline: request.deadline
    ) do |emit|
      response = @client.converse_stream(**request_parameters(request))
      response.stream.each { |event| emit.call(event) }
    end
    stream.each do |event|
      normalizer.consume(event_hash(event)).each do |normalized|
        partial_text ||= normalized.type == :text_delta && !normalized.data[:text].to_s.empty?
        yield normalized
      end
    end
    normalizer.finish.each do |event|
      partial_text ||= event.type == :text_delta && !event.data[:text].to_s.empty?
      yield event
    end
  rescue CancelledError, DeadlineExceededError, CleanupError
    raise
  rescue => error
    raise if error.is_a?(Error) && !error.is_a?(StreamError)

    if context_window_overflow?(error)
      raise ContextWindowOverflowError, "The model context window was exceeded"
    end
    raise provider_error(error) if !retryable?(error) || attempts >= @max_retries

    attempts += 1
    request.cancellation_token.raise_if_cancelled!
    delay = capped_retry_delay(request, retry_delay(attempts))
    @on_retry.call(attempts, error, delay)
    wait_before_retry(request, delay)
    yield StreamEvent.build(
      :model_retry,
      attempt: attempts,
      delay:,
      error_class: error.class.name,
      error_code: (error.event_type if error.is_a?(StreamError)),
      partial_text:
    )
    retry
  end
end