Class: Ask::Agent::Middleware::ModelFallback

Inherits:
Base
  • Object
show all
Defined in:
lib/ask/agent/middleware/model_fallback.rb

Overview

Switches to a fallback model+provider when the primary LLM call fails with a transient error (rate limit, server error, service unavailable).

Each fallback specifies both a model and a provider slug, so the middleware can switch from e.g. OpenAI to Anthropic transparently. Credentials for each provider are resolved automatically via Ask::Auth.resolve.

Examples:

Basic usage — fallback to Anthropic when OpenAI is overloaded

pipeline.use :model_fallback, fallbacks: [
  { model: "claude-sonnet-4",     provider: :anthropic },
  { model: "gemini-2.0-flash",    provider: :google }
]

With failure-trigger customization

pipeline.use :model_fallback, fallbacks: [
  { model: "claude-sonnet-4", provider: :anthropic, on_error: [Ask::RateLimitError, Ask::ServerError] }
]

Using the block form to choose fallbacks dynamically

pipeline.use :model_fallback, fallbacks: ->(error, request) {
  if request[:messages].sum { |m| m[:content].to_s.length } > 100_000
    [{ model: "claude-sonnet-4", provider: :anthropic }]  # use long-context model
  else
    [{ model: "gpt-4o-mini", provider: :openai }]          # use cheaper model
  end
}

Constant Summary collapse

DEFAULT_ELIGIBLE_ERRORS = 
[
  Ask::RateLimitError, Ask::ServerError, Ask::ServiceUnavailable
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(fallbacks:, eligible_errors: nil) ⇒ ModelFallback

Returns a new instance of ModelFallback.

Raises:

  • (ArgumentError) 


38
39
40
41
42
 
# File 'lib/ask/agent/middleware/model_fallback.rb', line 38

def initialize(fallbacks:, eligible_errors: nil)
  @fallbacks = fallbacks.respond_to?(:call) ? fallbacks : Array(fallbacks)
  @eligible_errors = Array(eligible_errors || DEFAULT_ELIGIBLE_ERRORS)
  raise ArgumentError, "At least one fallback is required" if Array(@fallbacks).empty?
end

Instance Method Details

#around_request(provider, request) ⇒ Object 



44
45
46
47
48
49
50
51
52
53
 
# File 'lib/ask/agent/middleware/model_fallback.rb', line 44

def around_request(provider, request)
  # Try primary provider
  begin
    return yield
  rescue *@eligible_errors => e
    result = try_fallbacks(request, error: e)
    return result if result
    raise
  end
end