Class: RailsErrorDashboard::Integrations::LlmMiddleware

Inherits:
Object
  • Object
show all
Defined in:
lib/rails_error_dashboard/integrations/llm_middleware.rb

Overview

Faraday middleware that captures LLM calls to OpenAI and Anthropic APIs as breadcrumbs. The Tier 2 path — for hosts using ‘ruby-openai` or `anthropic-sdk-ruby` directly without OpenTelemetry instrumentation.

Install in the host app:

# Anthropic SDK
Anthropic::Client.new do |f|
  f.use RailsErrorDashboard::Integrations::LlmMiddleware
end

# ruby-openai
OpenAI::Client.new do |f|
  f.use RailsErrorDashboard::Integrations::LlmMiddleware
end

IMPORTANT — does NOT subclass ::Faraday::Middleware. Doing so would NameError at file-load time on hosts without Faraday. Faraday accepts any object that responds to ‘#call(env)` and is initialized with `app`. Hosts that don’t use OpenAI/Anthropic SDKs simply won’t reference this class and never load the constant.

HOST APP SAFETY:

  • Wraps the upstream call in rescue, but ALWAYS re-raises (we are in the host’s request path — swallowing would break their app logic)

  • Our own bookkeeping (response parsing, breadcrumb emission) is wrapped separately in rescue StandardError => nil

  • No work happens unless enable_llm_observability AND enable_breadcrumbs

  • Non-LLM URLs (anything but api.openai.com / api.anthropic.com) skip straight through with one host-string comparison

  • Streaming responses (SSE) skipped — token counts only available in the final stream event, which we’d need to buffer to read

Constant Summary collapse

OPENAI_HOSTS = 
[ "api.openai.com" ].freeze
ANTHROPIC_HOSTS = 
[ "api.anthropic.com" ].freeze

Instance Method Summary collapse

Constructor Details

#initialize(app) ⇒ LlmMiddleware

Returns a new instance of LlmMiddleware.



41
42
43
 
# File 'lib/rails_error_dashboard/integrations/llm_middleware.rb', line 41

def initialize(app)
  @app = app
end

Instance Method Details

#call(env) ⇒ Object 



45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/rails_error_dashboard/integrations/llm_middleware.rb', line 45

def call(env)
  return @app.call(env) unless RailsErrorDashboard.configuration.enable_llm_observability
  return @app.call(env) unless RailsErrorDashboard.configuration.enable_breadcrumbs

  provider = detect_provider(env)
  return @app.call(env) unless provider

  request_body = safe_parse_body(env.body)
  model        = request_body.is_a?(Hash) ? request_body["model"] : nil
  started_at   = monotonic_ms

  response = nil
  upstream_error = nil
  begin
    response = @app.call(env)
  rescue StandardError => e
    upstream_error = e
    raise
  ensure
    # Record the breadcrumb whether the call succeeded, returned an HTTP
    # error, or raised mid-flight. NEVER raise from this block — the
    # host's app.call has either returned or is propagating an exception
    # via `raise` above, and we must not interfere with either path.
    begin
      duration_ms = (monotonic_ms - started_at).round(2)
      emit_breadcrumb(provider, model, request_body, response, upstream_error, duration_ms)
    rescue StandardError => e
      RailsErrorDashboard::Logger.debug("[RailsErrorDashboard] LlmMiddleware.emit failed: #{e.message}")
    end
  end

  response
end