RubyLLM::Monitoring

Monitor your LLM usage within your Rails application.

Installation

[!NOTE] This engine relies on RubyLLM. Make sure you have it installed and configured.

Add this line to your application's Gemfile:

gem "ruby_llm-monitoring"

And then execute:

$ bundle

To copy and migrate RubyLLM::Monitoring's migrations, run:

$ rails ruby_llm_monitoring:install:migrations db:migrate

And then mount the engine in your config/routes.rb:

Rails.application.routes.draw do
  # ...

  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
end

Now you should be able to browse to /monitoring and monitor your LLM usage.

metrics alerts

Authentication and authorization

RubyLLM::Monitoring leaves authentication and authorization to the user. If no authentication is enforced, /monitoring will be available to everyone.

To enforce authentication, you can use route constraints, or set up a HTTP Basic auth middleware.

For example, if you're using devise, you can do this:

# config/routes.rb
authenticate :user do
  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
end

See more examples here.

However, if you're using Rails' default authentication generator, or an authentication solution that doesn't provide constraints, you need to roll out your own solution:

# config/routes.rb
constraints ->(request) { Constraints::Auth.authenticated?(request) } do
  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
end

# lib/constraints/auth.rb
class Constraints::Auth
  def self.authenticated?(request)
    cookies = ActionDispatch::Cookies::CookieJar.build(request, request.cookies)

    Session.find_by id: cookies.signed[:session_id]
  end
end

You can also set up a HTTP Basic auth middleware in the engine:

# config/initializers/ruby_llm-monitoring.rb
RubyLLM::Monitoring::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
  ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_monitoring_username, username) &
    ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_monitoring_password, password)
end

Metrics

The dashboard displays four metrics by default: Throughput, Cost, Response Time, and Error Rate. You can customize which metrics are shown or add your own custom metrics.

Configuration

In config/initializers/ruby_llm_monitoring.rb, you can configure which metrics are displayed:

RubyLLM::Monitoring.metrics = [
  RubyLLM::Monitoring::Metrics::Throughput,
  RubyLLM::Monitoring::Metrics::Cost,
  RubyLLM::Monitoring::Metrics::ResponseTime,
  RubyLLM::Monitoring::Metrics::ErrorCount
]

To remove a metric, simply omit it from the array:

RubyLLM::Monitoring.metrics = [
  RubyLLM::Monitoring::Metrics::Throughput,
  RubyLLM::Monitoring::Metrics::Cost,
  RubyLLM::Monitoring::Metrics::ResponseTime
]

Custom metrics

Create custom metrics by inheriting from RubyLLM::Monitoring::Metrics::Base:

class CostByFeature < RubyLLM::Monitoring::Metrics::Base
  title "Cost by Feature"
  unit "money"

  private

  def metric_data
    # Extract metadata from JSON payload and group by feature
    scope.group("json_extract(payload, '$.metadata.feature')").sum(:cost)
  end

  def build_series(aggregated_data)
    aggregated_data
      .group_by { |(_, feature), _| [feature || "unknown"] }
      .transform_values { |entries|
        entries.map { |(timestamp, _), value|
          [timestamp.to_i * 1000, value || default_value]
        }
      }
      .map { |keys, data| { name: keys.first, data: data } }
  end
end

The scope is an ActiveRecord relation of Event records grouped by time bucket. Your metric_data method should return aggregated data that will be displayed as a time series chart.

Note: JSON extraction syntax varies by database:

  • SQLite: json_extract(payload, '$.metadata.feature')
  • PostgreSQL: payload->'metadata'->>'feature'
  • MySQL: payload->>'$.metadata.feature'

This example assumes you're setting metadata using RubyLLM::Instrumentation.with():

RubyLLM::Instrumentation.with(feature: "chat_assistant") do
  RubyLLM.chat.ask("Hello")
end

Then add your custom metric to the configuration:

RubyLLM::Monitoring.metrics = [
  RubyLLM::Monitoring::Metrics::Throughput,
  RubyLLM::Monitoring::Metrics::Cost,
  CostByFeature
]

Alerts

RubyLLM::Monitoring can send alerts when certain conditions are met. Useful for monitoring cost, errors, etc.

Configuration

In config/initializers/ruby_llm_monitoring.rb, you can set the notification channels and alert rules:

RubyLLM::Monitoring.channels = {
  email: { to: "team@example.com" },
  slack: { webhook_url: ENV["SLACK_WEBHOOK_URL"] },
}

# Default cooldown between repeated alerts (optional, defaults to 5 minutes)
RubyLLM::Monitoring.alert_cooldown = 15.minutes

RubyLLM::Monitoring.alert_rules += [{
  time_range: -> { 1.hour.ago.. },
  rule: ->(events) { events.where.not(exception_class: nil).count > 10 },
  channels: [:slack],
  message: { text: "More than 10 errors in the last hour" }
}, {
  time_range: -> { Time.current.at_beginning_of_month.. },
  rule: ->(events) { events.sum(:cost) >= 500 },
  channels: [:email, :slack],
  message: { text: "More than $500 spent this month" }
}]

Rule options

Option Required Description
time_range Yes Lambda returning a range for filtering events (e.g., -> { 1.hour.ago.. })
rule Yes Lambda receiving events scope, returns true to trigger alert
channels Yes Array of channel names to notify
message Yes Hash with :text key for the alert message
cooldown No Override default cooldown for this rule

Built-in channels

Slack

RubyLLM::Monitoring.channels = {
  slack: {
    webhook_url: ENV["SLACK_WEBHOOK_URL"]
  }
}

Email

RubyLLM::Monitoring.channels = {
  email: {
    to: "team@example.com",
    from: "alerts@example.com",  # optional
    subject: "LLM Alert"         # optional
  }
}

Custom channels

Register custom notification channels:

class PagerDutyChannel < RubyLLM::Monitoring::Channels::Base
  def self.deliver(message, config)
    # Your implementation
    # message[:text] contains the alert text
    # config contains channel configuration
  end
end

RubyLLM::Monitoring.channel_registry.register(:pagerduty, PagerDutyChannel)

RubyLLM::Monitoring.channels = {
  pagerduty: { api_key: ENV["PAGERDUTY_API_KEY"] }
}

Data retention

RubyLLM::Monitoring records an event for every instrumented LLM call. Those events are kept indefinitely, the engine does not automatically prune them. If you need a retention policy, add it in your application using whatever cadence and storage requirements fit your deployment.

For example, to keep 90 days of monitoring data:

# app/jobs/prune_ruby_llm_monitoring_events_job.rb
class PruneRubyLLMMonitoringEventsJob < ApplicationJob
  queue_as :default

  def perform(retention_period = 90.days)
    RubyLLM::Monitoring::Event
      .where(created_at: ...retention_period.ago)
      .in_batches(of: 1_000)
      .delete_all
  end
end

Then schedule the job with your application's scheduler of choice, such as cron, Solid Queue recurring tasks, or GoodJob cron.

Contributing

You can open an issue or a PR in GitHub.

License

The gem is available as open source under the terms of the MIT License.