Coolhand Ruby Monitor

Gem Version

Monitor and log LLM API calls — OpenAI, Anthropic, Google Gemini, Cohere, and any Faraday-based or Net::HTTP client — to the Coolhand analytics platform. Supports Ruby LLM monitoring, request logging, and feedback collection.

Scope: Coolhand intercepts outgoing HTTP requests to configured LLM API endpoints only. It does not read, scan, or transmit your source code files.

Installation

gem 'coolhand'

Getting Started

  1. Get API Key: Visit coolhandlabs.com to create a free account
  2. Install: gem install coolhand
  3. Initialize: Add configuration to your Ruby application
  4. Configure: Set your API key in the configuration block
  5. Deploy: Your AI calls are now automatically monitored!

Quick Start

Automatic Global Monitoring

🔥 Set it and forget it! Monitor ALL AI API calls across your entire application with minimal configuration.

# Add this configuration at the start of your application
require 'coolhand'

Coolhand.configure do |config|
  config.api_key = 'your_api_key_here'
  config.silent = true  # Set to false for debug output
end

# That's it! ALL AI API calls are now automatically monitored:
# ✅ OpenAI SDK calls
# ✅ Anthropic API calls
# ✅ Direct HTTP requests to AI APIs
# ✅ ANY library making AI API calls via Faraday

# NO code changes needed in your existing services!

✨ Why Automatic Monitoring:

  • 🚫 Zero refactoring - No code changes to existing services
  • 📊 Complete coverage - Monitors ALL AI libraries using Faraday automatically
  • 🔒 Security built-in - Automatic credential sanitization
  • ⚡ Performance optimized - Negligible overhead via async logging
  • 🛡️ Future-proof - Automatically captures new AI calls added by your team

Self-Hosted Deployments

Point the SDK at your own Coolhand-compatible endpoint via config.base_url for compliance or data-residency requirements. See Self-Hosted Deployments →.

Feedback API

Collect feedback on LLM responses to improve model performance.

Frontend Feedback Widget: For browser-based feedback collection, see coolhand-js - an accessible, lightweight JavaScript widget that leverages best UX practices to capture actionable user feedback on any AI output.

require 'coolhand'

# Create feedback for an LLM response
feedback_service = Coolhand::FeedbackService.new

feedback = feedback_service.create_feedback(
  llm_request_log_id: 'abc123def456', # hashid from a prior response; a raw integer FK also still works
  llm_provider_unique_id: 'req_xxxxxxx',
  client_unique_id: 'workorder-chat-456',
  creator_unique_id: 'user-789',
  original_output: 'Here is the original LLM response!',
  revised_output: 'Here is the human edit of the original LLM response.',
  explanation: 'Tone of the original response read like AI-generated open source README docs',
  sentiment: 'dislike'
)

For a full field reference and matching strategy, see Feedback API →.

Rails Integration

Configuration

Create an initializer file at config/initializers/coolhand.rb:

# config/initializers/coolhand.rb
Coolhand.configure do |config|
  # Your Coolhand API Key (Required)
  # Best practice: Use Rails credentials or environment-specific configuration
  config.api_key = Rails.application.credentials.coolhand_api_key

  # Set to true to suppress console output
  config.silent = Rails.env.production?

  # Specify which LLM endpoints to intercept (array of strings)
  # Optional - defaults to OpenAI, Anthropic, ElevenLabs, Google Gemini, and GitHub Models
  # config.intercept_addresses = ["api.openai.com", "api.anthropic.com", "api.cohere.ai"]
end

Rails Controller Example

class ChatController < ApplicationController
  def create_feedback
    feedback_service = Coolhand::FeedbackService.new

    feedback = feedback_service.create_feedback(
      llm_request_log_id: params[:log_id],
      creator_unique_id: current_user.id,
      original_output: params[:original_response],
      revised_output: params[:edited_response],
      explanation: params[:feedback_text],
      sentiment: params[:sentiment]
    )

    if feedback
      render json: { success: true, message: 'Feedback recorded' }
    else
      render json: { success: false, message: 'Failed to record feedback' }, status: 422
    end
  end
end

Background Job Example

class FeedbackCollectionJob < ApplicationJob
  def perform(feedback_data)
    feedback_service = Coolhand::FeedbackService.new

    feedback_service.create_feedback(
      llm_provider_unique_id: feedback_data[:request_id],
      creator_unique_id: feedback_data[:user_id],
      original_output: feedback_data[:original],
      explanation: feedback_data[:reason],
      sentiment: feedback_data[:sentiment]
    )
  end
end

Configuration Options

Configuration Parameters

Option Type Default Description
api_key String nil Your Coolhand API key. If absent, intercepted requests are skipped with a warning log rather than raising at boot time
enabled Boolean true Set to false to disable all patching and validation (e.g. Rails.env.production?)
capture Boolean true Whether to capture and forward intercepted requests. Set to false to monitor without forwarding, then use Coolhand.with_capture to re-enable selectively
silent Boolean false Whether to suppress console output
intercept_addresses Array ["api.openai.com", "api.anthropic.com"] Array of API endpoint strings to monitor

Usage Examples

With OpenAI Ruby Client

require 'openai'
require 'coolhand'

# Configure Coolhand
Coolhand.configure do |config|
  config.api_key = 'your_api_key_here'
end

# Use OpenAI normally - requests are automatically logged
client = OpenAI::Client.new(access_token: ENV['OPENAI_API_KEY'])

response = client.chat(
  parameters: {
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: "These pretzels are making me thirsty!"}],
    temperature: 0.7
  }
)

puts response.dig("choices", 0, "message", "content")
# The request and response have been automatically logged to Coolhand!

📖 Complete Anthropic Integration Guide → - Supports both official and community gems with automatic detection

Selective Capture

Use Coolhand.with_capture and Coolhand.without_capture to override the global capture setting for a specific block of code, without changing your configuration.

Capture a specific call when capture is globally disabled:

Coolhand.configure do |config|
  config.api_key = 'your_api_key_here'
  config.capture = false  # disabled globally
end

Coolhand.with_capture do
  response = openai_client.chat(parameters: {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Hello!" }],
  })
  # This request is captured and forwarded to Coolhand
end

Skip capture for a specific call when capture is globally enabled:

Coolhand.without_capture do
  response = openai_client.chat(parameters: { ... })
  # This request is NOT forwarded to Coolhand
end

Both methods are thread-safe. Blocks nest correctly — the innermost block takes precedence, and the outer setting is restored when the block exits.

Logging Inbound Webhooks

For inbound webhooks (like audio transcripts or tool calls), the automatic interceptor won't capture them since they're incoming requests TO your application. In these cases, use the simple forward_webhook helper method:

Webhook Forwarding Example (Recommended)

class WebhooksController < ApplicationController
  def elevenlabs
    raw_body = request.body.read
    webhook_data = JSON.parse(raw_body)

    # Forward to Coolhand with automatic field generation and binary filtering
    Thread.new do
      Coolhand.logger_service.forward_webhook(
        webhook_body: webhook_data,        # Required: webhook payload
        source: "elevenlabs",              # Required: service name
        event_type: webhook_data["type"],  # Optional: e.g., post_call_transcription
        headers: request.headers,          # Optional & recommended
      )
    end

    render json: { status: "success" }
  end
end

Required Parameters:

  • webhook_body - The webhook payload (Hash or parsed JSON)
  • source - Service name (String, e.g., "elevenlabs", "stripe", "twilio")

Optional Parameters:

  • event_type - Event type to append to URL (e.g., "post_call_transcription" → webhook://elevenlabs/post_call_transcription)
  • headers - Request headers (automatically sanitized)
  • conversation_id, agent_id, metadata - Custom fields for your tracking needs

Error Handling:

  • Silent mode = false: Raises ArgumentError if required parameters are missing
  • Silent mode = true: Logs warning and returns false if required parameters are missing

That's it! The forward_webhook method automatically:

  • ✅ Generates unique ID and timestamp
  • ✅ Filters out binary data (audio, images, etc.)

What Gets Logged

The monitor captures:

  • Request Data: Method, URL, headers, request body
  • Response Data: Status code, headers, response body
  • Metadata: Timestamp, protocol used
  • LLM-Specific: Model used, token counts, temperature settings

Headers containing API keys are automatically sanitized for security.

Binary Data Filtering

Coolhand does not track or store binary data. The gem automatically filters out:

  • Audio files and data (audio, audio_data, full_audio, raw_audio)
  • Image data (image_data, image_content)
  • File content (file_content, binary_content)
  • Base64 encoded data (audio_base64, base64_data)
  • Voice samples and audio URLs

This ensures:

  • ✅ Smaller payloads - Only text and metadata are sent
  • ✅ Faster processing - No bandwidth wasted on binary data
  • ✅ Privacy focused - Audio/video content never leaves your infrastructure
  • ✅ Clean logs - Focus on conversational data, not media files

The filtering is automatic and applies to all monitored API calls and webhook logging.

Supported Libraries

The monitor works with multiple transport layers and Ruby libraries:

Faraday-based libraries:

  • OpenAI Ruby SDK
  • ruby-anthropic gem (community Anthropic gem)
  • ruby-openai gem
  • LangChain.rb
  • Direct Faraday requests
  • Any other Faraday-based HTTP client

Native HTTP libraries:

  • Official Anthropic Ruby SDK (using Net::HTTP)
  • GitHub Models SDK / any client using models.github.ai
  • Any library using Net::HTTP directly

Universal Coverage: Since most Ruby HTTP libraries use Net::HTTP under the hood, Coolhand's single interceptor provides comprehensive monitoring without needing library-specific integrations.

How It Works

Coolhand uses a unified Net::HTTP interceptor to capture outgoing requests to configured LLM API endpoints:

Net::HTTP Interceptor

  • Extends Ruby's Net::HTTP using Module#prepend (a standard Ruby technique for wrapping library behavior)
  • Covers HTTP libraries that delegate to Net::HTTP under the hood (which is most of them)
  • Handles both standard requests and streaming responses via read_body interception
  • Thread-safe design using thread-local storage for streaming buffers

Request Flow

When a request matches configured LLM endpoints:

  1. The original request executes normally with zero performance impact
  2. Request and response data (body, headers, status) are captured by the interceptor
  3. For streaming requests, the complete accumulated response is captured (not individual chunks)
  4. Data is sent to the Coolhand API asynchronously in a background thread
  5. Your application continues without interruption

For non-matching endpoints, requests pass through unchanged.

Troubleshooting

Debugging Output

Enable verbose logging to see what's being intercepted:

Coolhand.configure do |config|
  config.api_key = 'your_api_key_here'
  config.silent = false  # Enable console output
end

Testing

In test environments, you may want to configure differently:

# config/initializers/coolhand.rb
if Rails.env.test?
  Coolhand.configure do |config|
    config.api_key = 'test_key'
    config.silent = true
  end
end

Non-Rails Applications

For standard Ruby scripts or non-Rails applications:

#!/usr/bin/env ruby
require 'coolhand'

Coolhand.configure do |config|
  config.api_key = 'your_api_key_here'  # Store securely, don't commit to git
  config.environment = 'production'
  config.silent = false
end

# Your application code here...

API Key

🆓 Sign up for free at coolhandlabs.com to get your API key and start monitoring your LLM usage.

What you get:

  • Complete LLM request and response logging
  • Usage analytics and insights
  • Feedback collection and quality scoring
  • No credit card required to start

Error Handling

The monitor handles errors gracefully:

  • Failed API logging attempts are logged to console but don't interrupt your application
  • Invalid API keys will be reported but won't crash your app
  • Network issues are handled with appropriate error messages

Batch webhook handler (OpenAI)

Automatically handle OpenAI batch event logs (batch.completed, batch.failed, batch.expired, batch.cancelled) by intercepting webhook requests and enqueuing your batch result processor.

Usage:

  • Include the interceptor in your controller: include Coolhand::WebhookInterceptor
  • Add the before_action to validate and populate @validator payload: before_action :intercept_batch_request, only: :openai
  • Ensure you skip CSRF for the webhook endpoint: skip_before_action :verify_authenticity_token
  • Override the webhook_secret method to return your OpenAI webhook secret

Minimal example (only key lines shown):

# app/controllers/webhooks/batch_api_requests_controller.rb
# ...existing code...
include Coolhand::WebhookInterceptor

skip_before_action :verify_authenticity_token
before_action :intercept_batch_request, only: :openai

def openai
  event = JSON.parse(@validator.payload)
  case event["type"]
  when "batch.completed", "batch.failed", "batch.expired", "batch.cancelled"
    batch_id = event.dig("data", "id")
    batch_request = BatchApiRequest.find_by(provider: "openai", provider_batch_id: batch_id)

    if batch_request
      OpenAi::BatchResultProcessor.perform_async(batch_request.id)
      Rails.logger.info("Queued batch result processing for BatchApiRequest #{batch_request.id}")
    else
      Rails.logger.warn("Could not find BatchApiRequest for OpenAI batch ID: #{batch_id}")
    end
  else
    Rails.logger.info("Unhandled OpenAI webhook event type: #{event["type"]}")
  end

  head :ok
rescue JSON::ParserError
  head :bad_request
rescue StandardError => e
  Rails.logger.error("OpenAI webhook error: #{e.message}")
  head :internal_server_error
end

def webhook_secret
  Rails.application.credentials.openai_webhook_secret
end
# ...existing code...

Batch webhook handler (Vertex)

Automatically handle Vertex batch event logs.

Usage:

  • call Coolhand::Vertex::BatchResultProcessor service with batch_info and download batch results

Minimal example (only key lines shown):

class Vertex::BatchCallbackProcessor < BaseService
  option :batch_request, model: BatchApiRequest
  option :batch_info

  def call
    case batch_info["state"]
    when "JOB_STATE_PENDING"
      nil
    when "JOB_STATE_RUNNING", "JOB_STATE_QUEUED"
      batch_request.update!(status: "processing")

      Coolhand::Vertex::BatchResultProcessor.new(batch_info:).call
    when "JOB_STATE_SUCCEEDED"
      output_file_id = batch_info["outputInfo"]["gcsOutputDirectory"]
      results = download_batch_results(output_file_id)
      results.each { |batch_item| process_batch_result(batch_item) }

      batch_request.update!(status: "completed", completed_at: Time.current, output_file_id:)

      Coolhand::Vertex::BatchResultProcessor.new(batch_info:).call(results)

      # Clean up GCS files after successful processing
      cleanup_gcs_files(output_file_id)
    when "JOB_STATE_FAILED"
      handle_failed_batch(batch_info["error"]["message"])
    end
  end
end

Documentation

  • Configuration — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
  • Feedback API — Full field reference, matching strategies, sentiment values
  • Anthropic Integration — Official and community Anthropic Ruby gems, streaming, dual gem handling, and troubleshooting
  • ElevenLabs Integration — Webhook capture, feedback submission, and Rails integration

Security

  • API keys in request headers are automatically redacted
  • No sensitive data is exposed in logs
  • All data is sent via HTTPS to Coolhand servers
  • Frontend (Feedback Collection Widget): coolhand-js - Frontend feedback widget for collecting user feedback on AI outputs
  • Node.js: coolhand-node package - Coolhand monitoring for Node.js applications
  • Python: coolhand package - Coolhand monitoring for Python applications

Community

About Coolhand Labs

Coolhand Labs builds LLM observability and feedback tooling so teams can monitor, understand, and improve their AI applications. Learn more at coolhandlabs.com.

License

Apache-2.0