Envoy

Envoy is a mountable Rails engine for governed, tool-calling agent chat. It wraps RubyLLM (~> 1.16, legacy acts_as mode) with:

  • a small DSL for defining toolsets the host app owns (app/envoy/**),
  • a guard that runs every tool call through the host's own authorization and translates exceptions into model-legible error payloads,
  • persistence for conversations/messages/tool calls, streamed over Turbo,
  • a debug console (mounted wherever you like, e.g. /envoy) for driving and inspecting chats.

1. What Envoy is / when to use it

Use Envoy when you want to let an LLM take actions inside your app — not just answer questions — while keeping every action inside your app's own authorization model. Envoy is not an authorization framework and not a model registry: it assumes the host already has an actor concept (a User, an API token, whatever) and existing scoped-query methods on it (actor.things, actor.writable_things, ...). Envoy's job is to:

  • expose a small subset of those actions to the model as named, described, parameterized tools (the toolset DSL),
  • run every tool call through a guard that turns your app's own exceptions (ActiveRecord::RecordNotFound, a raised Envoy::Forbidden, ArgumentError) into a structured result the model can read and explain, rather than a stack trace or a silent failure,
  • persist the conversation (messages, tool calls, statuses) and stream the reply over Turbo so a chat UI can show it live,
  • gate mutation entirely: a conversation can be marked read_only, in which case tools with access :write are never even sent to the model.

It has no dependency on any specific domain model. Reach for it when you're building an in-app assistant that needs to call real methods on real records — not for open-ended chat with no tool surface (plain RubyLLM is simpler for that) and not as a substitute for your own policy/ACL layer (Envoy calls into that layer, it doesn't replace it).

2. Install

gem "envoy_ai"

The gem is published as envoy_ai (envoy was taken on RubyGems), but the namespace it defines is Envoy. require "envoy_ai" and require "envoy" both load it; Bundler's default require of the gem name works with no require: option.

3. Configure

Envoy has two separate configuration surfaces: RubyLLM's own (API keys, default model, logging) and Envoy's (which provider/models are allowed, how to authenticate a request, how to resolve the acting record). Set both in one initializer, config/initializers/envoy.rb:

RubyLLM.configure do |config|
  config.use_new_acts_as   = false  # legacy acts_as: string model_id, no models table
  config.anthropic_api_key = ENV["ANTHROPIC_API_KEY"].to_s
  config.default_model     = "claude-sonnet-4-5"
  config.logger            = Rails.logger
end

Envoy.configure do |config|
  # Which provider RubyLLM's `with_model` resolves against.
  config.provider         = :anthropic

  # Fallback model id when a conversation doesn't pin its own.
  config.default_model    = "claude-sonnet-4-5"

  # The actual allow-list for model ids. Because RubyLLM runs in legacy
  # acts_as mode there's no models table to constrain this at the DB level —
  # this array is the real gate a conversation's model_id is checked against.
  config.available_models = [ "claude-sonnet-4-5", "claude-opus-4-1" ]

  # Runs as a before_action in every Envoy controller. Raise/redirect to deny
  # the request entirely (this is where your app's login check goes).
  config.authenticate     = ->(controller) { controller.authenticate_user! }

  # Returns whatever polymorphic record chats/tool calls should be scoped to.
  # This is the `actor:` that perform blocks receive (see section 7).
  config.actor_resolver   = ->(controller) { controller.current_user }

  # Prepended to every conversation's instructions, ahead of the toolset
  # description and the conversation's own system prompt. Optional — defaults
  # to a generic "You are a helpful assistant..." string.
  config.system_preamble  = "You are the assistant embedded in Acme's app."

  # Advanced/testing seam: a callable that builds the object driving a turn
  # (`#run(content:, tools:, instructions:) { |kind, payload| ... }`). Defaults
  # to `->(**opts) { Envoy::LLM.new(**opts) }`, which drives a real RubyLLM
  # `acts_as_chat`. Tests swap this for `Envoy::Testing::FakeLLM` — see
  # section 10.
  # config.llm = ->(conversation:) { Envoy::LLM.new(conversation: conversation) }
end

# Host toolset definitions live under app/envoy/**. They register themselves
# via Envoy.define_toolset at load time (side effects, not class bodies), so
# Zeitwerk's autoloader never triggers them on its own -- force-load them
# here instead.
Rails.application.config.to_prepare do
  Dir[Rails.root.join("app/envoy/**/*.rb")].each { |f| require_dependency f }
end

config.authenticate and config.actor_resolver are the two hooks that tie Envoy to the host's session. Everything a tool does should be reached through methods on the resolved actor (see section 8).

Because toolset files register themselves via a top-level Envoy.define_toolset(...) call rather than defining a class/module Zeitwerk can map to a filename, app/envoy must also be excluded from autoloading (config/application.rb):

Rails.autoloaders.main.ignore(Rails.root.join("app/envoy"))

4. Mount

# config/routes.rb
mount Envoy::Engine, at: "/envoy"

This mounts the debug console (a conversation list, a chat view, and system prompt CRUD) at whatever path you choose. Every request goes through Envoy::ApplicationController's before_action, which calls your authenticate lambda first, so the console inherits your app's own login. Conversations are scoped to envoy_current_actor (the actor_resolver result) — a user can only see and post to their own conversations.

5. Migrations

Envoy ships its own migrations (envoy_conversations, envoy_messages, envoy_tool_calls, envoy_system_prompts, envoy_system_prompt_versions, plus the FK linking a conversation to the system-prompt version it used):

bin/rails envoy:install:migrations
bin/rails db:migrate

6. Assets

The engine ships two Stimulus controllers under its own app/javascript (Enter-to-submit for the composer, autoscroll for the streaming transcript), exposed to the host's asset pipeline by the engine itself (the envoy.assets initializer adds engines/envoy — or wherever the gem is installed — app/javascript to config.assets.paths). Pin and register them in the host:

# config/importmap.rb
pin "envoy/controllers/composer_controller", to: "envoy/controllers/composer_controller.js"
pin "envoy/controllers/autoscroll_controller", to: "envoy/controllers/autoscroll_controller.js"

// app/javascript/controllers/index.js
import EnvoyComposer from "envoy/controllers/composer_controller"
import EnvoyAutoscroll from "envoy/controllers/autoscroll_controller"
application.register("envoy-composer", EnvoyComposer)
application.register("envoy-autoscroll", EnvoyAutoscroll)

If the host uses Tailwind with a content scan, add the engine's views to it (they live outside app/views):

@source "../../../<path-to-envoy-gem>/app/views/**/*.erb";

7. Toolsets DSL

A toolset is a named group of tools the host defines with a small DSL: define_toolset, description, tool, access, param, perform, and use for composing other toolsets in. Toolset files live under the host's app/envoy/** (one file per toolset is the convention) and self-register at load — see the to_prepare force-load block in section 3. For example, app/envoy/documents.rb:

Envoy.define_toolset :documents do
  description "Search documents and create new ones. Documents are " \
              "referenced by id."

  tool :search_documents do
    description "Search documents by title or body to get their id."
    access :read
    param :query, "Text to search for."
    perform do |actor:, query:|
      actor.visible_documents.where("title LIKE ?", "%#{query}%")
           .map { |d| { id: d.id, title: d.title } }
    end
  end

  tool :create_document do
    description "Create a new document."
    access :write
    param :title, "The document's title."
    param :body, "The document's body text."
    perform do |actor:, title:, body:|
      document = actor.writable_documents.create!(title: title, body: body)  # <-- authorization
      { created: true, document_id: document.id }
    end
  end
end

param takes a name, a description, and optional type: (:string by default) and required: (true by default) — these compile straight into RubyLLM::Tool parameter declarations.

Toolsets can compose other toolsets with use, so a conversation-level toolset can be assembled from smaller pieces without duplicating tool definitions:

Envoy.define_toolset :assistant do
  description "General assistant tools for this app."
  use :documents, :calendar
end

use resolves the named toolsets lazily (at tools_for/description time), so definition order across files doesn't matter; a composition cycle (a toolset that composes itself, directly or transitively) raises Envoy::ToolsetCycle.

access :read vs access :write: a conversation with status: "read_only" only receives tools whose access is :read — write tools are filtered out of the merged tool list entirely, so the model never even sees them. This is the governance lever for embedding a chat somewhere that must never mutate state.

8. Actor & authorization contract

actor is polymorphic — whatever your actor_resolver returns (a User, an API client record, anything). It's the sole authorization contract: every perform block receives it as a keyword argument alongside its declared params, and every lookup or mutation inside perform is expected to be scoped through it (actor.visible_documents.find(id) for reads, actor.writable_documents.create!(...) for writes) rather than through an unscoped model class.

There's no separate policy object or ACL layer inside Envoy. If a scoped lookup can't find the record, find raises ActiveRecord::RecordNotFound and the Guard turns that into a not_found result. If you want a clearer, intentional denial instead — the record exists but this actor may never touch it — raise Envoy::Forbidden directly from inside perform:

perform do |actor:, document_id:|
  document = Document.find(document_id)
  raise Envoy::Forbidden, "Not your document." unless document.owner == actor
  document.destroy!
  { deleted: true }
end

9. Guard semantics

Envoy::Guard.run wraps every perform block and translates what it raises or returns into a model-legible result — it never lets a tool call raise past it:

Raised / returned type tool-call status
Envoy::Forbidden forbidden denied (message relayed to the model)
ActiveRecord::RecordNotFound not_found failed (generic message, not the exception text)
ArgumentError invalid failed (message relayed to the model)
any other StandardError error failed (message not leaked to the model)
block returns a Hash with an :error key passthrough (whatever :error/other keys the block set) failed
anything else — succeeded

The model is told this convention in its instructions (Envoy::Runner::ERROR_CONVENTION) so it can explain a failure to the user instead of retrying blindly. Only Envoy::Forbidden and ArgumentError messages (both host-authored, so safe by construction) and the fixed not_found string reach the model; arbitrary StandardError messages are swallowed so internals never leak through a tool result.

10. Testing

Real RubyLLM calls hit the network, so the engine exposes one seam: Envoy.config.llm — a callable that builds whatever object satisfies #run(content:, tools:, instructions:) { |kind, payload| ... } for a given conversation. In production this defaults to Envoy::LLM.new(conversation:), which drives acts_as_chat for real. Tests swap in Envoy::Testing::FakeLLM, which persists the same Envoy::Message / Envoy::ToolCall rows a real run would, but replays a fixed script of directives instead of calling out to Anthropic:

Envoy.config.llm = ->(conversation:) do
  fake = Envoy::Testing::FakeLLM.new(conversation: conversation)
  fake.script = [
    { tool: "find_athlete", args: { name: "Travis" } },
    { tool: "log_score", args: { athlete_id: 1, scheduled_workout_id: 2,
                                  fields: { time_seconds: 500 } } },
    { text: "Logged your time of 8:20. Nice work!" },
  ]
  fake
end

Each {tool:, args:} directive actually invokes the compiled tool (through the real Guard, so denials/not-found/invalid branches are exercised), and each {text:} directive appends to the assistant's message content (multiple text directives accumulate, like real streaming deltas). Reset with Envoy.config.llm = nil in teardown — don't nil the whole Envoy.config object, or your authenticate/actor_resolver lambdas are lost for the rest of the (single-process) test run.

This covers everything except the actual RubyLLM ↔ provider wire protocol — that path needs a real API key and network access, so it isn't something FakeLLM (or CI) can exercise.

11. Development

git clone git@github.com:ellym/envoy.git
cd envoy
bundle install
bundle exec bin/test

bin/test is the gem's own runner (there's no root bin/rails here, since this is a gem, not an app) — it runs the full Minitest suite against the bundled test/dummy Rails app under test/dummy. Always invoke it with bundle exec so it resolves against this gem's own Gemfile.lock rather than whatever Rails/RubyLLM versions happen to be installed globally.

To run a single file or line, pass it straight through:

bundle exec bin/test test/envoy/guard_test.rb
bundle exec bin/test test/envoy/guard_test.rb:12

The debug console

Envoy ships a minimal chat UI mounted at whatever path the host chooses (section 4). Posting a message (MessagesController#create) enqueues Envoy::RunJob, which runs Envoy::Runner and broadcasts :delta (streamed reply text) and :tool (a completed tool call) events over Turbo Streams to the conversation's own broadcast channel. The view renders tool calls as collapsible <details> chips showing the call name, an outcome badge (succeeded/denied/failed), arguments, and the raw JSON result.

Real-gem notes worth recording

A few things about the installed RubyLLM (~> 1.16) that weren't obvious from its docs and cost time to pin down:

  • Legacy acts_as mode (config.use_new_acts_as = false): conversations store a plain string model_id column; there is no separate models table/migration to install. This is deliberate — Envoy doesn't manage a model registry, the host just lists allowed model ids (Envoy.config.available_models).
  • with_model(id, provider:, assume_exists: true) — note the keyword is assume_exists:, not assume_model_exists:. Passing the wrong keyword silently gets ignored by Ruby's **opts splatting in some call shapes and is easy to miss; verify against the installed gem (ruby_llm/active_record/chat_methods.rb, ruby_llm/chat.rb) if upgrading.
  • Every chat call passes assume_exists: true so a model_id that isn't in RubyLLM's own registered-models list (e.g. a new Claude model id before RubyLLM ships an update) still works — Envoy's own available_models list is the actual gate, not RubyLLM's.
  • commonmarker "~> 2.6" — pin this explicitly if your app also depends on it. The 1.1.x line does not build against Ruby 4.0's native extension toolchain.

Manual live smoke (not run in CI)

This validates the real RubyLLM/Anthropic path that FakeLLM stands in for in tests. It requires a real ANTHROPIC_API_KEY and network access, so it's a manual, non-CI gate — run it in a host app with Envoy mounted:

export ANTHROPIC_API_KEY=sk-ant-...
bin/dev
  1. Sign in as a web user.
  2. Visit the mounted path, start a new conversation with a toolset on claude-sonnet-4-5.
  3. Ask it to call one of the write tools it exposes.
  4. Confirm:
    • tool-call chips appear with a succeeded badge,
    • the assistant's reply streams in (not just appears fully-formed),
    • the underlying record was actually created/updated.

If this hasn't been run against the current code, treat RubyLLM-facing changes (model ids, with_model kwargs, provider config) as unverified until someone runs it with a key.