Envoy

Envoy is a mountable Rails engine for governed, tool-calling agent chat. It wraps RubyLLM (~> 1.16, legacy acts_as mode) with:

a small DSL for defining toolsets the host app owns (app/envoy/**),
a guard that runs every tool call through the host's own authorization and translates exceptions into model-legible error payloads,
persistence for conversations/messages/tool calls, streamed over Turbo,
a debug console (mounted wherever you like, e.g. /envoy) for driving and inspecting chats.

1. What Envoy is / when to use it

Use Envoy when you want to let an LLM take actions inside your app — not just answer questions — while keeping every action inside your app's own authorization model. Envoy is not an authorization framework and not a model registry: it assumes the host already has an actor concept (a User, an API token, whatever) and existing scoped-query methods on it (actor.things, actor.writable_things, ...). Envoy's job is to:

expose a small subset of those actions to the model as named, described, parameterized tools (the toolset DSL),
run every tool call through a guard that turns your app's own exceptions (ActiveRecord::RecordNotFound, a raised Envoy::Forbidden, ArgumentError) into a structured result the model can read and explain, rather than a stack trace or a silent failure,
persist the conversation (messages, tool calls, statuses) and stream the reply over Turbo so a chat UI can show it live,
gate mutation entirely: a conversation can be marked read_only, in which case tools with access :write are never even sent to the model.

It has no dependency on any specific domain model. Reach for it when you're building an in-app assistant that needs to call real methods on real records — not for open-ended chat with no tool surface (plain RubyLLM is simpler for that) and not as a substitute for your own policy/ACL layer (Envoy calls into that layer, it doesn't replace it).

2. Install

gem "envoy_ai"

The gem is published as envoy_ai (envoy was taken on RubyGems), but the namespace it defines is Envoy. require "envoy_ai" and require "envoy" both load it; Bundler's default require of the gem name works with no require: option.

3. Configure

Envoy has two separate configuration surfaces: RubyLLM's own (API keys, default model, logging) and Envoy's (which provider/models are allowed, how to authenticate a request, how to resolve the acting record). Set both in one initializer, config/initializers/envoy.rb:

RubyLLM.configure do |config|
  config.use_new_acts_as   = false  # legacy acts_as: string model_id, no models table
  config.anthropic_api_key = ENV["ANTHROPIC_API_KEY"].to_s
  config.default_model     = "claude-sonnet-4-5"
  config.logger            = Rails.logger
end

Envoy.configure do |config|
  # Which provider RubyLLM's `with_model` resolves against.
  config.provider         = :anthropic

  # Fallback model id when a conversation doesn't pin its own.
  config.default_model    = "claude-sonnet-4-5"

  # The actual allow-list for model ids. Because RubyLLM runs in legacy
  # acts_as mode there's no models table to constrain this at the DB level —
  # this array is the real gate a conversation's model_id is checked against.
  config.available_models = [ "claude-sonnet-4-5", "claude-opus-4-1" ]

  # Runs as a before_action in every Envoy controller. Raise/redirect to deny
  # the request entirely (this is where your app's login check goes).
  config.authenticate     = ->(controller) { controller.authenticate_user! }

  # Returns whatever polymorphic record chats/tool calls should be scoped to.
  # This is the `actor:` that perform blocks receive (see section 7).
  config.actor_resolver   = ->(controller) { controller.current_user }

  # Prepended to every conversation's instructions, ahead of the toolset
  # description and the conversation's own system prompt. Optional — defaults
  # to a generic "You are a helpful assistant..." string.
  config.system_preamble  = "You are the assistant embedded in Acme's app."

  # Advanced/testing seam: a callable that builds the object driving a turn
  # (`#run(content:, tools:, instructions:) { |kind, payload| ... }`). Defaults
  # to `->(**opts) { Envoy::LLM.new(**opts) }`, which drives a real RubyLLM
  # `acts_as_chat`. Tests swap this for `Envoy::Testing::FakeLLM` — see
  # section 10.
  # config.llm = ->(conversation:) { Envoy::LLM.new(conversation: conversation) }
end

# Host toolset definitions live under app/envoy/**. They register themselves
# via Envoy.define_toolset at load time (side effects, not class bodies), so
# Zeitwerk's autoloader never triggers them on its own -- force-load them
# here instead.
Rails.application.config.to_prepare do
  Dir[Rails.root.join("app/envoy/**/*.rb")].each { |f| require_dependency f }
end

config.authenticate and config.actor_resolver are the two hooks that tie Envoy to the host's session. Everything a tool does should be reached through methods on the resolved actor (see section 8).

Because toolset files register themselves via a top-level Envoy.define_toolset(...) call rather than defining a class/module Zeitwerk can map to a filename, app/envoy must also be excluded from autoloading (config/application.rb):

Rails.autoloaders.main.ignore(Rails.root.join("app/envoy"))

4. Mount

# config/routes.rb
mount Envoy::Engine, at: "/envoy"

This mounts the debug console (a conversation list, a chat view, and system prompt CRUD) at whatever path you choose. Every request goes through Envoy::ApplicationController's before_action, which calls your authenticate lambda first, so the console inherits your app's own login. Conversations are scoped to envoy_current_actor (the actor_resolver result) — a user can only see and post to their own conversations.

5. Migrations

Envoy ships its own migrations (envoy_conversations, envoy_messages, envoy_tool_calls, plus surface_key/context_key/prompt_key columns on envoy_conversations for surface-bound and DSL-prompt conversations):

bin/rails envoy:install:migrations
bin/rails db:migrate

If you installed Envoy before this version, envoy:install:migrations also copies over a migration that drops envoy_system_prompts and envoy_system_prompt_versions — see the Unreleased section of the CHANGELOG before running db:migrate on an app with real rows in those tables.

6. Assets

The engine ships two Stimulus controllers under its own app/javascript (Enter-to-submit for the composer, autoscroll for the streaming transcript), exposed to the host's asset pipeline by the engine itself (the envoy.assets initializer adds engines/envoy — or wherever the gem is installed — app/javascript to config.assets.paths). Pin and register them in the host:

# config/importmap.rb
pin "envoy/controllers/composer_controller", to: "envoy/controllers/composer_controller.js"
pin "envoy/controllers/autoscroll_controller", to: "envoy/controllers/autoscroll_controller.js"

// app/javascript/controllers/index.js
import EnvoyComposer from "envoy/controllers/composer_controller"
import EnvoyAutoscroll from "envoy/controllers/autoscroll_controller"
application.register("envoy-composer", EnvoyComposer)
application.register("envoy-autoscroll", EnvoyAutoscroll)

If the host uses Tailwind with a content scan, add the engine's views to it (they live outside app/views):

@source "../../../<path-to-envoy-gem>/app/views/**/*.erb";

7. Toolsets DSL

A toolset is a named group of tools the host defines with a small DSL: define_toolset, description, tool, access, param, perform, and use for composing other toolsets in. Toolset files live under the host's app/envoy/** (one file per toolset is the convention) and self-register at load — see the to_prepare force-load block in section 3. For example, app/envoy/documents.rb:

Envoy.define_toolset :documents do
  description "Search documents and create new ones. Documents are " \
              "referenced by id."

  tool :search_documents do
    description "Search documents by title or body to get their id."
    access :read
    param :query, "Text to search for."
    perform do |actor:, query:|
      actor.visible_documents.where("title LIKE ?", "%#{query}%")
           .map { |d| { id: d.id, title: d.title } }
    end
  end

  tool :create_document do
    description "Create a new document."
    access :write
    param :title, "The document's title."
    param :body, "The document's body text."
    perform do |actor:, title:, body:|
      document = actor.writable_documents.create!(title: title, body: body)  # <-- authorization
      { created: true, document_id: document.id }
    end
  end
end

param takes a name, a description, and optional type: (:string by default) and required: (true by default) — these compile straight into RubyLLM::Tool parameter declarations.

Toolsets can compose other toolsets with use, so a conversation-level toolset can be assembled from smaller pieces without duplicating tool definitions:

Envoy.define_toolset :assistant do
  description "General assistant tools for this app."
  use :documents, :calendar
end

use resolves the named toolsets lazily (at tools_for/description time), so definition order across files doesn't matter; a composition cycle (a toolset that composes itself, directly or transitively) raises Envoy::ToolsetCycle.

access :read vs access :write: a conversation with status: "read_only" only receives tools whose access is :read — write tools are filtered out of the merged tool list entirely, so the model never even sees them. This is the governance lever for embedding a chat somewhere that must never mutate state.

8. Actor & authorization contract

actor is polymorphic — whatever your actor_resolver returns (a User, an API client record, anything). It's the sole authorization contract: every perform block receives it as a keyword argument alongside its declared params, and every lookup or mutation inside perform is expected to be scoped through it (actor.visible_documents.find(id) for reads, actor.writable_documents.create!(...) for writes) rather than through an unscoped model class.

There's no separate policy object or ACL layer inside Envoy. If a scoped lookup can't find the record, find raises ActiveRecord::RecordNotFound and the Guard turns that into a not_found result. If you want a clearer, intentional denial instead — the record exists but this actor may never touch it — raise Envoy::Forbidden directly from inside perform:

perform do |actor:, document_id:|
  document = Document.find(document_id)
  raise Envoy::Forbidden, "Not your document." unless document.owner == actor
  document.destroy!
  { deleted: true }
end

9. Guard semantics

Envoy::Guard.run wraps every perform block and translates what it raises or returns into a model-legible result — it never lets a tool call raise past it:

Raised / returned	`type`	tool-call `status`
`Envoy::Forbidden`	`forbidden`	`denied` (message relayed to the model)
`ActiveRecord::RecordNotFound`	`not_found`	`failed` (generic message, not the exception text)
`ArgumentError`	`invalid`	`failed` (message relayed to the model)
any other `StandardError`	`error`	`failed` (message not leaked to the model)
block returns a Hash with an `:error` key	passthrough (whatever `:error`/other keys the block set)	`failed`
anything else	—	`succeeded`

The model is told this convention in its instructions (Envoy::Runner::ERROR_CONVENTION) so it can explain a failure to the user instead of retrying blindly. Only Envoy::Forbidden and ArgumentError messages (both host-authored, so safe by construction) and the fixed not_found string reach the model; arbitrary StandardError messages are swallowed so internals never leak through a tool result.

10. Testing

Real RubyLLM calls hit the network, so the engine exposes one seam: Envoy.config.llm — a callable that builds whatever object satisfies #run(content:, tools:, instructions:) { |kind, payload| ... } for a given conversation. In production this defaults to Envoy::LLM.new(conversation:), which drives acts_as_chat for real. Tests swap in Envoy::Testing::FakeLLM, which persists the same Envoy::Message / Envoy::ToolCall rows a real run would, but replays a fixed script of directives instead of calling out to Anthropic:

Envoy.config.llm = ->(conversation:) do
  fake = Envoy::Testing::FakeLLM.new(conversation: conversation)
  fake.script = [
    { tool: "find_athlete", args: { name: "Travis" } },
    { tool: "log_score", args: { athlete_id: 1, scheduled_workout_id: 2,
                                  fields: { time_seconds: 500 } } },
    { text: "Logged your time of 8:20. Nice work!" },
  ]
  fake
end

Each {tool:, args:} directive actually invokes the compiled tool (through the real Guard, so denials/not-found/invalid branches are exercised), and each {text:} directive appends to the assistant's message content (multiple text directives accumulate, like real streaming deltas). Reset with Envoy.config.llm = nil in teardown — don't nil the whole Envoy.config object, or your authenticate/actor_resolver lambdas are lost for the rest of the (single-process) test run.

This covers everything except the actual RubyLLM ↔ provider wire protocol — that path needs a real API key and network access, so it isn't something FakeLLM (or CI) can exercise.

11. Development

git clone git@github.com:ellym/envoy.git
cd envoy
bundle install
bundle exec bin/test

bin/test is the gem's own runner (there's no root bin/rails here, since this is a gem, not an app) — it runs the full Minitest suite against the bundled test/dummy Rails app under test/dummy. Always invoke it with bundle exec so it resolves against this gem's own Gemfile.lock rather than whatever Rails/RubyLLM versions happen to be installed globally.

To run a single file or line, pass it straight through:

bundle exec bin/test test/envoy/guard_test.rb
bundle exec bin/test test/envoy/guard_test.rb:12

12. Embedding a chat

The console (section 6 onward) is a full page. Most hosts instead want a small chat panel bolted onto an existing page — "ask about this record" next to a show view, say — without adopting the console's chrome or its mounted URL. That's what a surface is for: it binds a page to a toolset, a model and a prompt, and a mounted POST endpoint turns that binding into a Turbo Frame you can drop anywhere.

Define a surface

# config/initializers/envoy.rb, alongside the toolset force-load block
Envoy.define_surface :today do
  toolset :assistant                # a registered Envoy.define_toolset key
  model "claude-sonnet-4-5"          # optional; falls back to config.default_model
  system_prompt :today_page          # a Prompt key (see below), or a literal String
  read_only true                     # optional; hides `access :write` tools entirely

  # Runs in the job for every turn — see the callout below.
  context do |actor:, key:|
    date = key.delete_prefix("day:")
    "The user is looking at the training day for #{date}."
  end
end

toolset is required; Envoy.validate! (see below) fails fast if it names an unregistered toolset or an unregistered system_prompt key. model, read_only and system_prompt are all optional.

Define prompts and libraries

Envoy.define_prompt registers reusable system-prompt text, composable with use (like a toolset) so shared framing lives in one place:

Envoy.define_prompt :safety_framing do
  body "Never invent data; say when you don't know."
end

Envoy.define_prompt :today_page do
  use :safety_framing
  body { File.read(Rails.root.join("app/envoy/prompts/today_page.md")) }
end

A block body stays lazy, so editing the file on disk is picked up on the next to_prepare reload rather than requiring a boot.

Envoy.define_library is for reference documents the model should fetch on demand rather than have preloaded into every turn's instructions — it compiles to an ordinary toolset (so it composes via use on a real toolset), exposing one read_<key> tool with a closed enum of document keys:

Envoy.define_library :principles do
  directory Rails.root.join("docs/principles")   # README.md -> description, *.md -> documents
end

Envoy.define_toolset :assistant do
  description "General assistant tools."
  use :principles
end

Wire a page to it

Any host controller can call envoy_surface — it's mixed into ActionController::Base by the engine, no include needed:

class TodayController < ApplicationController
  envoy_surface :today, context_key: -> { "day:#{@date.to_fs(:iso8601)}" }
end

context_key is instance_exec'd in the controller, so instance variables the action set (@date here) are in scope. The helper method envoy_page_surface (available in views) returns { surface:, context_key: }, or nil if the controller never declared a surface — pages opt in, they don't get a bubble by default.

A context block runs in the job. There is no request, no params and no session — only actor and key. Everything it needs must be derivable from the context_key. It is also the authorization boundary: surface and context_key both arrive from the browser, so raise Envoy::Forbidden if this actor may not see this subject.

Render the panel

POST envoy.panel_path (mounted by resource :panel, only: :create) finds or creates the conversation for (actor, surface, context_key) and renders it inside a turbo_frame_tag "envoy_panel". Point a button_to at it with a matching frame already on the page — same "create-or-resume" semantics as opening a tab twice, so re-clicking never duplicates the conversation:

<% if (surface = envoy_page_surface) %>
  <%= turbo_frame_tag "envoy_panel" %>
  <%= button_to "Ask about this page", envoy.panel_path,
        params: { surface: surface[:surface], context_key: surface[:context_key] },
        data: { turbo_frame: "envoy_panel" } %>
<% end %>

An unregistered surface param renders 404; the endpoint is otherwise subject to the same authenticate/actor_resolver config as every other Envoy route, so a signed-out visitor never reaches it.

Style it

The panel (and the console) render the same app/views/envoy/panels/_transcript.html.erb partial, styled entirely by app/assets/stylesheets/envoy/chat.css — no body rules, no fixed widths, the host owns the chrome around it. console.css pulls it in automatically (@import url("chat.css"), so loading the console stylesheet already gives you these rules too), or import it directly from the host's own stylesheet if you don't want the console's chrome at all:

@import url("envoy/chat.css");

Use the url(...) form, not a bare string — Propshaft only fingerprints url(...) references in CSS, not @import strings, and its asset server 404s on an undigested path.

Every color it uses is a CSS custom property, and every declaration falls back to its own original console value when the var is unset, so it renders reasonably with zero host configuration and themes cleanly by setting only the vars below:

Var	Affects	Fallback (console light mode)
`--envoy-surface`	message bubble / input background	`#ffffff`
`--envoy-surface-alt`	assistant bubble background and tool-call panel background	`#eff6ff` / `#f9fafb` (each rule keeps its own original shade until you set this)
`--envoy-ink`	body text	`#111827`
`--envoy-subtle`	role labels, hints, placeholder text	`#6b7280`
`--envoy-line`	borders	`#e5e7eb`
`--envoy-accent`	primary button background	`#111827`
`--envoy-accent-ink`	primary button text	`#ffffff`
`--envoy-warn`	the failed-turn error bubble	`#b91c1c`
`--envoy-radius`	corner rounding (message/tool/input/button each keep their own default until set)	`0.5rem` / `0.375rem`

Setting --envoy-surface-alt or --envoy-radius themes every rule that uses it identically; the console itself never sets them, so it keeps its original, slightly more varied look.

Assets and boot-time validation

Pin the same two Stimulus controllers as the console (section 6) — the composer for Enter-to-submit, the autoscroll controller (now scrolls its own .envoy-chat__log container rather than the window, so it sticks the panel without hijacking the host page's scroll position):

# config/importmap.rb
pin "envoy/controllers/composer_controller", to: "envoy/controllers/composer_controller.js"
pin "envoy/controllers/autoscroll_controller", to: "envoy/controllers/autoscroll_controller.js"

Finally, call Envoy.validate! at the end of the to_prepare block that force-loads your definitions, so a typo'd toolset or prompt key fails on boot — not silently, inside a job, where the user just watches "working…" forever:

Rails.application.config.to_prepare do
  Dir[Rails.root.join("app/envoy/**/*.rb")].each { |f| require_dependency f }
  Envoy.validate!
end

The debug console

Envoy ships a minimal chat UI mounted at whatever path the host chooses (section 4). Posting a message (MessagesController#create) enqueues Envoy::RunJob, which runs Envoy::Runner and broadcasts :delta (streamed reply text) and :tool (a completed tool call) events over Turbo Streams to the conversation's own broadcast channel. The view renders tool calls as collapsible <details> chips showing the call name, an outcome badge (succeeded/denied/failed), arguments, and the raw JSON result.

Real-gem notes worth recording

A few things about the installed RubyLLM (~> 1.16) that weren't obvious from its docs and cost time to pin down:

Legacy acts_as mode (config.use_new_acts_as = false): conversations store a plain string model_id column; there is no separate models table/migration to install. This is deliberate — Envoy doesn't manage a model registry, the host just lists allowed model ids (Envoy.config.available_models).
with_model(id, provider:, assume_exists: true) — note the keyword is assume_exists:, not assume_model_exists:. Passing the wrong keyword silently gets ignored by Ruby's **opts splatting in some call shapes and is easy to miss; verify against the installed gem (ruby_llm/active_record/chat_methods.rb, ruby_llm/chat.rb) if upgrading.
Every chat call passes assume_exists: true so a model_id that isn't in RubyLLM's own registered-models list (e.g. a new Claude model id before RubyLLM ships an update) still works — Envoy's own available_models list is the actual gate, not RubyLLM's.
commonmarker "~> 2.6" — pin this explicitly if your app also depends on it. The 1.1.x line does not build against Ruby 4.0's native extension toolchain.

Manual live smoke (not run in CI)

This validates the real RubyLLM/Anthropic path that FakeLLM stands in for in tests. It requires a real ANTHROPIC_API_KEY and network access, so it's a manual, non-CI gate — run it in a host app with Envoy mounted:

export ANTHROPIC_API_KEY=sk-ant-...
bin/dev

Sign in as a web user.
Visit the mounted path, start a new conversation with a toolset on claude-sonnet-4-5.
Ask it to call one of the write tools it exposes.
Confirm:
- tool-call chips appear with a succeeded badge,
- the assistant's reply streams in (not just appears fully-formed),
- the underlying record was actually created/updated.

If this hasn't been run against the current code, treat RubyLLM-facing changes (model ids, with_model kwargs, provider config) as unverified until someone runs it with a key.