OmniAgent

OmniAgent is a Rails engine gem for building application-native AI agents with tools. It provides a small DSL to define agents, model/provider settings, prompt templates, tool schemas, and generation lifecycle callbacks.

What It Includes

• OmniAgent::Agent runtime with provider abstraction and tool-calling loop
• OmniAgent::Tool DSL with JSON-schema-style input definitions
• Prompt composition from ERB files in app/agents/<agent_name>/
• Agent callbacks (before_generation, after_generation)
• Agent and tool tags to support filtering strategies
• OpenAI provider integration out of the box
• Rake tasks and Rails generators for scaffolding

Installation

Add these lines to your application's Gemfile:

gem "omni_agent"

Add the provider you're using to the Gemfile as well:

gem "openai"

Then run:

bundle install

Quick Start

1. Install base directories:

bundle exec rails generate omni_agent:install

1. Generate an agent scaffold:

bundle exec rails generate omni_agent:agent ResearchAgent --model gpt-4.1-mini --with-tools WeatherLookup Summarize

1. Add your API key in .env:

OPENAI_ACCESS_TOKEN=your_api_key_here

1. Implement your agent prompt and optional tools under:

app/agents/
    research_agent.rb
    research_agent/
        prompt.md.erb
        tools/

Agent Example

class ResearchAgent < OmniAgent::Agent
    use_model "gpt-4o-mini"

    before_generation :set_current_user

    def set_current_user
        @user = "Test User"
    end
end

Tool Example

module ResearchAgent::Tools
    class GetWeather < OmniAgent::Tool
        description "Get current weather for a city"
        tags :weather
        metadata category: :utility

        input do
            string :city, description: "City name"
        end

        def execute(city:)
            "Sunny in #{city}"
        end
    end
end

Evals

OmniAgent::Eval lets you test agent quality: deterministic assertions (tool calls, output matching) and pluggable LLM-as-judge scoring.

class ResearchAgentEval < OmniAgent::Eval
    agent ResearchAgent

    eval_case "answers weather question" do
        input "What's the weather in Paris?"
        expect_tool_call :get_weather, with: { city: "Paris" }
        expect_output to_include: "Paris"
    end

    eval_case "is polite" do
        input "Tell me a joke"
        judge "Is the response friendly and on-topic?", threshold: 0.7
    end

    eval_case "summarizes via the :summarize run alias" do
        run_alias :summarize
        input "Some long article text...", with: { tone: "casual" }
        expect_output to_include: "summary"
    end
end

• input text, with: {}: with: is forwarded as the agent's context:, bound to matching instance variables during the run (e.g. with: { tone: "casual" } sets @tone).
• run_alias: Targets a method defined via run_aliases (or any zero-arg run entrypoint) instead of plain #run — useful when that alias renders a different prompt file (<method_name>.md.erb).

Judge provider resolution order: explicit provider: kwarg on judge → OmniAgent.configuration.eval_judge_provider/eval_judge_model → the agent's own provider (zero-config default).

Caching

Eval runs are cached by default, keyed on (agent class, run_alias, input, context). Re-running the same case (e.g. iterating on assertions) replays the cached output instead of calling the provider again, saving tokens. Configure or disable it:

OmniAgent.configure do |config|
    config.eval_cache_enabled = true # default
    config.eval_cache_path = "tmp/omni_agent_eval_cache.json" # default
end

Bypass the cache for a run (clears it before running, no manual file deletion needed):

bundle exec omni_agent eval evals/research_agent_eval.rb --fresh

Sample output:

[PASS] mentions lorem
[FAIL] mentions something the mock never says
  - output "Lorem ipsum dolor sit amet, consectetur adipiscing elit." does not include "this never appears"

1/2 cases passed

Each case prints [PASS]/[FAIL] plus its name; failing cases list every unmet assertion's message. Exits non-zero if any case failed.

For many input/expected-output pairs, load a YAML/JSON dataset instead of writing a case per row:

golden_set "evals/golden/research_agent.yml" do |row|
    expect_output to_include: row[:expected_output]
end

Scaffold an eval and run it:

bundle exec rails generate omni_agent:eval ResearchAgent

# run from your Rails app root, like running rspec
bundle exec omni_agent eval
bundle exec omni_agent eval evals/research_agent_eval.rb
bundle exec omni_agent eval evals/research_agent_eval.rb --fresh

There's also an equivalent rake omni_agent:eval task (rake "omni_agent:eval[pattern,fresh]") if you'd rather not use the binstub.

Calls real LLM providers (cost, non-determinism) — deliberately not part of bundle exec rspec or CI.

Configuration

Global defaults can be configured through OmniAgent.configure:

OmniAgent.configure do |config|
    config.default_provider = :openai
    config.default_model = "gpt-4o-mini"
end

Running Tests

bundle exec rspec

Contributing

Issues and pull requests are welcome.

License

The gem is available as open source under the terms of the MIT License.