Huginn logo

Huginn

Performant, elegant ActiveRecord datatables and tolerant search.
Huginn is the raven of Odin that represents thought and remembrance — the mate of Muninn.

🇺🇸 English · 🇧🇷 Português

Huginn is a lightweight query layer for Rails that turns a raw datatable request into a lean count, a paginated subset and one preload — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL fuzzy-search builder (pg_trgm similarity via the % operator with unaccent and ILIKE fallback) that is tolerant to typos and accents — and that can lean on a GIN index when present.

Highlights

  • Two-phase execution — association filters/orders/range become reflection-secured subqueries, then a lean count and preload only on the paginated subset.
  • Lean counts — the base relation never joins (COUNT(*) over a stripped relation); association matches run as pk subqueries instead.
  • SQL injection safe ordering/filtering — every column reference is resolved through Arel reflection, never string-interpolated.
  • Accent/typo tolerant search — pg_trgm similarity (% operator) OR unaccent+ILIKE, with a configurable fallback chain; uses a gin_trgm_ops GIN index when one exists.
  • Rails conventions — works with ActionController::Parameters, Railtie auto-includes both concerns (toggleable), zero boilerplate.

Development

The root Gemfile keeps only the tooling (rspec, appraisal, pry) — each supported Rails series lives in its own Appraisal. Use these to run the suite:

bundle install
bundle exec appraisal install        # generates gemfiles/*.gemfile + resolves
bundle exec appraisal rspec          # runs the full matrix (Rails 7.1/7.2/8.0)
bundle exec appraisal rails-8.0 rspec   # or a single series
bundle exec rake matrix              # alias for the full matrix

A bare bundle exec rspec needs an active environment: export BUNDLE_GEMFILE=gemfiles/rails_8.0.gemfile.

Supported versions

Component Range
Ruby >= 3.0 (no upper bound — Rails 8 + Ruby 4 supported)
Rails >= 7.1, < 9
Pagy >= 6 (runtime dependency, installed automatically)
PostgreSQL pg_trgm / unaccent / ILIKE search; degrades gracefully without them

The suite is verified against Rails 7.1, 7.2 and 8.0 across supported Rubies via Appraisal. Run the full matrix locally:

bundle exec appraisal install
bundle exec appraisal rspec

The gemfiles/*.gemfile are generated by Appraisal (committed); their .lock files are not — each CI cell resolves for its own Ruby/Rails pair.

Installation

gem "huginn"

Configuration

# config/initializers/huginn.rb
Huginn.configure do |config|
  # :pg_trgm   (recommended) — trigram similarity (%) OR unaccent+ILIKE
  # :full_text — PostgreSQL full text search over lexemes
  #              (to_tsvector @@ plainto_tsquery)
  # :unaccent               — unaccent + ILIKE only
  # :simple                 — plain LIKE
  # Pass an Array to combine strategies with OR, e.g.
  # config.search_strategy = [:pg_trgm, :full_text]
  config.search_strategy = :pg_trgm

  # Unaccent function used around search terms/columns. Defaults to the pg
  # builtin UNACCENT(). Point it at the IMMUTABLE wrapper created by
  # `rails g huginn:trigram_indexes` ("public.f_unaccent") so GIN trigram
  # indexes can actually be used.
  config.unaccent_function = "unaccent"

  # Full text search (strategy :full_text). The dictionary and the IMMUTABLE
  # tsvector wrapper (`rails g huginn:fts_indexes`) pinned to that dictionary;
  # they only apply when the wrapper exists, otherwise search degrades to the
  # unaccent fallback.
  # config.fts_dictionary = "portuguese"
  # config.fts_function   = "public.f_tsvector"

  config.pagy_items = 10         # default page size
  config.pagy_max_items = 500    # hard cap for per_page
end

Threshold: under :pg_trgm the similarity cutoff is the PostgreSQL GUC pg_trgm.similarity_threshold (default 0.3), not a gem-level setting. Tune it with SET pg_trgm.similarity_threshold = 0.4 / SELECT set_limit(0.4) on the database.

Strategy tradeoffs: :pg_trgm tolerates typos and substrings but has no notion of morphology. :full_text matches morphological variants (e.g. "correndo"/"correr" → same stem) but is blind to typos, and shines on longer text columns — for name/keyword lookups it mostly overlaps with trigram. Combining both ([:pg_trgm, :full_text]) unions the two result sets and requires both index sets (rails g huginn:trigram_indexes + rails g huginn:fts_indexes).

Railtie (automatic include)

By default the Railtie includes Huginn::Datatable and Huginn::Searchable into every ActiveRecord::Base model. You do not need include statements unless you opt in selectively:

Huginn.configure { |c| c.auto_include_datatable = false; c.auto_include_searchable = false }

Usage — datatable

class Plano < ApplicationRecord
  # Datatable + Searchable are auto-included via Railtie
end

result = Plano.datatable(
  params,
  allowed_paths: [:grupo, { operadora: [:pessoa] }], # associations filters/orders may use
  includes: [{ operadora: { pessoa: [:endereco, :contatos] } }] # preloaded on the page only
)

result[:total_count] # Integer (lean COUNT DISTINCT pk)
result[:data]        # ActiveRecord::Relation (paged + preloaded)

Supported params:

Key Behavior
page, per_page Pagination (clamped to pagy_max_items)
search Delegates to Huginn::Searchable.search
filters Hash / Array of hashes / pairs -> exact or IN conditions ("col" => "null" → IS NULL)
range_data { "created_at" => ["2024-01-01", "2024-12-31"] } — date or numeric ranges
orders [{ "pessoa.nome" => "desc" }] — plain or association-scoped columns

Scoped ordering / filtering

Any column or association.column reference is validated and mapped to its real reflected table:

Plano.datatable({ orders: [{ "operadora.pessoa.nome" => "asc" }] }, allowed_paths: [{ operadora: :pessoa }])

Association filters/range, ordering and search use reflection-resolved subqueries (see the "allowlist of associations" section below). The base relation stays singular and join-free, so the count is a plain COUNT(*).

Association allowlist (allowed_paths)

To protect the schema and keep the query lean, the datatable does not materialize left_joins to filter/order by associations. Instead:

  • Filters/ranges over association columns become pk IN (SELECT DISTINCT pk …) subqueries — the main relation is never multiplied;
  • Ordering by an association column uses a correlated scalar subquery (ORDER BY (SELECT … ORDER BY col ASC LIMIT 1)), which is deterministic even for has_many (smallest value);
  • Only authorized associations may be referenced. Pass allowed_paths: with the associations the caller may use in the query:
result = Plano.datatable(
  params,
  allowed_paths: [:grupo, { operadora: :pessoa }],  # associations filters/orders may use
  includes:      [{ operadora: { pessoa: [:endereco, :contatos] } }] # preload only the page
)
  • Deny-all by default: without allowed_paths:, no association is authorized for filtering/ordering — only columns of the table itself.
  • allowed_paths: accepts the same shapes Rails knows (:symbol, "string", nested Hash, mixed Array). Table names ("companies") are recognized as the matching association (:company).
  • includes: stays independent of allowed_paths: — it only controls the preload of the paginated page.

Field aliases & schema protection

Public APIs should not expose the database schema. Declare a mapping of public names to real columns/tables with huginn_attributes:

class User < ApplicationRecord
  # Public API name -> real column/table (association name or table name)
  huginn_attributes(
    name:         "users.name",
    email:        "users.email",
    created_at:   "users.created_at",
    company_name: "companies.name"   # association column, resolved via subquery
  )
end
  • Callers then filter/order/range only by the aliases: { filters: { company_name: "Acme Corp" } }, { orders: [{ company_name: "asc" }] }.
  • Strict by default: fields outside the mapping are silently rejected (they never reach SQL and are never answered). The schema stays hidden from API consumers.
  • Scoped aliases (companies.name) resolve through the association only if it is authorized in allowed_paths: (the same allowlist applies to aliases).
  • Without huginn_attributes, the model falls back to plain/reflected columns (name, company.name).
  • Without allowed_paths:, association-scoped aliases are denied; only plain columns are usable.
  • huginn_attributes({ ... }, strict: false) keeps alias translation but also accepts raw columns.

Usage — search

# Default: searches every :string / :text column of the model.
Person.search("kayky")            # typo/accent tolerant, case-insensitive

# Override which columns (including through associations) are searched:
class Person < ApplicationRecord
  searchable_columns :name, company: [:name, :cnpj]
end

Person.search("globex")                            # matches company.name via a pk subquery
Person.search("kayky", distinct: false)            # disable the implicit DISTINCT

Huginn::Datatable reuses Huginn::Searchable.search automatically when the model responds to search.

Trigram indexes (performance)

Under :pg_trgm, each searchable column produces an indexable predicate:

(UNACCENT(col) % UNACCENT('term')) OR (UNACCENT(col) ILIKE UNACCENT('%term%'))

Both branches are supported by a GIN trigram index, so the planner can run a BitmapOr on it. Because the pg builtin unaccent() is STABLE (not IMMUTABLE) on PostgreSQL 13+, it cannot be used directly in an index expression — you need an IMMUTABLE wrapper plus an expression index on it:

CREATE OR REPLACE FUNCTION public.f_unaccent(text)
RETURNS text AS $$
  SELECT public.unaccent('public.unaccent', $1);
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;

CREATE INDEX index_people_name_trgm ON people USING gin (public.f_unaccent(name) gin_trgm_ops);

Then point the gem at the wrapper:

Huginn.configure { |c| c.unaccent_function = "public.f_unaccent" }

Or scaffold the migration for every Searchable model (wrapper + indexes included):

rails g huginn:trigram_indexes                     # all Searchable models
rails g huginn:trigram_indexes Person Product      # specific models

The index is optional — without it the search still returns correct results (via a sequential scan), and the same indexes also accelerate the :unaccent (ILIKE) and :simple (LIKE) strategies.

Notes:

  • Requires the pg_trgm and unaccent extensions.
  • Only helpful for search terms of 3 or more characters — trigrams need that many to match. Shorter terms always scan.
  • To see it working, run a Person.search("term").explain with the wrapper configured and a gin_trgm_ops index present.

Query efficiency

phase 1  build the relation          subqueries (pk IN … / ORDER BY (SELECT …)) + search + filters + order   (no data in memory)
phase 2  count                       SELECT COUNT(*) ... (base relation is join-free)
          paginate                   offset / limit
          preload                    SELECT ... WHERE id IN (subset)        (2nd lightweight query)

For a Plano datatable with deep includes:, this is exactly 2 extra queries on the small page instead of one enormous JOIN.

Architecture

lib/huginn.rb                       entry, Huginn.configure, Huginn.instrument
lib/huginn/configuration.rb         search_strategy, unaccent_function, pagy_*
lib/huginn/railtie.rb               auto-includes concerns into ActiveRecord
lib/huginn/datatable.rb             Huginn::Datatable (aggregator)
lib/huginn/datatable/datatable.rb   the datatable Concern
lib/huginn/datatable/validator.rb   column/association validation + Arel resolution
lib/huginn/datatable/association_path.rb   resolution of association chains + correlated subqueries
lib/huginn/datatable/allowed_paths.rb      `allowed_paths:` allowlist expansion/authorization
lib/huginn/datatable/filter_normalizer.rb  functional param normalization
lib/huginn/datatable/paginator.rb    lean count, pagination, isolated preload
lib/huginn/searchable.rb            Huginn::Searchable (aggregator)
lib/huginn/searchable/searchable.rb the search Concern + DSL
lib/huginn/searchable/query.rb      tolerant search builder (subqueries + OR)
lib/huginn/searchable/fuzzy.rb      pg_trgm / unaccent / simple predicates
lib/generators/...                  `huginn:trigram_indexes` (GIN index migrations)

Instrumentation

Huginn.instrument wraps ActiveSupport::Notifications events under the huginn namespace (e.g. datatable.call.huginn). Subscribe with ActiveSupport::Notifications.subscribe(/\.huginn/).

License

MIT