AppQuery - raw SQL 🥦, cooked :stew:

A Rubygem :gem: that makes working with raw SQL queries in Rails projects more convenient.
Specifically it provides:

...a dedicated folder for queries
e.g. app/queries/reports/weekly.sql is instantiated via AppQuery["reports/weekly"].
...Rails/rspec generators
$ rails generate query reports/weekly create app/queries/reports/weekly.sql invoke rspec create spec/queries/reports/weekly_query_spec.rb
...helpers to rewrite a query for introspection during development and testing
See what a CTE yields: query.select_all(select: "SELECT * FROM some_cte").
Query the end result: query.select_one(select: "SELECT COUNT(*) FROM _ WHERE ...").
Append/prepend CTEs: ruby query.prepend_cte(<<~CTE) articles(id, title) AS ( VALUES(1, 'Some title'), (2, 'Another article')) CTE

...rspec-helpers

RSpec.describe "AppQuery reports/weekly", type: :query do
describe "CTE some_cte" do
  # see what this CTE yields
  expect(described_query.select_all(select: "select * from some_cte")).to \
    include(a_hash_including("id" => 1))

  # shorter: the query and CTE are derived from the describe-descriptions so this suffices:
  expect(select_all).to include ...

[!IMPORTANT]
Status: alpha. API might change. See the CHANGELOG for breaking changes when upgrading.

Installation

Install the gem and add to the application's Gemfile by executing:

bundle add appquery

Usage

[!NOTE] The following (trivial) examples are not meant to convince you to ditch your ORM, but just to show how this gem handles raw SQL queries.

Create

[!NOTE] The included example Rails app contains all data and queries described below.

Create a query:

rails g query recent_articles

Have some SQL (for SQLite, in this example):

-- app/queries/recent_articles.sql
WITH settings(default_min_published_on) as (
  values(datetime('now', '-6 months'))
),

recent_articles(article_id, article_title, article_published_on, article_url) AS (
  SELECT id, title, published_on, url
  FROM articles
  RIGHT JOIN settings
  WHERE published_on > COALESCE(?1, settings.default_min_published_on)
),

tags_by_article(article_id, tags) AS (
  SELECT articles_tags.article_id,
    json_group_array(tags.name) AS tags
  FROM articles_tags
  JOIN tags ON articles_tags.tag_id = tags.id
  GROUP BY articles_tags.article_id
)

SELECT recent_articles.*,
       group_concat(json_each.value, ',' ORDER BY value ASC) tags_str
FROM recent_articles
JOIN tags_by_article USING(article_id),
  json_each(tags)
WHERE EXISTS (
  SELECT 1
  FROM json_each(tags)
  WHERE json_each.value LIKE ?2 OR ?2 IS NULL
)
GROUP BY recent_articles.article_id
ORDER BY recent_articles.article_published_on

The result would look like this:

[{"article_id"=>292,
 "article_title"=>"Rails Versions 7.0.8.2, and 7.1.3.3 have been released!",
 "article_published_on"=>"2024-05-17",
 "article_url"=>"https://rubyonrails.org/2024/5/17/Rails-Versions-7-0-8-2-and-7-1-3-3-have-been-released",
 "tags_str"=>"release:7x,release:revision"},
...
]

Even for this fairly trivial query, there's already quite some things 'encoded' that we might want to verify or capture in tests:

only certain columns
only published articles
only articles with tags
only articles published after some date
- either provided or using the default
articles are sorted in a certain order
tags appear in a certain order and are formatted a certain way

Using the SQL-rewriting capabilities shown below, this library allows you to express these assertions in tests or verify them during development.

Verify query results

[!NOTE] There's AppQuery#select_all, AppQuery#select_one and AppQuery#select_value to execute a query. select_(all|one) are tiny wrappers around the equivalent methods from ActiveRecord::Base.connection.
Instead of positional arguments, these methods accept keywords select, binds and cast. See below for examples.

Given the query above, you can get the result like so:

AppQuery[:recent_articles].select_all.entries
# =>
[{"article_id"=>292,
 "article_title"=>"Rails Versions 7.0.8.2, and 7.1.3.3 have been released!",
 "article_published_on"=>"2024-05-17",
 "article_url"=>"https://rubyonrails.org/2024/5/17/Rails-Versions-7-0-8-2-and-7-1-3-3-have-been-released",
 "tags_str"=>"release:7x,release:revision"},
...
]

# we can provide a different cut off date via binds^1:
AppQuery[:recent_articles].select_all(binds: [1.month.ago]).entries

1) note that SQLite can deal with unbound parameters, i.e. when no binds are provided it assumes null for $1 and $2 (which our query can deal with).
  For Postgres you would always need to provide 2 values, e.g. `binds: [nil, nil]`.

We can also dig deeper by query-ing the result, i.e. the CTE _:

AppQuery[:recent_articles].select_one(select: "select count(*) as cnt from _")
# => {"cnt" => 13}

# For these kind of aggregate queries, we're only interested in the value:
AppQuery[:recent_articles].select_value(select: "select count(*) from _")
# => 13

Use AppQuery#with_select to get a new AppQuery-instance with the rewritten SQL:

puts AppQuery[:recent_articles].with_select("select * from _")

Verify CTE results

You can select from a CTE similarly:

AppQuery[:recent_articles].select_all(select: "SELECT * FROM tags_by_article")
# => [{"article_id"=>1, "tags"=>"[\"release:pre\",\"release:patch\",\"release:1x\"]"},
      ...]

# NOTE how the tags are json strings. Casting allows us to turn these into proper arrays^1:
types = {"tags" => ActiveRecord::Type::Json.new}
AppQuery[:recent_articles].select_all(select: "SELECT * FROM tags_by_article", cast: types)

1) PostgreSQL, unlike SQLite, has json and array types. Just casting suffices:
AppQuery("select json_build_object('a', 1, 'b', true)").select_one(cast: true)
# => {"json_build_object"=>{"a"=>1, "b"=>true}}

Using the methods (prepend|append|replace)_cte, we can rewrite the query beyond just the select:

AppQuery[:recent_articles].replace_cte(<<~SQL).select_all.entries
settings(default_min_published_on) as (
  values(datetime('now', '-12 months'))
)
SQL

You could even mock existing tables (using PostgreSQL):

# using Ruby data:
sample_articles = [{id: 1, title: "Some title", published_on: 3.months.ago},
                   {id: 2, title: "Another title", published_on: 1.months.ago}]
# show the provided cutoff date works
AppQuery[:recent_articles].prepend_cte(<<-CTE).select_all(binds: [6.weeks.ago, nil, JSON[sample_articles]).entries
  articles AS (
    SELECT * from json_to_recordset($3) AS x(id int, title text, published_on timestamp)
  )
CTE

Use AppQuery#with_select to get a new AppQuery-instance with the rewritten sql:

puts AppQuery[:recent_articles].with_select("select * from some_cte")

Spec

When generating a query reports/weekly, a spec-file like below is generated:

# spec/queries/reports/weekly_query_spec.rb
require "rails_helper"

RSpec.describe "AppQuery reports/weekly", type: :query, default_binds: [] do
  describe "CTE articles" do
    specify do
      expect(described_query.select_all(select: "select * from :cte")).to \
        include(a_hash_including("article_id" => 1))

      # short version: query, cte and select are all implied from descriptions
      expect(select_all).to include(a_hash_including("article_id" => 1))
    end
  end
end

There's some sugar:

described_query
...just like described_class in regular class specs.
It's an instance of AppQuery based on the last word of the top-description (i.e. "reports/weekly" from "AppQuery reports/weekly").
:cte placeholder
When doing select_all, you can rewrite the SELECT of the query by passing select. There's no need to use the full name of the CTE as the spec-description contains the name (i.e. "articles" in "CTE articles").
default_binds
The binds-value used when not explicitly provided.
E.g. given a query with a where-clause WHERE published_at > COALESCE($1::timestamp, NOW() - '3 month'::interval), when setting defaults_binds: [nil] then select_all works like select_all(binds: [nil]).

💎 API Doc 💎

generic

AppQuery(sql) ⇒ AppQuery::Q

### Examples ```ruby AppQuery("some sql") ```

module AppQuery

AppQuery[query_name] ⇒ AppQuery::Q

### Examples ```ruby AppQuery[:recent_articles] AppQuery["export/articles"] ```

AppQuery.configure {|Configuration| ... } ⇒ void

Configure AppQuery. ### Examples ```ruby AppQuery.configure do |cfg| cfg.query_path = "db/queries" # default: "app/queries" end ```

AppQuery.configuration ⇒ AppQuery::Configuration

Get configuration ### Examples ```ruby AppQuery.configure do |cfg| cfg.query_path = "db/queries" # default: "app/queries" end AppQuery.configuration ```

class AppQuery::Q

Instantiate via AppQuery(sql) or AppQuery[:query_file].

AppQuery::Q#cte_names ⇒ [Array< String >]

Returns names of CTEs in query. ### Examples ```ruby AppQuery("select * from articles").cte_names # => [] AppQuery("with foo as(select 1) select * from foo").cte_names # => ["foo"] ```

AppQuery::Q#recursive? ⇒ Boolean

Returns whether or not the WITH-clause is recursive or not. ### Examples ```ruby AppQuery("select * from articles").recursive? # => false AppQuery("with recursive foo as(select 1) select * from foo") # => true ```

AppQuery::Q#select ⇒ String

Returns select-part of the query. When using CTEs, this will be ``. ### Examples ```ruby AppQuery("select * from articles") # => "select * from articles" AppQuery("with foo as(select 1) select * from foo") # => "select * from foo" ```

query execution

AppQuery::Q#select_all(select: nil, binds: [], cast: false) ⇒ AppQuery::Result

`select` replaces the existing select. The existing select is wrapped in a CTE named `_`. `binds` array with values for any (positional) placeholder in the query. `cast` boolean or `Hash` indicating whether or not (and how) to cast. E.g. `=> ActiveRecord::Type::Date.new`. ### Examples ```ruby # SQLite aq = AppQuery(<<~SQL) with data(id, title) as ( values('1', 'Some title'), ('2', 'Another title') ) select * from data where id=?1 or ?1 is null SQL # selecting from the select aq.select_all(select: "select * from _ where id > 1").entries #=> [...] # selecting from a CTE aq.select_all(select: "select id from data").entries # casting aq.select_all(select: "select id from data", cast: => ActiveRecord::Type::Integer.new) # binds aq.select_all(binds: ['2']) ```

AppQuery::Q#select_one(select: nil, binds: [], cast: false) ⇒ AppQuery::Result

First result from `AppQuery::Q#select_all`. See examples from `AppQuery::Q#select_all`.

AppQuery::Q#select_value(select: nil, binds: [], cast: false) ⇒ AppQuery::Result

First value from `AppQuery::Q#select_one`. Typically for selects like `select count(*) ...`, `select min(article_published_on) ...`. See examples from `AppQuery::Q#select_all`.

query rewriting

AppQuery::Q#with_select(sql) ⇒ AppQuery::Q

Returns new instance with provided select. The existing select is available via CTE `_`. ### Examples ```ruby puts AppQuery("select 1").with_select("select 2") WITH _ as ( select 1 ) select 2 ```

AppQuery::Q#prepend_cte(sql) ⇒ AppQuery::Q

Returns new instance with provided CTE. ### Examples ```ruby query.prepend_cte("foo as (values(1, 'Some article'))").cte_names # => ["foo", "existing_cte"] ```

AppQuery::Q#append_cte(sql) ⇒ AppQuery::Q

Returns new instance with provided CTE. ### Examples ```ruby query.append_cte("foo as (values(1, 'Some article'))").cte_names # => ["existing_cte", "foo"] ```

AppQuery::Q#replace_cte(sql) ⇒ AppQuery::Q

Returns new instance with replaced CTE. Raises `ArgumentError` when CTE does not already exist. ### Examples ```ruby query.replace_cte("recent_articles as (select values(1, 'Some article'))") ```

Compatibility

💾 tested with SQLite and PostgreSQL
🚆 tested with Rails v6.1, v7 and v8.0
💎 requires Ruby >v3.1
Goal is to support maintained Ruby versions.

Development

After checking out the repo, run bin/setup to install dependencies. Make sure to check it exits with status code 0.

Using direnv for env-vars recommended.

Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/eval/appquery.

License

The gem is available as open source under the terms of the MIT License.