Flight Control

A Rails-based frontend for managing Solid Queue jobs through Active Job. It lets you inspect job queues and jobs in every status, retry or discard failed jobs, manage recurring tasks and workers, and enqueue any job defined in your application straight from the UI.

Flight Control is a fork of Mission Control — Jobs that focuses exclusively on Solid Queue. Resque support has been removed.

Installation

Add this line to your application's Gemfile:

gem "flight_control"

And then execute:

$ bundle install

Flight Control requires Solid Queue (>= 1.0) configured as your Active Job queue adapter.

Basic configuration

Mount the Flight Control engine where you wish to have it accessible from your app, in your routes.rb file:

Rails.application.routes.draw do
  # ...
  mount FlightControl::Engine, at: "/jobs"

Once you set up Authentication, you should be able to access Flight Control's UI, where you can browse the existing queues, jobs pending in these queues, jobs in different statuses, and discard and retry failed jobs:

API-only apps or apps using vite_rails and other asset pipelines outside Rails

If you want to use this gem with an API-only Rails app or an app that's using vite_ruby/vite_rails, or some other custom asset pipeline different from Sprockets and Propshaft, you need one more thing: configure an asset pipeline so you can serve the JavaScript and CSS included in this gem. We recommend to use Propshaft. You simply need to add this line to your application's Gemfile:

gem "propshaft"

Then execute

$ bundle install

Then, make sure you add a step to your deployment pipeline to precompile assets:

RAILS_ENV=production rails assets:precompile

For example, if you're using the Dockerfile generated by Rails with an API-only app or having skipped the assets pipeline, re-add:

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile

Authentication

Flight Control comes with HTTP basic authentication enabled and closed by default. Credentials are stored in Rails's credentials like this:

flight_control:
  http_basic_auth_user: dev
  http_basic_auth_password: secret

If no credentials are configured, Flight Control won't be accessible. To set these up, you can run the generator provided like this:

bin/rails flight_control:authentication:configure

To set them up for different environments you can use the RAILS_ENV environment variable, like this:

RAILS_ENV=production bin/rails flight_control:authentication:configure

User and password can also be configured by hand like this:

Rails.application.configure do
  FlightControl.http_basic_auth_user = "dev"
  FlightControl.http_basic_auth_password = "secret"
end

Custom authentication

You can provide your own authentication mechanism, for example, if you have a certain type of admin user in your app that can access Flight Control. To make this easier, you can specify a different controller as the base class for Flight Control's controllers. By default, Flight Control's controllers will extend the host app's ApplicationController, but you can change this easily:

Rails.application.configure do
  FlightControl.base_controller_class = "AdminController"
end

Or, in your environment config or application.rb:

config.flight_control.base_controller_class = "AdminController"

If you do this, you can disable the default HTTP Basic Authentication using the following option:

config.flight_control.http_basic_auth_enabled = false

Other configuration settings

Besides base_controller_class, you can also set the following for FlightControl or config.flight_control:

• logger: the logger you want Flight Control to use. Defaults to ActiveSupport::Logger.new(nil) (no logging). Notice that this is different from Active Job's logger or Active Job's backend's configured logger.
• delay_between_bulk_operation_batches: how long to wait between batches when performing bulk operations, such as discard all or retry all jobs—defaults to 0
• internal_query_count_limit: in count queries, the maximum number of records that will be counted if the adapter needs to limit these queries. True counts above this number will be returned as INFINITY. This keeps count queries fast—defaults to 500,000
• scheduled_job_delay_threshold: the time duration before a scheduled job is considered delayed. Defaults to 1.minute (a job is considered delayed if it hasn't transitioned from the scheduled status 1 minute after the scheduled time).
• show_console_help: whether to show the console help. If you don't want the console help message, set this to false—defaults to true.
• backtrace_cleaner: a backtrace cleaner used for optionally filtering backtraces on the Failed Jobs detail page. Defaults to Rails::BacktraceCleaner.new. See the Advanced configuration section for how to configure/override this setting on a per application/server basis.
• filter_arguments: an array of strings representing the job argument keys you want to filter out in the UI. This is useful for hiding sensitive user data. Currently, only root-level hash keys are supported.

This library extends Active Job with a querying interface and the following setting:

• config.active_job.default_page_size: the internal batch size that Active Job will use when sending queries to the underlying adapter and the batch size for the bulk operations defined above—defaults to 1000.

Advanced configuration

Without adding any additional configuration to the one described before, Flight Control will be configured with one single app and a single Solid Queue server.

If you manage multiple apps' job backends from a single, centralized app, you can configure the different apps and/or different servers in an initializer like this:

FlightControl.applications.add("hey", solid_queue: ActiveJob::QueueAdapters::SolidQueueAdapter.new)
FlightControl.applications.add("campfire", solid_queue: ActiveJob::QueueAdapters::SolidQueueAdapter.new)

When you have multiple apps and servers configured, you can choose between them with select and toggle menus:

Basic UI usage

Besides inspecting the queues and the jobs in them, and discarding and retrying failed jobs, you can inspect jobs in different statuses, filter them by queue name and job class name, pause and un-pause queues, inspect workers, know which jobs are being run by what worker, check a specific job or a specific worker, and manage recurring tasks.

Job lists are sorted newest first: the most recently enqueued (or finished) jobs appear at the top. Scheduled jobs are the exception; they're listed in the order they will run.

Running jobs from the UI

The Run job tab lets you enqueue any job class defined in your application. Pick the job class, provide its arguments as a JSON array, and optionally override the queue. A trailing JSON object is passed to the job as keyword arguments; for example, for a job defined as def perform(post_id, notify: false) you'd enter:

[ 123, { "notify": true } ]

Arguments must be expressible as JSON, so jobs that take records or other rich objects as arguments can't be launched this way.

Console helpers, scripting and dealing with big sets of jobs

Besides the UI, Flight Control provides a light console helper to switch between applications and servers. Some potentially destructive actions aren't exposed via the UI (for example, discarding jobs that aren't failed, although this might change in the future), but you can always perform these from the console if you know very well what you're doing.

It's also possible that you need to deal with very big sets of jobs that are unmanageable via the UI or that you wish to write a script to deal with an incident, some cleanup or some data migration. The console helpers and the querying API with which we've extended Active Job come in handy here.

First, when connecting to the Rails console, you'll see this new message:

 bin/rails c


Type 'jobs_help' to see how to connect to the available job servers to manage jobs

Typing jobs_help, you'll get clear instructions about how to switch between applications and servers:

>> jobs_help
You can connect to a job server with
  connect_to "<app_id>:<server_id>"

Available job servers:
  * hey:solid_queue
  * campfire:solid_queue

And then:

>> connect_to hey:solid_queue
Connected to hey:solid_queue

Now you're ready to query and operate over jobs for this server via the API. Some examples of queries:

# All jobs
ActiveJob.jobs

# All failed jobs
ActiveJob.jobs.failed

# All pending jobs in some queue
ActiveJob.jobs.pending.where(queue_name: "some_queue")

# All failed jobs of a given class
ActiveJob.jobs.failed.where(job_class_name: "SomeJob")

# All pending jobs of a given class with limit and offset
ActiveJob.jobs.pending.where(job_class_name: "SomeJob").limit(10).offset(5)

# All scheduled/in-progress/finished jobs of a given class
ActiveJob.jobs.scheduled.where(job_class_name: "SomeJob")
ActiveJob.jobs.in_progress.where(job_class_name: "SomeJob")
ActiveJob.jobs.finished.where(job_class_name: "SomeJob")

# All jobs in progress being run by a given worker
ActiveJob.jobs.in_progress.where(worker_id: 42)

Some examples of bulk operations:

# Retry all the jobs (only possible for failed jobs)
ActiveJob.jobs.failed.retry_all

# Retry all the jobs of a given class (only possible for failed jobs)
ActiveJob.jobs.failed.where(job_class_name: "SomeJob").retry_all

# Discard all failed jobs
ActiveJob.jobs.failed.discard_all

# Discard all pending jobs of a given class
ActiveJob.jobs.pending.where(job_class_name: "SomeJob").discard_all
# Or all pending jobs in a given queue:
ActiveJob.jobs.pending.where(queue_name: "some-queue").discard_all

When performing these bulk operations in the console, a delay of 2 seconds between batches processed will be introduced, set via delay_between_bulk_operation_batches. You can modify it as

FlightControl.delay_between_bulk_operation_batches = 5.seconds

Contributing

Thanks for your interest in contributing! To get the app running locally, just run:

bin/setup

This will load a bunch of jobs as seeds.

We have both unit and functional tests and system tests. If you want to run system tests, you'd need to install ChromeDriver. Then, you'll be able to run the tests as:

bin/rails test test/system

License

The gem is available as open source under the terms of the MIT License. Flight Control is a fork of Mission Control — Jobs, © 37signals, also MIT-licensed.