tj-scale

TJ Scale (tj-scale) is a Ruby gem that automatically monitors your background job queues — Delayed Job or Sidekiq — and your web traffic, and sends metrics to a remote monitoring service for auto-scaling purposes. It integrates seamlessly with Rails applications and Heroku deployments.

Features

• 🔍 Automatic Queue Monitoring: Continuously monitors Delayed Job or Sidekiq queues for pending jobs
• 🔀 Pluggable Job Backend: Flip between Delayed Job and Sidekiq with one env var (TJ_SCALE_JOB_BACKEND); auto-detected by default
• 📊 Metrics Reporting: Sends job counts, queue latency, and web queue/response times to a remote monitoring service
• 🚀 Heroku Integration: Automatically runs on the first web or worker dyno
• ⚙️ Configurable: Customizable priority/queue filters and API endpoints
• 🔔 Slack Notifications (control plane): message your team whenever web or worker dynos scale up/down, or when scaling fails
• 🔒 Error Handling: Robust error handling with comprehensive logging
• 🧪 Well Tested: Comprehensive test suite with RSpec

Control plane (Rails + PostgreSQL)

The control plane lives in its own project, tj-scale-dashboard (a sibling directory / repository to this gem): dashboard, per-app scaling rules (web queue time + worker job counts), metrics ingest API compatible with the gem, Heroku formation updates, and Slack notifications for every scale event (Settings → Slack notifications). It uses PostgreSQL — see its README.md and .env.example in ../tj-scale-dashboard/.

Installation

As a Gem

Add this line to your application's Gemfile:

gem "tj-scale", "~> 1.0"

And then execute:

$ bundle install

Or install it yourself as:

$ gem install tj-scale

From Source

1. Clone the repository:

git clone https://github.com/Untechnickle/tj-scale-gem.git
cd tj-scale-gem

1. Install dependencies:

bundle install

1. Run the setup script:

bin/setup

End-to-end guide

Build the gem, publish or reference it, add it to your Rails app, and set configuration (env / Heroku). Follow these steps in order the first time you ship and adopt the gem.

1. Prerequisites

• Ruby 3.0+ and Bundler 2+
• A Rails 6.1+ app with one job backend:
  • delayed_job_active_record (and a DB-backed delayed_jobs table), or
  • sidekiq (and Redis)
• Heroku CLI (if you deploy to Heroku)
• A RubyGems.org account (if you publish publicly)

2. Prepare and build the gem (in this repository)

1. Track files for the gem package — the gemspec uses git ls-files, so the repo must be a git repository and files must be committed:

   cd /path/to/tj-scale
   git init   # if needed
   git add .
   git commit -m "Release tj-scale"

1. Install dependencies and verify quality:

   bundle install
   bundle exec rspec
   bundle exec rake standard

1. Set the version in lib/tj_scale_ruby/version.rb (for example 1.0.0).

2. Build the package:

   gem build tj-scale.gemspec

This produces tj-scale-<version>.gem in the current directory.

3. Publish the gem (choose one)

RubyGems.org (public)

1. Create an API key at rubygems.org/api_keys.
2. Push the file you built (version must match VERSION):

   gem push tj-scale-1.0.1.gem

Private gem host

gem push tj-scale-1.0.1.gem --host https://your-gem-server.com

Configure Bundler in the app if your host needs authentication (see your provider’s docs).

Without publishing — use the repo directly

In the app’s Gemfile:

gem "tj-scale", git: "https://github.com/Untechnickle/tj-scale-gem.git", branch: "main"
# or path for local development:
# gem "tj-scale", path: "../tj-scale"

Then bundle install.

4. Add the gem to your Rails application

1. Open the app’s Gemfile and add (adjust version or source as above):

   gem "tj-scale", "~> 1.0"

1. Run:

   bundle install

1. No extra require is needed in application.rb: the gem registers a Rails Railtie and loads with Rails. Ensure your app already bundles its job backend — delayed_job_active_record or sidekiq (the gem auto-detects which one is loaded, or set TJ_SCALE_JOB_BACKEND explicitly).

2. Deploy or run the server as usual. The monitoring thread starts only on web.1 or worker.1, depending on TJ_SCALE_MONITOR_PROCESS (see below).

5. Configure environment variables

Configuration is entirely via environment variables (and Heroku config vars in production). Copy .env.example to .env for local use if you use dotenv-rails or similar.

Variable	Required	Purpose
TJ_SCALE_API_TOKEN	Yes	Sent as LOGPLEX_DRAIN_TOKEN and Authorization: Bearer … on each ingest POST.
TJ_SCALE_API_URL	Yes	Full URL of the Agent API ingest path: POST /api/v1/metrics or POST /api/v1/sys-logs (same handler). Example local: http://127.0.0.1:5001/api/v1/metrics.
TJ_SCALE_TARGET_APP	No	Heroku app name to scale; defaults to HEROKU_APP_NAME if unset.
TJ_SCALE_TARGET_PROCESS	No	web or worker to scale; defaults to TJ_SCALE_MONITOR_PROCESS.
TJ_SCALE_MONITOR_PROCESS	No	web or worker — only that process’s .1 dyno runs the reporter (default worker).
TJ_SCALE_INTERVAL_SECONDS	No	Seconds between posts (default 20).
TJ_SCALE_JOB_BACKEND	No	Queue backend for the worker reporter: delayed_job, sidekiq, or auto (default). auto uses Sidekiq when the sidekiq gem is loaded, otherwise Delayed Job.
TJ_SCALE_SIDEKIQ_QUEUES	No	Sidekiq backend only: comma-separated queue names to count (empty = all queues).
TJ_MIN_PRIORITY / TJ_MAX_PRIORITY	No	Limit which job priorities are counted (0 = no filter; worker reporter, Delayed Job backend only).
TJ_SCALE_QUEUE_TIME_MS	No	When TJ_SCALE_MONITOR_PROCESS=web, supplies queue_time_ms if no middleware snapshot exists.
TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE	No	Set to 1, true, yes, or on to enable Rack middleware that records queue_time_ms from X-Request-Start (Heroku).

Payload by reporter: web — queue_time_ms (when known), job_count (requests since last tick), optional response_time_ms. worker — job_count, queue_time_s (oldest waiting job age / queue latency, or 0), and job_backend (delayed_job or sidekiq). Both send heroku_app and target_process. Scaling limits (min/max dynos) are configured in the control plane's dashboard settings, not by this gem. Autoscaling runs on the control plane after ingest; this gem does not call Heroku.

Heroku sets DYNO (for example web.1, worker.1) and typically HEROKU_APP_NAME; do not set DYNO manually in production.

Example: my-app-web (web) and my-app-workers (workers)

Use the same token on both apps if one backend handles scaling.

# my-app-web — reporter on web.1, scale web dynos
heroku config:set TJ_SCALE_API_TOKEN=your_token \
  TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics \
  TJ_SCALE_TARGET_APP=my-app-web \
  TJ_SCALE_TARGET_PROCESS=web \
  TJ_SCALE_MONITOR_PROCESS=web \
  -a my-app-web

# my-app-workers — reporter on worker.1, scale worker dynos
heroku config:set TJ_SCALE_API_TOKEN=your_token \
  TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics \
  TJ_SCALE_TARGET_APP=my-app-workers \
  TJ_SCALE_TARGET_PROCESS=worker \
  TJ_SCALE_MONITOR_PROCESS=worker \
  -a my-app-workers

On the web app, enable TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1 (and restart) so queue_time_ms is populated from Heroku’s router; the web dyno does not need a shared Delayed Job DB for that path.

6. Wire your monitoring service (Agent API)

The TJ Scale Agent API accepts the same JSON on either route:

• POST /api/v1/metrics
• POST /api/v1/sys-logs

Set TJ_SCALE_API_URL to the full URL of one of them (for example http://127.0.0.1:5001/api/v1/metrics in development). Authenticate with the ingest token using LOGPLEX_DRAIN_TOKEN or Authorization: Bearer (same secret as TJ_SCALE_API_TOKEN); the gem sends both headers.

The gem POSTs JSON with timestamp, heroku_app, target_process, plus web metrics (queue_time_ms, job_count, optional response_time_ms) or worker metrics (job_count, queue_time_s). Your control plane ingests samples and applies scaling rules (thresholds, cooldowns) when calling the Heroku Platform API.

7. Verify

• At least one web dyno on my-app-web and one worker dyno on my-app-workers so web.1 and worker.1 exist.
• Check Rails logs for TjScaleRuby lines (debug/warn/error).
• Confirm your API receives periodic requests with the expected payload.

Optional: manual calls in Ruby

TjScaleRuby::JobMonitor.pending_job_count
TjScaleRuby::JobMonitor.oldest_pending_queue_seconds
TjScaleRuby.record_web_queue_time_ms!(42) # optional; middleware does this on each request
TjScaleRuby::JobMonitor.send_log

Configuration

Environment Variables

Create a .env file in your project root (or set these in your Heroku config vars):

# Required: API token and ingest URL for your control plane
TJ_SCALE_API_TOKEN=your_api_token_here
TJ_SCALE_API_URL=http://127.0.0.1:5001/api/v1/metrics

# --- Scaling metadata (sent to your monitoring API; defaults shown) ---
# Heroku app name to scale (defaults to HEROKU_APP_NAME when unset).
TJ_SCALE_TARGET_APP=my-heroku-app

# Process type to scale on that app: "web" or "worker" (defaults to TJ_SCALE_MONITOR_PROCESS).
TJ_SCALE_TARGET_PROCESS=web

# Which process type runs the reporter loop: only the first dyno of this type (web.1 or worker.1).
# Use "web" on the web app, "worker" on the worker app.
TJ_SCALE_MONITOR_PROCESS=worker

# Seconds between metric posts (default 20).
TJ_SCALE_INTERVAL_SECONDS=20

# Optional: Minimum priority filter for jobs (0 = no filter)
# Only count jobs with priority >= this value
TJ_MIN_PRIORITY=0

# Optional: Maximum priority filter for jobs (0 = no filter)
# Only count jobs with priority <= this value
TJ_MAX_PRIORITY=0

Heroku sets HEROKU_APP_NAME for the current app. If you omit TJ_SCALE_TARGET_APP, the payload uses that value so each app reports scaling targets for itself.

Heroku Configuration

Set the environment variables on Heroku:

heroku config:set TJ_SCALE_API_TOKEN=your_api_token_here
heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics

The DYNO environment variable is automatically set by Heroku and is used to determine which dyno should run the monitoring loop (only web.1 or worker.1, depending on TJ_SCALE_MONITOR_PROCESS).

Example: web on my-app-web, workers on my-app-workers

Use two Heroku apps (or one app with both process types—same idea). Configure each app’s config vars so the monitoring service receives heroku_app, target_process, and job_count on each POST. Scaling bounds (min/max dynos) are set in the control plane's dashboard.

App my-app-web (web dynos) — run the reporter on the first web dyno and scale web dynos:

heroku config:set TJ_SCALE_API_TOKEN=your_token -a my-app-web
heroku config:set TJ_SCALE_MONITOR_PROCESS=web -a my-app-web
heroku config:set TJ_SCALE_TARGET_APP=my-app-web -a my-app-web
heroku config:set TJ_SCALE_TARGET_PROCESS=web -a my-app-web
heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics -a my-app-web
heroku config:set TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1 -a my-app-web

App my-app-workers (worker dynos) — run the reporter on worker.1 and scale worker dynos:

heroku config:set TJ_SCALE_API_TOKEN=your_token -a my-app-workers
heroku config:set TJ_SCALE_MONITOR_PROCESS=worker -a my-app-workers
heroku config:set TJ_SCALE_TARGET_APP=my-app-workers -a my-app-workers
heroku config:set TJ_SCALE_TARGET_PROCESS=worker -a my-app-workers
heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics -a my-app-workers

Ensure both apps have the web / worker process types in the Procfile as needed, deploy, and scale at least one dyno of each monitored type so web.1 / worker.1 exist.

Monitoring API: Your receiver at TJ_SCALE_API_URL should read the JSON body and enforce the min/max dyno bounds from its own dashboard settings when calling the Heroku Platform API (or your own scaler). Fields sent include job_count, timestamp, and when set, heroku_app and target_process.

Usage

Automatic Monitoring

Once installed and configured, the gem starts monitoring on the first dyno of the configured process type (web.1 or worker.1, see TJ_SCALE_MONITOR_PROCESS). The loop runs every TJ_SCALE_INTERVAL_SECONDS (default 20) and POSTs metrics to the configured API.

Manual Usage

You can also use the gem's methods manually in your code:

# Get the count of pending jobs
count = TjScaleRuby::JobMonitor.pending_job_count
puts "Pending jobs: #{count}"

# Send current job count to the monitoring service
response = TjScaleRuby::JobMonitor.send_log
if response
  puts "Log sent successfully"
end

Priority Filtering

You can filter jobs by priority using environment variables:

# Only count jobs with priority between 5 and 10
ENV['TJ_MIN_PRIORITY'] = '5'
ENV['TJ_MAX_PRIORITY'] = '10'

How It Works

1. Initialization: When your Rails application starts, the Railtie checks if it's running on the first monitored dyno (web.1 or worker.1, per TJ_SCALE_MONITOR_PROCESS)
2. Background Thread: If conditions are met, a background thread is started
3. Monitoring Loop: Every N seconds (TJ_SCALE_INTERVAL_SECONDS), the thread POSTs a payload: on worker reporters it counts waiting jobs on the active backend; on web reporters it sends router queue time (middleware / env) plus request volume and average response time for the interval.
4. Job counting (worker reporters, Delayed Job backend): pending_job_count filters jobs based on:
   • Jobs past their run_at time (with 5 second buffer)
   • Jobs that haven't failed (failed_at IS NULL)
   • Jobs that aren't locked (locked_at IS NULL)
   • Optional priority range filters
5. Job counting (worker reporters, Sidekiq backend): pending_job_count sums enqueued jobs across queues (all queues, or only those in TJ_SCALE_SIDEKIQ_QUEUES), and queue_time_s is the latency of the slowest monitored queue.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

Running Tests

# Run all tests
bundle exec rspec

# Run tests with documentation format
bundle exec rspec --format documentation

# Run a specific test file
bundle exec rspec spec/lib/tj_scale_ruby/models/tj_scale_ruby_spec.rb

Code Quality

The project uses Standard for code formatting:

# Check code style
bundle exec rake standard

# Auto-fix code style issues
bundle exec rake standard:fix

Building and Publishing the Gem

For a full checklist (git, tests, version, integrate in the app, Heroku), see End-to-end guide above.

Building the Gem

1. Update the version in lib/tj_scale_ruby/version.rb
2. Ensure files are tracked by git (the gemspec uses git ls-files).
3. Build the gem:

gem build tj-scale.gemspec

This will create a .gem file in the current directory.

Publishing to RubyGems

1. Make sure you have a RubyGems account
2. Get your API key from https://rubygems.org/api_keys
3. Configure your credentials:

gem push --key your_api_key tj-scale-1.0.1.gem

Or use the credentials file:

# Create ~/.gem/credentials
# Add your API key:
# ---
# :rubygems_api_key: your_api_key_here

gem push tj-scale-1.0.1.gem

Publishing to a Private Gem Server

If you're using a private gem server:

gem push tj-scale-1.0.1.gem --host https://your-gem-server.com

Deployment

Heroku Deployment

1. Add the gem to your Gemfile:

gem "tj-scale", "~> 1.0"

1. Set the required environment variables:

heroku config:set TJ_SCALE_API_TOKEN=your_token

1. Deploy your application:

git push heroku main

The gem will automatically start monitoring on the first worker dyno.

Docker Deployment

1. Add the gem to your Gemfile
2. Set environment variables in your Docker configuration
3. The gem will automatically start when the Rails application initializes

Other Platforms

The gem works with any Rails application. Just ensure:

• Delayed Job or Sidekiq is configured
• Environment variables are set
• The application has worker dynos (for automatic monitoring)

Requirements

• Ruby >= 3.0.0
• Rails >= 6.1
• One job backend (worker reporters only; bring your own gem):
  • delayed_job_active_record >= 4.1 and PostgreSQL, or
  • sidekiq and Redis

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/Untechnickle/tj-scale-gem.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for your changes
5. Ensure all tests pass (bundle exec rspec)
6. Ensure code style is correct (bundle exec rake standard)
7. Commit your changes (git commit -am 'Add some amazing feature')
8. Push to the branch (git push origin feature/amazing-feature)
9. Open a Pull Request

License

The gem is available as open source under the terms of the MIT License.

Changelog

See CHANGELOG.md for a list of changes and version history.

Support

For issues, questions, or contributions, please open an issue on the GitHub repository.

Authors

• Tanuj - Initial work - tanuj@untechnickle.com

Acknowledgments

• Built for Untechnickle
• Works with Delayed Job and Sidekiq for background job processing