batchwatch — Ruby client

Client for batchwatch.dev: crowdsourced measurement of queue time on LLM batch APIs.

Batch endpoints cost 50% of the synchronous ones, but "completes within 24 hours" is impossible to plan around. batchwatch measures what the queue actually does and answers one question: should I use batch for this job?

No dependencies. The standard library — net/http, json, uri, socket, tmpdir — and nothing else.

Install

Point your Gemfile at the subdirectory:

gem "batchwatch", git: "https://github.com/batchwatch/client",
                  glob: "clients/ruby/*.gemspec"

or vendor the three files under lib/ — they have no third-party imports. A RubyGems release is on the way.

Two lines

require "batchwatch"

bw = Batchwatch::Client.new(token: "bw_...")  # token optional; falls back to $BATCHWATCH_TOKEN

# 1. before you submit — does this belong in the queue?
if bw.should_batch("gpt-5.6-sol", max_wait: "15m")
  job = client.batches.create(...)
else
  answer = client.chat.completions.create(...)
end

# 2. measure it, so the next person gets a better answer
bw.track("gpt-5.6-sol", input_tokens: 9720) do |t|
  result = wait_for(job)
  t.done(output_tokens: result.usage.completion_tokens)
end

With a block, track closes the measurement for you, including on the error path: an exception raised inside your block is recorded as failed and re-raised untouched. Without a block it returns the tracking handle and you call t.done(...) yourself.

Get a key with no email and no card:

curl -X POST https://batchwatch.dev/v1/keys -d '{"label":"my pipeline"}'

It fails open, always

If batchwatch is down, slow, or broken, your job must not notice. That is the first requirement, ahead of collecting any data at all.

  • Every submission runs on a background thread. track() and t.done() do no network I/O on your thread.
  • Two-second timeout by default (BATCHWATCH_TIMEOUT), applied to both the connect and the read, so a server that accepts but never answers cannot hold you.
  • Every batchwatch error is swallowed and passed to the optional logger at debug level. Nothing is printed unless you wire one up.
  • should_batch() is the one synchronous call, because you are waiting for the answer. If it cannot answer, you get your own default back — never a guess. The default is false, "run it synchronously": being wrong that way costs money, being wrong the other way blows a deadline.
  • An exception raised inside your own track block is recorded as failed and re-raised untouched. We swallow our errors, never yours.

test/test_fail_open.rb proves it against a port nothing listens on and against a socket that accepts but never answers.

It never sends your content

No prompts, no completions, no system prompts, no tool calls, no file names. The request body is built from a fixed allowlist — provider, model, mode, endpoint, request count, token counts, timestamps, status — and everything else is dropped by Batchwatch.sanitize on the way out. There is no field to put text in.

test/test_no_content.rb asserts it on the bytes a real HTTP server received, and includes a positive control so the test cannot pass by the client simply sending nothing.

output_tokens defaults to nil, never 0

You know your input tokens. You cannot know your output tokens before the model has answered. So the default is absence (nil), not zero.

Zero is not a harmless placeholder here: output costs five to six times as much as input, so a saving computed on zero output is systematically too low — measured at 3.4x too low on a real model — and nothing in the response would tell you. If you know a ceiling, pass max_tokens instead and the answer comes back labelled as a ceiling.

Passing output_tokens: 0 really does send 0: zero is a measurement, absence is not.

Spooling

When a measurement cannot be delivered, the completed record is appended to a JSONL file and replayed later through POST /v1/calls/complete. Losing measurements exactly when the network is bad means losing them exactly when they are most interesting.

  • Default path: $BATCHWATCH_SPOOL, or batchwatch-spool.jsonl in the system temp directory (Dir.tmpdir). Set BATCHWATCH_SPOOL="" or pass spool: nil to turn it off.
  • The spool is replayed automatically, at most once a minute, right after a successful call — that is the moment we know the network is up. Call bw.flush_spool yourself from a shutdown hook if you want it drained on exit.
  • Spooling requires a token. /v1/calls/complete takes your own timestamps, so it is closed to anonymous callers; without a key a spool file could never be sent, and writing one would just leak disk. bw.spool is nil when no token is set.
  • The file is capped at 5 MB. Beyond that, measurements are dropped rather than filling your disk.
  • A replayed measurement can arrive twice if the original PATCH reached the server but the response did not. That is deliberate: a duplicate is visible in the dataset, a lost measurement is not.
  • The file format is identical across the Python, TypeScript, Go and Ruby clients, so a spool written by one can be flushed by another.
  • Threads are handled by a mutex. Two processes sharing one spool file may send a record twice — give each process its own BATCHWATCH_SPOOL if that matters.

Configuration

Argument Environment Default
token BATCHWATCH_TOKEN none (anonymous)
base_url BATCHWATCH_URL https://batchwatch.dev
timeout BATCHWATCH_TIMEOUT 2.0 seconds
spool BATCHWATCH_SPOOL <tempdir>/batchwatch-spool.jsonl
enabled — true
logger — nil (nothing logged)

enabled: false turns every network call into a no-op, which is what you want in CI.

API

  • should_batch(model, max_wait: nil, default: false, **kw) -> true/false
  • advice(model, max_wait: nil, provider: "openai", input_tokens: nil, output_tokens: nil, max_tokens: nil, risk: "p90") -> Hash | nil
  • wait_now(model, provider: "openai", mode: "batch") -> Hash | nil
  • track(model, provider: "openai", mode: "batch", requests: 1, input_tokens: nil, endpoint: nil) { |t| ... } — with or without a block
    • t.done(output_tokens: nil, status: "completed", ttfb_ms: nil)
    • t.failed(status: "failed")
    • t.started(input_tokens: ...) when the count is only known after submission
  • flush(timeout: 5.0) -> true/false — wait for outstanding submissions before exit
  • flush_spool(timeout: nil) -> Integer — send what is on disk, returns accepted

Tests

rake test
# or a single suite:
ruby -Ilib -Itest test/test_fail_open.rb

26 tests, no network beyond loopback. They start real HTTP servers on ephemeral ports (raw TCPServer, port 0) rather than stubbing Net::HTTP: the thing under test is network behaviour, so the network should be in the test. The allowlist and fail-open tests carry positive controls, so a client that sent nothing at all would fail them rather than pass.

Requirements

Requires Ruby 3.0 or newer. Tested on 3.1.

Licence

MIT