Class: SpecGuard::RSpecFormatter

Inherits:
RSpec::Core::Formatters::BaseFormatter
  • Object
show all
Defined in:
lib/specguard/rspec/formatter.rb

Overview

A NOTE ON CONSTANT RESOLUTION — read this before editing

SpecGuard::RSpec is this gem's own namespace. Inside a module SpecGuard body an unqualified RSpec::Core::Formatters therefore resolves to SpecGuard::RSpec::Core::Formatters and dies with NameError: uninitialized constant SpecGuard::RSpec::Core — on the very first line of the class definition, before anything else can go wrong.

Every reference to the real RSpec below is consequently top-level-qualified with ::. It is also why this class is SpecGuard::RSpecFormatter, a sibling of SpecGuard::RSpec rather than a member of it: the sibling name keeps the two namespaces from shadowing each other for readers as well as for the interpreter.

What it captures, and for whom

Every example, annotated or not. That is the whole point: SpecGuard's premise is that an unannotated test is an anonymous coordinate, and you cannot report on a gap you never recorded. Filtering to the annotated minority here would make the very first run of a new adopter look empty.

id              example.id — this example's identity within the run
spec_file_path  the spec file that *ran* the example, relative to the root
file_path       example.metadata[:file_path], relative to the project root
line_number     example.metadata[:line_number]
name            example.full_description — the composed describe/context/it
duration        example.execution_result.run_time, seconds, per example
outcome         example.execution_result.status — passed / failed / pending
status          "annotated" / "unannotated" — see {AnnotationLookup}
intent          the parsed annotation when annotated, null when not

status is not decoration: Ingest::Payload validates it against a two-value enum for every spec and collects the failures globally, so a payload missing the key is not a payload with a gap — it is a 400 with one error per example.

Why id and spec_file_path exist alongside the coordinate

(file_path, line_number) is the coordinate of the code, not of the example, and two entirely ordinary suite shapes put several examples on one coordinate:

* a table-driven loop — `CASES.each { |c| it("...#{c}") { ... } }` writes
the `it` once, so all N examples report the same line;
* a shared example group — every including file reports the coordinate of
`spec/support/shared.rb`, and the file that actually ran the example
appears nowhere at all.

Measured on a probe suite of a 3-case loop plus a 2-example shared group included by two files: 7 examples, 3 distinct coordinates. A key that folds three examples onto one row cannot carry a per-example duration, cannot follow one test's outcome across runs, and hands the duplicate-cluster surface rows that look identical because the key collapsed them — a manufactured duplicate in the product's headline answer.

RSpec already ships the fix. Example#id is "#{metadata[:rerun_file_path]}[#{metadata[:scoped_id]}]" (rspec-core 3.13.6, example.rb:117 → metadata.rb:105), it is rooted at the including file rather than the defining one, and it is also RSpec's own re-run argument — so a row that turns up slow or flaky is directly actionable: rspec './spec/table_spec.rb[1:2]'.

id is unique within a run, not stable across refactors. scoped_id is positional, so reordering examples changes it — exactly as line_number changes when a line is inserted above it. It is the run-local primary key and nothing more; matching one test across runs remains name plus file.

spec_file_path is metadata[:rerun_file_path], which RSpec defaults to the defining file (metadata.rb:160) and overrides with the including file for a shared example. It is therefore equal to file_path for an ordinary example and differs only for a shared one — which is what makes duration-by-file aggregate to the file that ran the test rather than to a spec/support/ helper.

file_path and line_number keep their existing meaning — the definition site — and are deliberately not repurposed: the annotation lookup reads @intent: from exactly that line, so they are the coordinate the intent came from.

Where the run goes

api_key set → one POST <endpoint>/api/v1/ingest. api_key unset → appended to output_path, one JSON object per line.

The credential is the switch on purpose: local development is then the default and needs no opt-out. See #deliver for what happens when the POST does not land.

A malformed or schema-invalid annotation is recorded as unannotated with a null intent rather than shipped or shouted about; AnnotationLookup documents why, and the linter is the half of this gem that tells the author.

The never-block-CI contract, and why it lives here

SpecGuard's non-negotiable is that telemetry never fails a build. It is tempting to read that as a property of the transport — a rescue around the POST — but RSpec does not sandbox formatters, so it is a property of the capture layer too. A raise in any of the three hooks below escapes RSpec::Core::Runner.run entirely; RSpec's own exit code is then never returned at all, and the process exits 1. Probed against rspec-core 3.13.6:

hook=example_finished  process_exit=1  rspec's exit status: never reached
hook=stop              process_exit=1  rspec's exit status: never reached
hook=close             process_exit=1  rspec's exit status: never reached

In the close case the suite had already printed 2 examples, 0 failures and the process still exited 1 — a green suite turned red by telemetry. A nil line number, an unwritable log/, a full disk: each of those is somebody's broken build, for tests that all passed.

#seed was added after that probe and is guarded for the same reason, with the stakes slightly higher: it is dispatched from Reporter#start, before any example has run, so a raise there costs the whole suite rather than its exit code.

So every hook body runs inside #never_fail_the_run, which swallows StandardError and ScriptError (an autoload blowing up is not a StandardError, and a bare rescue would miss it), warns once, and returns. Interrupt, SignalException and SystemExit are deliberately not caught: Ctrl-C must stay Ctrl-C.

spec/specguard/rspec/formatter_spec.rb fails if any of those rescues is removed.

Constant Summary collapse

WARNING_PREFIX =

Prefixed so it is obvious in a CI log which tool is talking, and worded so a reader knows immediately that it is not their tests that broke.

"SpecGuard: test telemetry failed and was skipped"
DELIVERY_WARNING_PREFIX =

The other half of that: the run was captured fine, the delivery did not land. Kept distinct from WARNING_PREFIX because the reader's situation is different — nothing was lost, the payload is sitting in a file, and the thing to fix is a key or a URL rather than a broken sink.

"SpecGuard: could not deliver test telemetry"
DRY_RUN_WARNING_PREFIX =

The third shape, and the only one that reports a deliberate refusal rather than something going wrong. Nothing failed and nothing is recoverable from a file, because there was nothing worth keeping — see #dry_run?. It still gets said out loud: a user who wired the formatter up and then saw neither a POST nor a line would otherwise have to go looking for a bug that is not there.

"SpecGuard: skipped test telemetry for a dry run"
STATUS_ANNOTATED =

Ingest::Payload::STATUSES, restated. The platform validates every spec against this pair (payload.rb:17), so they are the contract and not a local naming choice.

"annotated"
STATUS_UNANNOTATED = 
"unannotated"
REPORTS_THE_RUN =

The published formatter-protocol methods that mean "this formatter tells a human what the suite did", as opposed to RSpec's own auxiliaries, which speak only about deprecations and timings. #reports_the_run? is where this set is justified, and where the respond_to? approximation's limits are written down.

%i[
  example_started example_passed example_failed example_pending dump_summary
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(output = nil, error_stream: $stderr, annotations: SpecGuard::RSpec::AnnotationLookup.new) ⇒ RSpecFormatter

Returns a new instance of RSpecFormatter.

Parameters:

  • output (IO) (defaults to: nil) —

    the stream RSpec hands every formatter. This class's product is a file, so nothing in the capture path writes here — but the stream is not unused, and a reader who is here because stdout went wrong is in the right place. #message writes to it, via #relay_message below: having taken over the FallbackMessageFormatter RSpec would otherwise have appointed, this formatter owes that formatter's duty, and printing a message no other registered formatter will print is precisely its job. That is the only write.

    The nil default has no caller. RSpec always supplies a stream, and the two direct constructions in the repo — both in formatter_spec.rb — pass one explicitly, so nothing exercises it. It survives because narrowing a public constructor's signature is not this slice's business, not because anything depends on it. It cannot reach output.puts: #relay_message returns early unless this object is in RSpec.configuration.formatters, which only RSpec puts it in, and doing so means RSpec built it. A nil that somehow got there would raise inside #never_fail_the_run and downgrade to a warning rather than taking the suite with it.

  • error_stream (IO) (defaults to: $stderr) —

    where the one-shot warning goes. Injectable so a spec can read it back without reassigning $stderr globally.

  • annotations (AnnotationLookup) (defaults to: SpecGuard::RSpec::AnnotationLookup.new) —

    resolves each example's @intent:. One per run: it is where the per-file scan is memoized, so sharing it across the run is what makes the cost O(files) rather than O(examples).

    Fully qualified for the same reason every ::RSpec above is — and it is the same trap seen from the other side. This class is a sibling of SpecGuard::RSpec, not a member, so a bare AnnotationLookup here is looked up as SpecGuard::RSpecFormatter::AnnotationLookup and then SpecGuard::AnnotationLookup, neither of which exists — a NameError raised from a formatter's constructor, which RSpec reports as "No examples found".



255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
 
# File 'lib/specguard/rspec/formatter.rb', line 255

def initialize(output = nil, error_stream: $stderr,
               annotations: SpecGuard::RSpec::AnnotationLookup.new)
  super(output)
  @error_stream = error_stream
  @annotations = annotations
  @specs = []
  @warned = false
  # Stamped here rather than from a `start` hook on purpose. `:start` is not
  # in this formatter's own registered set — it arrives only because
  # BaseFormatter registered for it — and the wall clock a CI operator cares
  # about includes loading the spec files, which happens before `start`
  # fires. `duration_seconds` is therefore a superset of RSpec's own
  # "Finished in N seconds", not a contradiction of it.
  @started_at = monotonic_now
  @duration_seconds = nil
end

Instance Method Details

#close(notification) ⇒ Object

Last hook of the run: flush what we captured.

super restores the output stream's sync setting, which BaseFormatter changed on our behalf at start (a notification it registers for, and so one this subclass receives too). Overriding close without calling it would leave stdout unbuffered for whatever runs next in the process.



456
457
458
459
460
461
462
 
# File 'lib/specguard/rspec/formatter.rb', line 456

def close(notification)
  never_fail_the_run do
    @duration_seconds ||= elapsed_since_start
    deliver(payload)
  end
  never_fail_the_run { super(notification) }
end

#example_finished(notification) ⇒ Object

One example finished — passed, failed or pending. Called for every example in the run, in the order they complete.



439
440
441
 
# File 'lib/specguard/rspec/formatter.rb', line 439

def example_finished(notification)
  never_fail_the_run { @specs << capture(notification.example) }
end

#message(notification) ⇒ Object

RSpec::Core::Formatters::FallbackMessageFormatter, moved inside this class — the other half of #seed's repair, and the one that stops it printing everything twice.

Why this method has to exist at all

setup_default ends with (formatters.rb:133-135):

unless existing_formatter_implements?(:message)
add FallbackMessageFormatter, output_stream
end

Somebody must print reporter.message output — a --seed banner, "No examples found.", and every non-example exception, since notify_non_example_exception routes through it (reporter.rb:163-170). progress and documentation do it via BaseTextFormatter#message; when no registered formatter listens for :message, RSpec appoints a fallback.

Before this method, this class did not listen for :message, so the fallback was always appointed — and then #seed added progress, which listens for it too. Two listeners, one stream, every message twice. On NON_EXAMPLE_ERROR_SUITE the error block printed once without SpecGuard and twice with it, which is the claim that matters and the one the spec asserts; the streams were ~335 and ~546 bytes.

The fallback cannot be withdrawn once appointed: Reporter#register_listener has no inverse, and Configuration#formatters hands back a dup (configuration.rb:1024-1026), so deleting from it changes nothing. The repair is therefore to stop it being appointed — this class listens for :message, existing_formatter_implements?(:message) is true, and the duty lands here instead.

When it actually prints

Only when it is the last resort, which is exactly the fallback's own contract: if any other registered formatter handles :message, that one prints and this stays quiet. So the wirings behave identically to a run without this gem — progress prints it under the README's Ruby form once #seed has restored it, documentation prints it for a developer who asked for documentation, json swallows it into its hash exactly as it does on its own, and --format html (which has no message) gets it from here, just as it would have got it from the fallback.

The check is made per message rather than cached, because a message can arrive before :seed: Runner#setup calls world.announce_filters, which reports "No examples found." through reporter.message before Reporter#report is ever entered. In that window this formatter really is the only listener, and it prints — which is what the fallback would have done.

respond_to? is the public-API approximation of the question rspec-core answers with registered_listeners(:message); #reports_the_run? documents how the two can differ. For :message specifically they agree across every formatter rspec-core ships, because the only classes that define message are the ones that register it.



433
434
435
 
# File 'lib/specguard/rspec/formatter.rb', line 433

def message(notification)
  never_fail_the_run { relay_message(notification) }
end

#payload ⇒ Hash

The run, as it will be written or POSTed. Public so a caller — or a spec — can inspect what was captured without going anywhere near the filesystem or the network.

Key names match the platform's ingest contract (Ingest::Payload: commit_sha / branch / ci_run_id / duration_seconds / specs), which is what made adding transport a transport change rather than a reshaping of everything above it. Transport sends this Hash verbatim.

ci_run_id is the field that keeps a sharded suite honest: every shard of one CI run emits the same one, and the platform folds them onto a single TestRun instead of recording one row per shard with a quarter of the denominator in it. nil here — a laptop run, an unrecognised provider — means "this run is its own run", which is the pre-existing behaviour and is left exactly alone.

shard_id says which slice of that run this is, and it is what makes the fold idempotent. A CI run id survives a re-run by design (GitHub's GITHUB_RUN_ID is documented as unchanged across attempts), so without a per-shard key the platform could only add a re-delivered slice, never recognise it, and "re-run failed jobs" would report a suite bigger than the suite. With it, a retried shard replaces its own previous numbers. nil is allowed and still counts — see Configuration::SHARD_ID_KEYS.

The settings are run_id / shard_id and the wire fields are ci_run_id / shard_id, deliberately. Configuration names are this gem's own (SPECGUARD_RUN_ID, next to output_path and endpoint); every key in this Hash is the platform's, spelled exactly as TestRun spells it. That rule is what lets a reader check the envelope against the schema without a translation table — and a run identity that two sides spell differently is one more way to split a run, which is the whole defect this field closes.

Returns:

  • (Hash) 


499
500
501
502
503
504
505
506
507
508
509
510
 
# File 'lib/specguard/rspec/formatter.rb', line 499

def payload
  configuration = SpecGuard::RSpec.configuration

  {
    "commit_sha" => configuration.commit_sha,
    "branch" => configuration.branch,
    "ci_run_id" => configuration.run_id,
    "shard_id" => configuration.shard_id,
    "duration_seconds" => @duration_seconds,
    "specs" => @specs
  }
end

#seed(notification) ⇒ Object

The first notification of the run — and the one hook here that exists for the human half of the output rather than the telemetry half.

The defect this repairs

RSpec installs its own default formatter only if the user registered no formatter at all (rspec-core 3.13.6, lib/rspec/core/formatters.rb:127):

add default_formatter, output_stream if @formatters.empty?

Registering this formatter makes that list non-empty, so progress is never added — and since this formatter's product is a file, the run prints nothing. Measured on formatter_run_spec.rb's MIXED_SUITE: ~900 bytes of stdout without SpecGuard, 0 with it. No dots, no failure message, no diff, no file:line, no re-run command, and still exit 1. The telemetry was written perfectly (1 line, all 3 specs); the developer was left blind.

(Two rules for the byte counts in this class's comments, both learned the hard way. Name the suite — they come from different ones and are not comparable across methods; this paragraph's and #reports_the_run?'s are MIXED_SUITE, #message's are NON_EXAMPLE_ERROR_SUITE, both defined in formatter_run_spec.rb. And prefer a delta or a count to a total: a total carries the run's wall-clock digits, so it is not reproducible even on the same suite — MIXED_SUITE's control measured 955 and 956 bytes on consecutive invocations, and NON_EXAMPLE_ERROR_SUITE's 333 through 335. Hence the ~ above, and hence the figures that carry the argument elsewhere are deltas — 0, a 27-byte hole, one error block versus two.)

It hit exactly the wrong person. The .rspec wiring escaped it only because the README spelled it with a second --format progress line, and a developer who explicitly chose --format documentation was unaffected — so the casualty was the reader who followed the README's first wiring block and customised nothing. (That --format progress line is gone from the README now: with this repair in place neither documented form needs it, which is what finally makes the two forms equivalent.)

Why here, and not in the user's spec_helper

The obvious repair — config.add_formatter(:progress) if config.formatters.empty? inside RSpec.configure — is wrong, and silently so. configuration_options.rb:21-25 applies --require (i.e. spec_helper.rb) before --format, so at that moment the list is [] no matter what the user asked for. Probed directly: with .rspec set to --format documentation, config.formatters still returns [] inside the helper. That repair gives the documentation user documentation and progress dots.

:seed is the first notification Reporter#start sends (reporter.rb:92, ahead of :start on :93), and Reporter#notify runs ensure_listeners_ready — hence setup_default — before dispatching anything (reporter.rb:207-208). So by the time this runs, RSpec has finished deciding, and the decision can be read rather than predicted. Repairing here rather than in start is what keeps the output byte-identical: a formatter added during :seed still receives :start, and the only notification it misses from Reporter#start onwards is this one, which is forwarded to it by hand below. (Not "the only notification it misses" flatly — a :message can arrive before Reporter#report is ever entered, which is a real case and is #message's to handle, not this method's.)

The forwarding is not decoration, and it is worth knowing what goes if it goes: :seed is what prints Randomized with seed N, so without it a randomly-ordered run loses its head banner and keeps only the one RSpec re-sends at the end (reporter.rb:189). That is the line a developer needs to reproduce a flaky failure. Measured on MIXED_SUITE under --order random: banner twice both with and without this gem, and once if the forwarding loop is deleted (a 27-byte hole, the banner and its blank line). Under RSpec's defined ordering nothing prints a banner either way, which is why the parity example that pins this had to ask for random ordering explicitly.

Arriving late is not the only way the restored formatter can diverge from one RSpec installed itself, and the second way cost a review round. By the time it is added, setup_default has already decided that nobody handles :message and installed a FallbackMessageFormatter to cover for it — so the newcomer, which does handle :message, became the second listener on that notification and every message printed twice. Measured on NON_EXAMPLE_ERROR_SUITE: one error block without SpecGuard, two with it (~335 bytes against ~546). #message is the fix, and it works by making setup_default's premise true rather than by undoing its conclusion — there is no public way to withdraw a registered listener.

What counts as "the human formatter is missing"

Not "the list is empty" — by now setup_default has appended RSpec's own DeprecationFormatter (and a ProfileFormatter under --profile). The question is whether any other registered formatter will give a human an account of the run, so that is what is asked: does it respond to any of example_started, example_passed, example_failed, example_pending or dump_summary? See #reports_the_run? for why that set, and for what the respond_to? approximation can and cannot see.

Nothing here touches anything rspec-core marks @private: formatters, add_formatter and default_formatter are documented public API, and every method name in that set is part of the published formatter protocol (formatters/protocol.rb).

default_formatter rather than a hard-coded :progress, for the same reason: a user who set config.default_formatter = 'doc' asked for that, and it is the value rspec-core's own line would have used.



374
375
376
 
# File 'lib/specguard/rspec/formatter.rb', line 374

def seed(notification)
  never_fail_the_run { restore_suppressed_default_formatter(notification) }
end

#stop(_notification) ⇒ Object

The suite is over but the process is still alive. Sealing the duration here rather than in close keeps the number from absorbing the time the other formatters spend dumping their summaries.



446
447
448
 
# File 'lib/specguard/rspec/formatter.rb', line 446

def stop(_notification)
  never_fail_the_run { @duration_seconds = elapsed_since_start }
end