Class: Wurk::Telemetry::ServerMiddleware

Inherits:
Object
  • Object
show all
Includes:
Middleware::ServerMiddleware
Defined in:
lib/wurk/telemetry/server_middleware.rb

Overview

Consumer half of the pair: a <klass> process span around the rest of the server chain and perform, tied back to the producer span whose W3C trace context ClientMiddleware wrote onto the job hash.

Registered only by install! — gem present and host opted in — one position inside Middleware::InterruptHandler, so the span covers every other middleware (limiter, batch, metrics, status) and the job body, and its duration is the job's real wall clock rather than perform's alone. Inside rather than outside the interrupt handler because that handler's rescue is what turns a cooperative stop into a re-push plus JobRetry::Skip: from outside, the span would also be wrapping a Redis RPUSH that is not part of running the job.

A job with no traceparent — pushed by stock Sidekiq, by a Wurk process with tracing off, or straight into Redis by hand — still gets a span. It is simply a root, with nothing to point at.

Constant Summary collapse

PARENT_WINDOW_SECONDS =

How stale a producer context may be and still be used as the span's parent rather than as a link.

Under this, enqueue and execute are one causal step and the pair belongs in one trace: the parent edge is what OTel's messaging conventions ask for, and it is what makes "show me this request" include the work it queued. Past it, the job is a scheduled one, a cron tick, or a retry backoff — the producer's trace was exported, sampled and closed minutes to hours ago, and hanging a new span off it yields a trace the backend will split, drop, or hold open for a day. So past the window the relationship is recorded as a span link: same information, no unbounded trace.

60s is chosen to sit above normal enqueue→execute latency on a healthy queue (sub-second) and below the decision window of the tail samplers that would otherwise have already ruled on the producer's trace.

60
SECONDS_SHAPED_BELOW =

created_at is epoch millis today — JobUtil#now_in_millis, and upstream's own since Sidekiq 8.x — but Sidekiq <= 7.x stamped a Float of epoch seconds, and those payloads outlive the upgrade in the retry, scheduled and dead sets. Read shape-agnostically on the same threshold JobRecord#parse_time and Stats use: no epoch-second stamp reaches ten digits until 2286, and no epoch-milli one has been under it since April 1970. Assuming millis would date every seconds-shaped job to 1970 and demote a perfectly fresh one from parent to link.

10_000_000_000

Instance Attribute Summary

Attributes included from Middleware::ServerMiddleware

#config

Instance Method Summary collapse

Methods included from Middleware::ServerMiddleware

#logger, #redis, #redis_pool

Instance Method Details

#call(_worker, job, queue) ⇒ Object

Retries carry the original traceparent: JobRetry#schedule_retry re-ZADDs the same hash, so the client chain — and with it injection — never runs again. That is deliberate and it is the answer to "one trace per logical job vs. a fresh root per attempt": every attempt is reachable from the enqueue that caused it. Attempt 1 usually lands inside the window and is a child; later attempts, pushed out by exponential backoff, land outside it and are roots carrying a link back. Neither is ever an unrelated root.



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/wurk/telemetry/server_middleware.rb', line 64

def call(_worker, job, queue)
  span = start(job, queue)
  # `yield` rather than taking the chain's block as `&block`: capturing it
  # allocates a Proc for every job in the process, and this frame runs on
  # every one of them.
  ::OpenTelemetry::Trace.with_span(span) { yield } # rubocop:disable Style/ExplicitBlockArgument
rescue Wurk::JobRetry::Handled, Wurk::Job::Interrupted
  # The canonical not-an-error list (see Batch::ServerMiddleware): a
  # cooperative interruption, and every `Handled` — which is where
  # `JobRetry::Skip` and, under it, `Limiter::Rescheduled` live. None of
  # them is a job that went wrong; each is one that was put back and will
  # run again. Marking the span an error would paint a rate-limited or
  # gracefully-paused deploy as an outage.
  raise
rescue Exception => e # rubocop:disable Lint/RescueException
  # Everything else that unwinds through here, `Wurk::Shutdown` included:
  # a job killed by a hard shutdown did not complete, and that is exactly
  # the signal an operator tuning `shutdown_timeout` is looking for.
  span&.record_exception(e)
  span&.status = ::OpenTelemetry::Trace::Status.error("Unhandled exception of type: #{e.class}")
  raise
ensure
  span&.finish
end