Module: HotCell::Codes

Defined in:: lib/hot_cell/codes.rb

Overview

Every failure carries a code and a permanent flag, and permanent is the only distinction that changes what a caller must do.

Terminal means the same request fails the same way until the input or the code changes — not until the load or the deployment changes. A permanent failure may be recorded against a blob and served from a cache. A non-permanent one must be retried and must never be written down.

The flag travels on the wire, set by the side that knows, rather than being derived by each caller from the code. That is what makes a code added later safe: an old client will not recognise it but will still dispose of it correctly.

Constant Summary collapse

PERMANENT =

{
  "unreadable"  => true,   # the input could not be decoded — the operation said so explicitly
  "invalid"     => true,   # malformed request, or a descriptor that failed its access-mode check
  "failed"      => false,  # the operation raised something nobody classified — see below
  "unsupported" => false,  # this cell does not carry that operation — see below
  "protocol"    => false,  # version mismatch, which heals when the accessory reboots
  "capacity"    => false,  # the queue is full
  "unavailable" => false,  # no connection, or a connection closed with no response
  "timeout"     => false,  # the client's own deadline fired
}.freeze

FSIZE =

killed splits on what the worker hit, because a caller cannot otherwise tell a decompression bomb from a slow afternoon. Size and memory are properties of the input, so the same bytes will do it again on an idle cell. A deadline is as much a property of the load, and treating it as permanent means a busy hour permanently condemns whatever was uploaded during it. crashed is the cell's own fault rather than the input's — a worker that died without answering, which a misconfigured cell does on every request. Recording that against a blob would condemn everything uploaded during a broken deploy, so it is transient. An older client that has never heard of it still disposes of it correctly, because permanent travels on the wire.

A limit this table has never heard of is not permanent, and that default is the point of the table. A cell mints these; adding a kill reason to the supervisor without adding a row here used to make it permanent, silently, and permanent is the answer that cannot be taken back. The names below are constants so that the two places that mint them cannot spell one the table does not carry.

Raising instead — the way an unknown code raises — would be worse than the bug. Supervisor#answer_for builds its Failure as an argument to answer, so the raise would land before that method's rescue, and neither reap nor drain_signals nor run catches it. A typo would take the whole cell down from the one path whose job is reporting a dead worker.

"fsize"

MEMORY =

"memory"

DEADLINE =

"deadline"

CRASHED =

"crashed"

PERMANENT_BY_CAUSE = signal is the unexplained death, and it is not permanent, because a signal says how a process died and never why. The supervisor knows it sent SIGKILL for a deadline and says so. Every other signal arrived from somewhere it cannot see: a cgroup OOM kill chosen on aggregate pressure across concurrent workers, or one worker signalling another — they share a uid, and nothing stops that. Attributing either to the input this worker happened to be holding condemns a file for something it did not do.

{
  FSIZE    => true,
  MEMORY   => true,
  DEADLINE => false,
  CRASHED  => false,
}.freeze

KILLED =

"killed"

Class Method Details

.known?(code) ⇒ `Boolean`

Returns:

(Boolean)



101
102
103

# File 'lib/hot_cell/codes.rb', line 101

def known?(code)
  code.to_s == KILLED || PERMANENT.key?(code.to_s)
end

.permanent?(code, cause: nil) ⇒ `Boolean`