Module: HotCell::Codes

Defined in:: lib/hot_cell/codes.rb

Overview

Every failure carries a code and a permanent flag, and permanent is the only distinction that changes what a caller must do.

Terminal means the same request fails the same way until the input or the code changes — not until the load or the deployment changes. A permanent failure may be recorded against a blob and served from a cache. A non-permanent one must be retried and must never be written down.

The flag travels on the wire, set by the side that knows, rather than being derived by each caller from the code. That is what makes a code added later safe: an old client will not recognise it but will still dispose of it correctly.

Constant Summary collapse

PERMANENT =

{
  "unreadable"  => true,   # the input could not be decoded — the operation said so explicitly
  "invalid"     => true,   # malformed request, or a descriptor that failed its access-mode check
  "failed"      => false,  # the operation raised something nobody classified — see below
  "unsupported" => false,  # this cell does not carry that operation — see below
  "protocol"    => false,  # version mismatch, which heals when the accessory reboots
  "capacity"    => false,  # the queue is full
  "unavailable" => false,  # no connection, or a connection closed with no response
  "timeout"     => false,  # the client's own deadline fired
}.freeze

FSIZE =

killed splits on what the worker hit, because a caller cannot otherwise tell a decompression bomb from a slow afternoon. Size and memory are properties of the input, so the same bytes will do it again on an idle cell. A deadline is as much a property of the load, and treating it as permanent means a busy hour permanently condemns whatever was uploaded during it. crashed is the cell's own fault rather than the input's — a worker that died without answering, which a misconfigured cell does on every request. Recording that against a blob would condemn everything uploaded during a broken deploy, so it is transient. An older client that has never heard of it still disposes of it correctly, because permanent travels on the wire.

A limit this table has never heard of is not permanent, and that default is the point of the table. A cell mints these; adding a kill reason to the supervisor without adding a row here used to make it permanent, silently, and permanent is the answer that cannot be taken back. The names below are constants so that the two places that mint them cannot spell one the table does not carry.

Raising instead — the way an unknown code raises — would be worse than the bug. Supervisor#answer_for builds its Failure as an argument to answer, so the raise would land before that method's rescue, and neither reap nor drain_signals nor run catches it. A typo would take the whole cell down from the one path whose job is reporting a dead worker.

"fsize"

MEMORY =

"memory"

DEADLINE =

"deadline"

CRASHED =

"crashed"

PERMANENT_BY_CAUSE = signal is the unexplained death, and it is not permanent, because a signal says how a process died and never why. The supervisor knows it sent SIGKILL for a deadline and says so. Every other signal arrived from somewhere it cannot see: a cgroup OOM kill chosen on aggregate pressure across concurrent workers, or one worker signalling another — they share a uid, and nothing stops that. Attributing either to the input this worker happened to be holding condemns a file for something it did not do. So the supervisor never infers either of them from a signal. They are decided in the worker, which is the process that holds the request: memory when it catches NoMemoryError itself, and fsize when a write of its own returns EFBIG. See Worker#disarm_file_size_signal for why the file-size verdict has to be earned that way rather than read off a wait status. That is a narrower guarantee than "a permanent verdict cannot be forged", and the difference matters. It removes the supervisor as an instrument: a sibling's signal no longer travels through a wait status into someone else's blob. It does nothing about a cell that has been compromised outright and answers a connection itself — from_wire believes a permanent boolean off the wire, and a worker that stole work.sock writes whatever it likes. That is the socket-theft residual docs/DESIGN.md records, and it is not closed here.

{
  FSIZE    => true,
  MEMORY   => true,
  DEADLINE => false,
  CRASHED  => false,
}.freeze

KILLED =

"killed"

Class Method Details

.known?(code) ⇒ `Boolean`

Returns:

(Boolean)



113
114
115

# File 'lib/hot_cell/codes.rb', line 113

def known?(code)
  code.to_s == KILLED || PERMANENT.key?(code.to_s)
end

.permanent?(code, cause: nil) ⇒ `Boolean`