Class: Tina4::QueueBackends::LiteBackend

Inherits:
Object
  • Object
show all
Defined in:
lib/tina4/queue_backends/lite_backend.rb

Overview

File-based queue backend — JSON files on disk. Zero dependencies.

Each job is stored as a separate .queue-data file under

//, where is Tina4::Queue.base_path (TINA4_QUEUE_PATH, else data/queue) — the canonical cross-framework layout Python, PHP and Node all use. Dead-lettered jobs (those that exhausted their retries) live under the shared /dead_letter/ directory, tagged with their topic.

Dequeue policy: highest priority first, ties broken oldest-first by the stored created_at (ISO-8601, so lexicographic == chronological). The file name is NOT the ordering key — the stored priority/created_at are.

Constant Summary collapse

JOB_EXTENSION =

The store's layout — extension, topic segment, dead-letter directory — is defined once on Tina4::Queue and shared with the dev-admin panel that reads the same files.

Tina4::Queue::JOB_EXTENSION
LEGACY_DIRNAME =

The pre-alignment Ruby store: Dir.pwd/.queue//.json. Read once per backend so an upgrading app's jobs are rescued rather than stranded (see migrate_legacy_store).

".queue"
LEGACY_JOB_EXTENSION = 
".json"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ LiteBackend

Returns a new instance of LiteBackend.



35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 35

def initialize(options = {})
  # An explicit dir: is the caller's own store — take it as given. Only a
  # store we RESOLVED is one the legacy rescue may touch.
  resolved_store = options[:dir].nil?
  @dir = options[:dir] || Tina4::Queue.base_path
  @dead_letter_dir = File.join(@dir, Tina4::Queue::DEAD_LETTER_DIRNAME)
  # Retry policy. Mirrors the Python lite backend: a failed job is
  # re-enqueued while attempts < max_retries, then dead-lettered.
  @max_retries = options[:max_retries] || 3
  # Seconds to delay a job's next attempt when fail() re-enqueues it.
  # 0 (default) = retry on the very next pop/consume iteration.
  @retry_backoff = options[:retry_backoff] || 0
  # Reservation/visibility timeout (seconds). When a job is popped it is
  # held in <topic>/reserved/ with available_at = now + visibility_timeout.
  # If the consumer dies before complete()/fail() (crash, OOM, k8s
  # eviction) the next pop() reclaims it once the window expires —
  # incrementing attempts and re-enqueuing, or dead-lettering past
  # max_retries. <= 0 disables the reclaim (a reservation then lasts until
  # the consumer acks — the old at-most-once behaviour).
  @visibility_timeout = options[:visibility_timeout] || 300.0
  FileUtils.mkdir_p(@dir)
  FileUtils.mkdir_p(@dead_letter_dir)
  @mutex = Mutex.new
  migrate_legacy_store if resolved_store
end

Instance Attribute Details

#max_retries ⇒ Object

Retry policy — settable so a Queue can propagate its own max_retries / retry_backoff onto a backend instance passed directly (legacy path).



33
34
35
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 33

def max_retries
  @max_retries
end

#retry_backoff ⇒ Object

Retry policy — settable so a Queue can propagate its own max_retries / retry_backoff onto a backend instance passed directly (legacy path).



33
34
35
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 33

def retry_backoff
  @retry_backoff
end

#visibility_timeout ⇒ Object

Retry/visibility policy is settable so a Queue can propagate its own configuration onto a backend instance passed directly (legacy path).



63
64
65
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 63

def visibility_timeout
  @visibility_timeout
end

Instance Method Details

#acknowledge(message) ⇒ Object 



134
135
136
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 134

def acknowledge(message)
  # File already deleted on dequeue — nothing to do.
end

#clear(topic) ⇒ Object

Remove all pending jobs from a topic (and any held reservations). Returns count removed.



332
333
334
335
336
337
338
339
340
341
342
343
344
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 332

def clear(topic)
  count = 0
  [topic_path(topic), reserved_path(topic)].each do |dir|
    next unless Dir.exist?(dir)
    job_files(dir).each do |file|
      File.delete(file)
      count += 1
    rescue Errno::ENOENT
      next
    end
  end
  count
end

#close ⇒ Object

No-op: the file backend holds no connection to release.

It exists so Queue#close can call ONE method on every backend instead of testing for it, and so switching TINA4_QUEUE_BACKEND to "lite" never turns a working close into a NoMethodError. It was the ONLY backend without one, so every close if respond_to?(:close) guard in the tree silently did nothing here. Idempotent by construction - nothing to drop.



211
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 211

def close; end

#complete(message) ⇒ Object 



138
139
140
141
142
143
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 138

def complete(message)
  # The pending file was claimed on dequeue and a reservation record
  # written; complete() is terminal, so drop the reservation. The job is
  # done and gone.
  clear_reservation(message.topic, message.id)
end

#dead_letter(message) ⇒ Object 



191
192
193
194
195
196
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 191

def dead_letter(message)
  path = job_file(@dead_letter_dir, message.id)
  data = message.to_hash
  data[:status] = "dead"
  File.write(path, JSON.generate(data))
end

#dead_letter_count(topic) ⇒ Object

Count dead-letter / failed messages for a topic.



221
222
223
224
225
226
227
228
229
230
231
232
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 221

def dead_letter_count(topic)
  return 0 unless Dir.exist?(@dead_letter_dir)

  count = 0
  job_files(@dead_letter_dir).each do |file|
    data = JSON.parse(File.read(file))
    count += 1 if data["topic"] == topic.to_s
  rescue JSON::ParserError
    next
  end
  count
end

#dead_letters(topic, max_retries: 3) ⇒ Object

Get dead letter jobs for a topic — messages that exceeded max retries. Returns Hashes (raw job data with status "dead").



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 243

def dead_letters(topic, max_retries: 3)
  return [] unless Dir.exist?(@dead_letter_dir)

  files = job_files(@dead_letter_dir).sort_by { |f| File.mtime(f) }
  jobs = []

  files.each do |file|
    data = JSON.parse(File.read(file))
    next unless data["topic"] == topic.to_s
    next if (data["attempts"] || 0) < max_retries
    data["status"] = "dead"
    jobs << data
  rescue JSON::ParserError
    next
  end

  jobs
end

#dequeue(topic) ⇒ Object 



74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 74

def dequeue(topic)
  @mutex.synchronize do
    # First return any reservations whose consumer died mid-flight.
    reclaim_expired(topic)
    candidate = available_candidates(topic).first
    return nil unless candidate

    # Write the reservation BEFORE claiming the pending file, so a crash
    # between claim and reserve can never strand the job. Only the worker
    # that wins the delete owns — and returns — it.
    write_reserved(candidate[:data], topic)
    begin
      File.delete(candidate[:file])
    rescue Errno::ENOENT
      return nil # already consumed by another worker
    end
    job_from_data(candidate[:data], topic)
  end
end

#dequeue_batch(topic, count) ⇒ Object 



94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 94

def dequeue_batch(topic, count)
  @mutex.synchronize do
    reclaim_expired(topic)
    chosen = available_candidates(topic).first(count)
    chosen.filter_map do |c|
      write_reserved(c[:data], topic)
      begin
        File.delete(c[:file])
      rescue Errno::ENOENT
        next
      end
      job_from_data(c[:data], topic)
    end
  end
end

#enqueue(message) ⇒ Object 



65
66
67
68
69
70
71
72
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 65

def enqueue(message)
  @mutex.synchronize do
    topic_dir = topic_path(message.topic)
    FileUtils.mkdir_p(topic_dir)
    path = job_file(topic_dir, message.id)
    File.write(path, message.to_json)
  end
end

#fail(job, error = "") ⇒ Object

Record a failed attempt. Increments attempts and stores the error. While attempts < max_retries the job is re-enqueued to pending (after the configured retry_backoff). Once attempts >= max_retries it is moved to the dead-letter store.



153
154
155
156
157
158
159
160
161
162
163
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 153

def fail(job, error = "")
  # Clear the reservation — the consumer acknowledged (with a failure).
  clear_reservation(job.topic, job.id)
  job.attempts += 1
  job.error = error
  if job.attempts < @max_retries
    requeue_job(job, delay_seconds: @retry_backoff, error: error)
  else
    move_to_dead_letter(job, error)
  end
end

#failed(topic, max_retries: 3) ⇒ Object

Jobs that have failed at least once but are still being retried.

Under the auto-retry lifecycle a failed-but-retryable job lives in the pending queue (not the dead-letter dir), so this scans the topic dir for jobs with 0 < attempts < max_retries. Dead-lettered jobs are returned by dead_letters(). Returns Hashes.



352
353
354
355
356
357
358
359
360
361
362
363
364
365
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 352

def failed(topic, max_retries: 3)
  dir = topic_path(topic)
  return [] unless Dir.exist?(dir)
  jobs = []
  job_files(dir).sort_by { |f| File.mtime(f) }.each do |file|
    data = JSON.parse(File.read(file))
    attempts = data["attempts"] || 0
    next unless attempts > 0 && attempts < max_retries
    jobs << data
  rescue JSON::ParserError
    next
  end
  jobs
end

#find_by_id(topic, id) ⇒ Object

Find a specific pending job by id, claim it (delete the file) and return it. Returns nil when no pending job with that id exists.



112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 112

def find_by_id(topic, id)
  @mutex.synchronize do
    dir = topic_path(topic)
    return nil unless Dir.exist?(dir)

    target = id.to_s
    job_files(dir).each do |f|
      data = JSON.parse(File.read(f))
      next unless data["id"].to_s == target

      # Reserve (so a dead consumer's job is reclaimable) then claim the
      # pending file — mirrors dequeue.
      write_reserved(data, topic)
      File.delete(f)
      return job_from_data(data, topic)
    rescue JSON::ParserError
      next
    end
    nil
  end
end

#purge(topic, status) ⇒ Object

Delete messages by status. For "failed"/"dead"/"dead_letter", removes from the dead_letter directory. For "completed"/"pending", removes matching jobs from the topic (pending) directory. Returns count purged.



265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 265

def purge(topic, status)
  count = 0

  if dead_status?(status)
    return 0 unless Dir.exist?(@dead_letter_dir)

    job_files(@dead_letter_dir).each do |file|
      data = JSON.parse(File.read(file))
      if data["topic"] == topic.to_s
        File.delete(file)
        count += 1
      end
    rescue JSON::ParserError
      next
    end
  else
    dir = topic_path(topic)
    return 0 unless Dir.exist?(dir)

    job_files(dir).each do |file|
      data = JSON.parse(File.read(file))
      if data["status"].to_s == status.to_s
        File.delete(file)
        count += 1
      end
    rescue JSON::ParserError
      next
    end
  end

  count
end

#requeue(message) ⇒ Object 



145
146
147
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 145

def requeue(message)
  enqueue(message)
end

#reserved_count(topic) ⇒ Object

Count currently-reserved (in-flight) jobs for a topic.



214
215
216
217
218
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 214

def reserved_count(topic)
  dir = reserved_path(topic)
  return 0 unless Dir.exist?(dir)
  job_files(dir).length
end

#retry(job, delay_seconds: 0) ⇒ Object

Explicit re-queue requested by the caller (job.retry()). Always re-enqueues regardless of the retry limit — a manual override, distinct from the automatic fail() path. Cleans up BOTH the reservation record and any dead-letter file for this id, so a caller who iterates dead_letters and calls job.retry on each doesn't leave the dead-letter directory carrying duplicates (3.13.105, parity port of PY-12-05). Aligns with retry_job(id) which had always unlinked the dead-letter file — two spellings of the same intent that previously diverged. Increments attempts, clears the error.



174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 174

def retry(job, delay_seconds: 0)
  clear_reservation(job.topic, job.id)
  # Drop any dead-letter file for this id BEFORE the re-queue: if this
  # job came from dead_letters it lives in @dead_letter_dir and would
  # otherwise stay on disk while a fresh pending file appears in
  # topic_path, so the next dead_letters call reports the job again
  # and a consumer processes it twice.
  begin
    File.unlink(job_file(@dead_letter_dir, job.id))
  rescue Errno::ENOENT
    # No dead-letter file for this id (manual .retry on a live job) —
    # nothing to clean up.
  end
  job.attempts += 1
  requeue_job(job, delay_seconds: delay_seconds, error: nil)
end

#retry_failed(topic, max_retries: 3) ⇒ Object

Revive dead-letter jobs (under max_retries) back to pending. Returns the number of jobs re-queued.



300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 300

def retry_failed(topic, max_retries: 3)
  return 0 unless Dir.exist?(@dead_letter_dir)

  dir = topic_path(topic)
  FileUtils.mkdir_p(dir)
  count = 0

  job_files(@dead_letter_dir).each do |file|
    data = JSON.parse(File.read(file))
    next unless data["topic"] == topic.to_s
    next if (data["attempts"] || 0) >= max_retries

    msg = Tina4::Job.new(
      topic: data["topic"],
      payload: data["payload"],
      id: data["id"],
      priority: data["priority"] || 0,
      attempts: data["attempts"] || 0,
      error: data["error"]
    )
    enqueue(msg)
    File.delete(file)
    count += 1
  rescue JSON::ParserError
    next
  end

  count
end

#retry_job(topic, job_id: nil, delay_seconds: 0) ⇒ Object

Revive a specific dead-letter job by id back to the pending queue. When job_id is nil, revives every dead-letter for the topic.

This is a manual override (Queue#retry(job_id)) — it always revives a dead-letter regardless of attempt count, mirroring job.retry. Returns true if any job was re-queued.



373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 373

def retry_job(topic, job_id: nil, delay_seconds: 0)
  return false unless Dir.exist?(@dead_letter_dir)

  available_at = delay_seconds > 0 ? Time.now + delay_seconds : nil
  count = 0

  job_files(@dead_letter_dir).each do |file|
    data = JSON.parse(File.read(file))
    next unless data["topic"] == topic.to_s
    next if job_id && data["id"] != job_id.to_s

    msg = Tina4::Job.new(
      topic: data["topic"],
      payload: data["payload"],
      id: data["id"],
      priority: data["priority"] || 0,
      attempts: (data["attempts"] || 0) + 1,
      available_at: available_at,
      error: nil
    )
    enqueue(msg)
    File.delete(file)
    count += 1
    break if job_id  # found the specific job, stop scanning
  rescue JSON::ParserError
    next
  end

  count > 0
end

#size(topic) ⇒ Object 



198
199
200
201
202
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 198

def size(topic)
  dir = topic_path(topic)
  return 0 unless Dir.exist?(dir)
  job_files(dir).length
end

#topics ⇒ Object 



234
235
236
237
238
239
 
# File 'lib/tina4/queue_backends/lite_backend.rb', line 234

def topics
  return [] unless Dir.exist?(@dir)
  Dir.children(@dir)
     .reject { |d| d == Tina4::Queue::DEAD_LETTER_DIRNAME }
     .select { |d| File.directory?(File.join(@dir, d)) }
end