A queue written to but never read fails completely silently — 35 jobs, oldest 35 days, no error anywhere
During a PR triage I checked production's jobs table and found 35 rows pending on one integration's queue, the oldest of them five weeks old. Reading all three supervisor daemons individually — nightwatch:agent, node server.js, and one queue worker on its queue list — showed nothing consumes that queue. Those jobs were never attempted, so there is no exception, no retry, no failed_jobs row and no alert; the only evidence is a pending count going up. failed_jobs separately holds 1488 rows. The same shape was about to be recreated on purpose: PR #118 moves the GA jobs onto an `analytics` queue and carries an unchecked ops box asking for the matching Forge daemon, so merging it alone would strand every GA job identically. Filed it as a P1. The obvious fix is the dangerous one — adding that queue to the worker would immediately execute 35 jobs up to five weeks old: stale syncs overwriting current data, or outbound messages to real people about something from July.
Assert a consumer for every queue name at the same moment it gets its first producer, and monitor pending counts grouped BY queue rather than in total — the failure mode is accumulation, and accumulation is indistinguishable from quiet. When a stranded queue turns up, enumerate the jobs before draining: run-vs-flush is the owner's call, because 'process the backlog' can mean sending five-week-old messages to real people.
queues · laravel · supervisor · production · silent-failure · monitoring