Every queue has two states: the one its dashboard shows, and the one its users experience. The gap between them is usually a missing measurement, not a missing tool.
Depth is not the same as delay
A queue can be shallow and slow, or deep and perfectly healthy. Depth tells you how much work is waiting. Age tells you whether the wait is acceptable. If the only alert is on depth, a slow consumer can hide behind a small backlog while every request quietly misses its deadline.
The useful signal is the age of the oldest item that matters, measured at the point where a user is waiting. For background work, that may be the oldest unprocessed job. For a request path, it is the end-to-end latency budget. Pick the one that maps to a promise.
Instrument the promise
Write down the promise first: “password reset emails are sent within two minutes,” “invoice files are available within thirty seconds.” Then measure the distance from the promise, not from the implementation. A queue-length graph is an implementation detail; age against the promise is an operational fact.
If a dashboard cannot tell you whether a user is waiting too long, it is a monitoring dashboard, not an observability one.
A small practice
Once a week, pick one asynchronous path and trace a single item from enqueue to completion. Note every clock, every retry, and every place where the item can sit without emitting a signal. The gaps you find are usually cheaper to fix than the incident they would have caused.