RabbitMQ Troubleshooting Interview Questions: Unroutable Messages, Prefetch, and Channel Errors

A message was published successfully, yet no consumer received it. The queue has work waiting, but the consumer stops accepting more. An acknowledgement suddenly closes a channel. In a RabbitMQ troubleshooting interview, “add retries” is not a diagnosis for any of these observations.

The useful first step is to identify which boundary failed: routing into a queue, delivery to a consumer with available credit, or acknowledgement on the correct channel. This guide follows three original, locally executed experiments using RabbitMQ 4.3.5, Erlang 28.5.0.6, Pika 1.4.4, and AMQP 0-9-1. They are preparation exercises, not reported employer questions.

For a related implementation drill, start with Implement Event Filtering and Queue Routing. Explain how you would observe an event that matches no destination before you design a retry policy.

Identify the failing boundary before changing anything

Our lab uses an isolated local broker, a direct exchange named lab.direct, and a queue bound with the routing key invoice.created. No alternate exchange is configured. A second queue, lab.work, holds five messages for the prefetch experiment. The consumer uses manual acknowledgements.

We declared durable lab queues because RabbitMQ 4.3 disables transient, non-exclusive classic queues by default. The experiment does not test restart survival: queue durability alone does not make every message persistent or establish a complete recovery guarantee. We removed the lab queues and exchange afterward. Queue documentation

For every observation, record the virtual host, exchange, routing key, queue, connection, and channel. A screenshot of an empty queue is weak evidence if the publisher and consumer are using different virtual hosts. Likewise, a process being alive does not tell you whether its AMQP channel remains usable.

Use this initial diagnostic split:

These are starting hypotheses. Queue counts can change while you look at them, and consumers can drain a correctly routed message quickly. Reproduce a stable state or correlate events before declaring a cause.

Can a publisher confirm arrive for an unroutable message?

Yes. Our first publish used invoice.create, missing the final d in the binding key. The exchange existed, and publisher confirms were enabled. We also set mandatory=True and registered a returned-message callback.

The observed callback sequence was:

basic.return: reply_code=312, reply_text=NO_ROUTE
Basic.Ack: delivery_tag=1

Official behavior: when a mandatory AMQP 0-9-1 publish is unroutable, RabbitMQ returns it. A publisher confirm can still follow; the confirm concerns the broker's handling of the publish, not successful consumer processing. For this unroutable case, the return precedes the acknowledgement. Acknowledgements and confirms

That pair of events answers the apparent contradiction. The application observed a positive confirmation but ignored the separate routing result. In our fixture, the broker had no queue to accept the message under that key. There was nothing for a consumer to retrieve.

Changing only the key to invoice.created produced a publish acknowledgement without a return. We then fetched the expected body, one, from the bound queue. This second check matters: it verifies the intended route and payload instead of treating absence of an error as sufficient evidence.

A minimal topology and publish setup is:

ch.exchange_declare('lab.direct', exchange_type='direct')
ch.queue_declare('lab.route', durable=True)
ch.queue_bind('lab.route', 'lab.direct', 'invoice.created')
# Enable confirms and return handling using your adapter's API.
ch.basic_publish(
    exchange='lab.direct',
    routing_key='invoice.create',  # deliberate mismatch
    body=b'one',
    mandatory=True,
)

Our event recorder used Pika's asynchronous SelectConnection adapter so return and confirm callbacks were visible separately. A blocking adapter can surface an unroutable confirmed publish through an exception instead. Explain the protocol event and the client API separately; do not assume every library exposes the same return value. Pika channel API

Official distinction: publishing to a nonexistent exchange is a different failure from publishing to an existing exchange with no matching route. The former closes the channel. With non-mandatory publishing, unroutable-message handling also depends on the alternate-exchange configuration. Our no-alternate-exchange result should not be generalized to a topology that routes rejected matches elsewhere. Publishers

In an interview, propose a targeted verification: inspect the actual binding and publish a uniquely identifiable test message with return handling enabled in an authorized environment. Do not restart consumers to repair a spelling mismatch in a routing key.

Why does prefetch stop delivery while messages remain Ready?

For the second experiment, we published five messages and started one consumer with:

ch.basic_qos(prefetch_count=2, global_qos=False)
consumer_tag = ch.basic_consume(
    queue='lab.work',
    on_message_callback=record_delivery,
    auto_ack=False,
)

The callback recorded delivery tags without acknowledging them. After two deliveries, the broker reported Ready = 3, Unacked = 2. The consumer had reached its outstanding-delivery limit. The remaining three messages had not vanished; they were waiting in the queue.

We acknowledged one recorded tag on that same channel and allowed the client event loop to process another delivery. The next snapshot was Ready = 2, Unacked = 2. One completed message freed capacity, and one waiting message immediately occupied it.

A flat Unacked count therefore does not prove acknowledgements are failing. In this controlled experiment, it stayed flat precisely because acknowledgement and replacement delivery both worked. The message ledger reconciles the state: five published equals one acknowledged plus two ready plus two unacknowledged.

We obtained the snapshots with:

rabbitmqctl -n prachub_lab@localhost list_queues   name messages_ready messages_unacknowledged --formatter json

In production, add a time dimension: compare acknowledgement rate, delivery rate, processing latency, and queue growth. A single snapshot cannot distinguish a busy healthy consumer from a permanently stuck callback. Ask whether the worker is awaiting a downstream service, failing before acknowledgement, or losing access to its channel.

Official behavior: RabbitMQ's non-global prefetch limit applies separately to each new consumer. It is not automatically one shared budget across every consumer on a channel. A separate global limit can constrain the channel, and a configured count of zero means no prefetch limit rather than zero deliveries. Consumer prefetch

Our numerical result applies to one consumer with manual acknowledgement and a limit of two. If the interview changes the setup to two consumers, ask whether the limit is per consumer or shared before predicting total Unacked. Do not reuse the number two merely because the connection count stayed unchanged.

Increasing prefetch is a tuning experiment, not a repair for a handler that never finishes. A larger outstanding set may increase memory use and the amount of work held by one worker. State the throughput or latency constraint you intend to improve, then measure whether the change improves that outcome without creating unacceptable in-flight work.

Why does a duplicate acknowledgement close the channel?

The third experiment received one message with manual acknowledgement. We acknowledged its delivery tag and then acknowledged that same tag again. The broker closed the channel with:

406 PRECONDITION_FAILED - unknown delivery tag 1

We also received a message on channel A and attempted to acknowledge its tag on channel B, which had no outstanding deliveries. Channel B closed with the same error. Channel A remained open; acknowledging through A succeeded, and the final ready count was zero.

Official behavior: delivery tags are scoped to a channel. A delivery must be acknowledged on the channel where it was received, and an already acknowledged tag cannot be acknowledged again. Acknowledgements and confirms

The controlled wrong-channel test intentionally used a second channel with no deliveries. In a larger application, equal numeric tags on different channels refer to different channel-local histories. Treating the integer as a globally unique message identifier is unsafe even when it appears familiar in a log.

The diagnostic unit is therefore the connection/channel context plus delivery tag, not the tag alone. Log that context when scheduling work and preserve it when completion is reported. Also inspect acknowledgement ownership: is a shared error handler acknowledging something the success path already acknowledged? Does a library automatically acknowledge while application code also sends an acknowledgement?

This is the faulty pattern in isolation:

method, properties, body = channel_a.basic_get(
    'lab.work', auto_ack=False
)
channel_b.basic_ack(method.delivery_tag)  # wrong channel

The repair is to acknowledge once through the receiving channel, after the intended processing outcome. If processing moves to another thread, use the client's supported mechanism to schedule completion on the owning connection rather than calling arbitrary channel methods from that worker. The simple lab used one thread; it does not validate a multithreaded consumer implementation.

Official behavior: broker channel errors are asynchronous. A subsequent operation can be where the client surfaces a closure caused by earlier work. Once closed, that channel cannot be reused; opening another channel does not correct the application defect that caused the original exception. Channel lifecycle and errors

Our blocking test issued a passive queue declaration after each bad acknowledgement to force a broker round trip and observe the closure. The declaration was the observation point, not the cause. This distinction is especially useful when an interview gives you only the final stack trace.

Choose the next check from the evidence

A useful answer preserves what you have already learned:

For a repair review, retain one positive control and one negative control. The corrected route must deliver the expected body, while the deliberately wrong route must still be detectable. The correct-channel acknowledgement must succeed, while a test should continue to expose duplicate completion. Otherwise, a change that merely suppresses exceptions can look like a fix.

These experiments verify routing, bounded outstanding delivery, and channel-local acknowledgement behavior. They do not exercise cluster failover, broker crashes, persistent-message recovery, or end-to-end business idempotency. Keep those as separate contracts rather than adding an untested reliability claim to a small successful lab.

Use these PracHub questions to transfer the reasoning into implementation and design discussions:

Continue with Design High-Throughput Messaging Infrastructure, and justify each capacity change with an observation you would expect to improve.

Sources and Further Reading

• RabbitMQ: Publishers
• RabbitMQ: Consumer prefetch
• RabbitMQ: Consumer acknowledgements and publisher confirms
• RabbitMQ: Channels
• RabbitMQ: Queues
• Pika: Channel API