Why am I getting the warning: deadlock detection timed out after 1,000 milliseconds, assumed to be an external deadlock, queue expanding?

Error message explanation and resolution

A queue between two containers has a buffer of a specific size (initially 8 tuples). When the buffer needs to grow because it is full, it will double in size. When the buffer reaches 1% of memory (configurable), it will stop growing and push back on the upstream container, blocking progress in the upstream container.

If the upstream container blocks for more than 1 second (configurable) without any space clearing in the buffer, then StreamBase assumes some kind of deadlock is happening, and grow the buffer again to try to break the deadlock.

So, why would that happen?

1) This can happen if the downstream container is much slower than your upstream container, and taking more than 1 second to process a batch of tuples. Growing the queue in this case will eventually exhaust Java heap memory.

2) It can happen if the downstream container is blocking on an external event. Growing the queue in this case will not help, but sometimes waiting will allow the downstream container to make progress.

3) It can happen if there is a loop between the downstream container and the upstream container and downstream needs upstream to process some of its input before downstream can continue, which is a case where growing the queue will help. The loop could involve a queue, or it could be because of shared tables use by a parallel module.

In all cases, this warning may be avoided by preventing queuing by making sure the downstream application implementation processes tuples faster than the upstream implementation.