How to know when the last row has been emitted from a Query read operation
Article ID: KB0075331
Updated On:
| Products | Versions |
|---|---|
| TIBCO Streaming | - |
Description
An operation like a Query read may return none, one, or multiple tuples.
How can we signal to following operations that the last tuple of the set has been received?
How can we signal to following operations that the last tuple of the set has been received?
Issue/Introduction
Use a Split operator to provide an ending tuple after a multiple-tuple operation is complete.
Resolution
A common design pattern is to use a Split operator before the operator that may return varying numbers of output tuples for each input tuple, and then use the second Split output port to generate an additional tuple to signal the end.
Here is an example where the Query output is to go into an Aggregate operator. In this case the Aggregate dimension is closed by looking for isnull(id) to return true, where all tuples emitted from the Query have a non-null 'id' field value. Only the tuple emitted from the second Split output port will have a null value for this field (assuming 'id' is not set by the input tuple). In this way, the Aggregate knows to close the window and perform the aggregate calculation.
Supplying a tuple with a null value field into the Aggregate operator to trigger calculation is safe and will not alter the output of aggregate functions. For example, if you are counting tuples and use count(theField) and ' theField' in a tuple has a null value, that tuple will not contribute to the count. This is described in the product documentation here:
TIBCO Streaming > Authoring Guide > Miscellaneous Topics > Using Nulls
"How Each Operator Handles Nulls"
The Split pattern works because there is no concurrency in this EventFlow. The tuple will emit from Split port 1 and not emit from Split port 2 until all actions resulting from the tuple from Split port 1 are complete. EventFlow processes one tuple at a time according to the concurrency rules described in the product documentation here:
TIBCO Streaming > StreamBase Admin Guide > Tuning StreamBase Applications > StreamBase Execution Order and Concurrency
If between the Split and the Query concurrency is introduced, then as soon as the tuple from Split port 1 leaves this thread and enters another thread, then the tuple from Split port 2 would be released, possibly bypassing some of the output from the Query operator. Concurrency when there are side-by-side flows may remove guarantees of order between tuples that follow different flows.
Here is an example where the Query output is to go into an Aggregate operator. In this case the Aggregate dimension is closed by looking for isnull(id) to return true, where all tuples emitted from the Query have a non-null 'id' field value. Only the tuple emitted from the second Split output port will have a null value for this field (assuming 'id' is not set by the input tuple). In this way, the Aggregate knows to close the window and perform the aggregate calculation.
Supplying a tuple with a null value field into the Aggregate operator to trigger calculation is safe and will not alter the output of aggregate functions. For example, if you are counting tuples and use count(theField) and ' theField' in a tuple has a null value, that tuple will not contribute to the count. This is described in the product documentation here:
TIBCO Streaming > Authoring Guide > Miscellaneous Topics > Using Nulls
"How Each Operator Handles Nulls"
The Split pattern works because there is no concurrency in this EventFlow. The tuple will emit from Split port 1 and not emit from Split port 2 until all actions resulting from the tuple from Split port 1 are complete. EventFlow processes one tuple at a time according to the concurrency rules described in the product documentation here:
TIBCO Streaming > StreamBase Admin Guide > Tuning StreamBase Applications > StreamBase Execution Order and Concurrency
If between the Split and the Query concurrency is introduced, then as soon as the tuple from Split port 1 leaves this thread and enters another thread, then the tuple from Split port 2 would be released, possibly bypassing some of the output from the Query operator. Concurrency when there are side-by-side flows may remove guarantees of order between tuples that follow different flows.
Feedback
Yes
No
Powered by
