Responding to sbfeedsim latency warnings
Article ID: KB0076113
Updated On:
| Products | Versions |
|---|---|
| TIBCO Streaming | 7 |
Description
The StreamBase Feed Simulator (sbfeedsim) reports warnings like:
2009-10-21 10:01:04.309-0400 [Thread-169] WARN c.s.s.f.r.FeedSimCmdLineRunner$FeedSimThread$SimThread - WARN: 2,805 tuples in 1 unique time stamps from ts 171 to ts 172 were delivered later then their intended time slot
Issue/Introduction
Responding to sbfeedsim latency warnings
Resolution
This is not an indication of out-of-order input, but an indication that tuples could not be delivered to the server at the time specified by the timestamp column in the supplied input data.
This sort of error will be more often observed when running a feed simulation at multiples greater than 1X (the speed at which the data was initially recorded).
Note that to guarantee in-order delivery across multiple streams, the Feed Simulation configuration file (.sbfs) needs to have the "Include in synchronized timestamp group" check box marked. Multiple streams will not be forced to be synchronized with each other without checking this box.
Run at a faster rate and depending on the system hardware sbfeedsim might not be able to deliver tens or hundreds of thousands of tuples per second. If it gets "behind" - i.e. it's takes longer then a second to send a seconds-worth of data - it will emit the summary warning messages as above.
No data is dropped just because things get behind.
It's possible that the application can push back on sbfeedsim too. Imagine if an app has no concurrent regions and can only process 10K TPS. Try to send this app greater then 10K TPS and eventually the network pushes back, and that will push back on sbfeedsim. Feedsim will still report it is falling behind, but the reason in this case is because of network backup caused by an app that can not process the requested rate.
Running from sbfeedsim from the command line you can use the "-a" flag. This will send data as fast as possible. It sill respects any synchronized groups. It will never emit the "falling behind" warning messages since they don't make sense in that context.
If there is a lot of disk IO going on, for example: sbfeedsim reading CSV input data and the application writing a number of output CSV files (unless you have very fast disk IO and slow CPUs) throughput will increase if you used use both compressed input and output files.
This sort of error will be more often observed when running a feed simulation at multiples greater than 1X (the speed at which the data was initially recorded).
Note that to guarantee in-order delivery across multiple streams, the Feed Simulation configuration file (.sbfs) needs to have the "Include in synchronized timestamp group" check box marked. Multiple streams will not be forced to be synchronized with each other without checking this box.
Run at a faster rate and depending on the system hardware sbfeedsim might not be able to deliver tens or hundreds of thousands of tuples per second. If it gets "behind" - i.e. it's takes longer then a second to send a seconds-worth of data - it will emit the summary warning messages as above.
No data is dropped just because things get behind.
It's possible that the application can push back on sbfeedsim too. Imagine if an app has no concurrent regions and can only process 10K TPS. Try to send this app greater then 10K TPS and eventually the network pushes back, and that will push back on sbfeedsim. Feedsim will still report it is falling behind, but the reason in this case is because of network backup caused by an app that can not process the requested rate.
Running from sbfeedsim from the command line you can use the "-a" flag. This will send data as fast as possible. It sill respects any synchronized groups. It will never emit the "falling behind" warning messages since they don't make sense in that context.
If there is a lot of disk IO going on, for example: sbfeedsim reading CSV input data and the application writing a number of output CSV files (unless you have very fast disk IO and slow CPUs) throughput will increase if you used use both compressed input and output files.
Feedback
Yes
No
Powered by
