Handling Async or Out-of-Order Data in Canary SQL Collector and MQTT
Hello,
I am using Canary with a SQL Server as a data source, and I have tags that sometimes come in asynchronously or out of time sequence. This happens mostly because older data arrives late due to service downtime or interruptions. However, I am finding that the Canary SQL Collector is not able to backfill or insert that older delayed data correctly into the historian.
My questions are:
How does Canary handle async or out-of-sequence data ingestion in general, specifically with the SQL Collector?
Will I face the same issue with MQTT data collection in Canary, where late-arriving or out-of-sync timestamps might not be properly backfilled or inserted?
Any advice or best practices around configuring Canary to cope with this kind of data delay or how MQTT compares in this regard would be appreciated.
Thanks in advance!
1 reply
-
Hi ,
Both Collectors are able to insert historical data if it's arriving out of order. Now, it's not ideal if it constantly doing this because the historian is not as proficient in inserting historical data as it is appending it to the end of the archive, but as long as it's not a lot of data. It shouldn't matter that much. The only Collectors that do not allow inserts are OPC DA and UA.
If it's inserting historical data, you should be able to verify this by looking at the data within the Historian files. You should see an <Added> Good quality.
The MQTT Collector has a setting (Reordering Queue Size) which will buffer out-of-order messages. By default, this is set to 0 as it's expected that messages should arrive sequentially. If it receives out-of-order messages, it requests a birth message from the publishing device to start the sequencing over again. If you change this setting (255 is the max), it will buffer that number of messages until it receives the expected message based on the sequencing.
Print this page