Store and Forward Buffer
Hi,
I upgraded a dev server to the new Store and Forward (S+F) system and want to understand how the buffer behaves.
It does its job of storing and forwarding as it says on the tin, and works perfectly if I stop and resume the service on the historian.
I am particularly interested in the S+F buffer, as the manual states the following:
- Buffer Backup (days) - stores buffer files in a backup folder (C:\ProgramData\Canary\StoreAndForward\Backup) for the specified number of days after data has successfully been sent to destination and buffer files are deleted. By default, buffer files are deleted immediately after the data is sent. We recommend turning this feature on for customers with highly sensitive data to allow a window of recovery if a disaster event occurs and data is lost or corrupted at the destination.
To test, I set the buffer on the collector (and on the historian) to 7 days and let it run. I then stopped the S+F on the historian, took the dataset file offline and deleted it. This was to simulate a corruption or accidental deletion.
Once I resumed the S+F service on the collector, I expected back dated data to be resent to the historian. This did not happen.
Is there a special procedure to have the data resent? I would have preferred if it sent the data again automagically.
3 replies
-
Hi deon. korb ,
There's a lot to unpack here and some of what you are seeing is expected based upon the current functionality. I'll try to elaborate as best as I can and point out the deficiencies.
1. The backup directory (C:\ProgramData\Canary\StoreAndForward\Backup) doesn't get created until the buffer files have been processed. Therein, a backup is only created when either:
- The buffer files have rolled over due to size. This is controlled by the "Buffer File Rollover Size".
- Or a session stops logging and all of its data has been written. Once the SaF service reconnects to the destination and the session is processed, the buffered file would be deleted and moved to the Backup archive. This may be the case when there are multiple sessions waiting to be sent as a result of starting/stopping the collector or SaF service. In that case, all of the sessions would have their own buffered files which would get backed up once they are processed with the exception of the active session that is receiving all of the newest data. We do not backup the active session because it is being written to.
2. In theory, you should be able to stop the SaF service and move the sessions from the Backup folder into the Buffer folder to be processed again, but the developer thinks that the buffered files may not be in the proper state to do that. The data is intact, but there is a "header" that moves through the file as it being processed. When the file is finished being processed, the header is left at the end of the file where it is then moved to the Backup directory. In order to properly reprocess it, the file would need to be reset with the header at the beginning again. It doesn't do this currently. The developer is looking into changing that functionality so that the file gets reset before storing it in the Backup folder. For now, if you wanted to resend the data, you would need to send us the backup files and we could reset them for you.
Hope this helps
- The buffer files have rolled over due to size. This is controlled by the "Buffer File Rollover Size".
-
Thanks for the clarification. I now understand the limitation, and in the event of a catastrophic failure, at least we can send the buffer backup to you to assist with the recovery.
It would be great if the S+F service can automatically determine that the buffer and backup content is not in the historian and reprocess it. Feature request?
Print this page