Knowledge Base / Version 24 / System Admin Duties / Data Collection

Store and Forward Tile (version 24)

updated 3 mths ago

Streams - number of data streams coming through the Store and Forward (SaF) service
Buffer - amount of updates, or TVQs (Timestamp, Value, Quality), buffering in the SaF service. A buffer occurs whenever the Input/Sec exceeds the Output/Sec.
Input/Sec - amount of TVQs per second flowing into the SaF service
Output/Sec - amount of TVQs per second flowing out of the SaF service
Version - versioning number of the SaF service

The Store and Forward (SaF) service is responsible for both sending and receiving data from one Canary server to the next. As such, it also responsible for buffering data in the event that it cannot connect to the SaF service on the receiving end. Buffered data is cached to disk until the connection between the two SaF services is established allowing data to flow again. This connection is initiated from the "sender" side. The SaF service on the "receiving" side listens on the gRPC endpoint (55291) for any inbound connections. Data in transit is encrypted using TLS.

Prior to version 24, this process was handled with two separate services, the Sender and Receiver. Therein, each Canary Collector (e.g. MQTT Collector) is paired with a SaF service to send data to its next destination, whether it be a proxy server or a Historian, which is also paired with a SaF service to receive the data. Calcs & Events also uses the SaF service to send calc tags to the Historian.

Additionally, the SaF service contains a Write API (both gRPC and https) to allow the user to store data using a custom collector.

Status Screen

The 'Status' screen displays all of the sessions that are currently active.

PURGE - opens a new window to delete the selected session(s) from the SaF service. Multiple sessions can be selected using Shift+Click or Ctrl+Click.
MESSAGES - opens the Messages tile displaying all messages for the selected session

SHOW QUEUE STREAMS - filters out other sessions to only display the selected session and any other sessions waiting to be processed that come from the same data source
COMPARE - opens a CSV file containing the settings for the selected session(s) along with all the tags within the session(s). Multiple sessions can be selected using Shift+Click or Ctrl+Click.
COLUMN VISIBILITY - opens a new window where the user can decide which columns to display on the Status screen

Column Name	Description
Creation Time	Datetime of when the session was created
Session Id	A unique ID generated for each session. It is prefaced with a datetime that corresponds to its creation time.
Name	The session name as configured in the Canary Collector. "Calculations" if coming from the Calculation service. "DiagnosticHealthSystem", if enabled, coming from the Admin service.
Collector	The name of the Canary Collector/Administrator/Calculations/API logging the data
Source	The server name of the SaF service where the session originated
Hops	A sequential list of 1 or more servers where the session originated up until the current SaF service. If the session originated from source “A”, was sent through proxy “B”, then arrived at destination “C”, the field would display: “A → B → C”.
Hop Count	The number of previous machines the session came from. In the example above with machines A, B, and C, the value displayed would be 2 because data was transferred across two machines before arriving at its current destination.
Destination	The server name to where the data is being sent next. If localhost, the data will be stored in the Historian. If a remote machine, the data will be sent to another SaF service that resides on that machine.
Tags	The number of tags in the session
Input/Sec	The rate of TVQs, properties, and/or annotations received each second
Input Count	An accumulating amount of TVQs, properties, and/or annotations the session has received since the session was created
Input State	Open - the session is active, ready to receive incoming data Closed - the session can no longer receive incoming data
Output/Sec	The rate of TVQs, properties, and/or annotations sent each second
Output Count	An accumulating amount of TVQs, properties, and/or annotations the session has sent since the session was created
Output State	Open - the session is actively sending data to the next destination Closed - the session is disconnected from its destination, not sending data Hold - the session is waiting on another session to complete sending its data Throttled - connected to destination, but currently receiving a bad status code from the destination. The session enters a throttled state because something went wrong at the destination that probably requires user intervention to resolve.
Buffer Count	The amount of TVQs, properties, and/or annotations that are currently in the SaF service waiting to be sent to the next destination
Queue Order	Number of preceding sessions that must finish sending their data before this session can send its data

Configuration Screen

Endpoints

IPC - endpoint used by other local Canary services to write data to one or more destinations
gRPC Admin (55294) - endpoint used specifically by the local Admin service for configuring settings and requesting status information
gRPC API (55291) - endpoint used by other remote SaF services to write data to one or more destinations. It is also used by 3rd-party clients wishing to write data using the gRPC API.
Write API (55293) - endpoint used by 3rd-party clients to write data to one or more destinations using the Web API

CERTIFICATE (Secure/Username)

By default, Canary generates a self-signed certificate for its services that require a secure connection. These can be reconfigured if wishing to use a different certificate that has been installed on the server. Clicking the INFO button will open a new window displaying the certificate's details.

Kind - the kind of certificate the SaF service is configured to use
- SelfSignedCertificate - this is the default certificate that Canary uses. It is stored in the Local Computer\Personal store.
- Certificate - use this option if wishing to upload your own certificate. Doing so will present more options to configure.
  - Store Name - the local computer store name where the certificate is installed
  - Find Type - criteria by which the SaF service searches for the certificate
    - FindBySubjectName
    - FindByThumbprint
    - FindByTemplateName
Subject Name - set to 'default' if using the SelfSignedCertificate option. Otherwise, the value of the Subject, Thumbprint, or Template name if using the Certificate setting.

Settings

SERVICE SETTINGS

Buffer Backup (days) - stores buffer files in a backup folder (C:\ProgramData\Canary\StoreAndForward\Buffer) for the specified number of days after data has successfully been sent to destination and buffer files are deleted. By default, buffer files are deleted immediately after the data is sent. We recommend turning this feature on for customers with highly sensitive data to allow a window of recovery if a disaster event occurs and data is lost or corrupted at the destination.
Buffer Disk Space Limit (megabytes) - will buffer data up until the specified amount of disk space is left, at which point, data will not be buffered
Buffer File Rollover Size (megabytes) - a new buffer file will be created when this limit is reached
Max Buffer Stream Flush Tasks - maximum number of tasks that will be used in parallel to send data to destinations
Identity Refresh Tag Permissions (minutes) - amount of time Tag Security is cached before refreshing. Used in conjunction with the Identity API Token below if Tag Security is enabled in the Identity service.

IDENTITY

Identity API Token - token sent to a remote Historian when writing data. If Tag Security is enabled in the Identity service, a remote SaF service must be configured with an API token to write to the Historian. This token must be linked to a Canary user who is given the proper Write permissions within Tag Security. (See How to Configure a Collector when Tag Security is Enabled for more details.)

SESSION DEFAULT SETTINGS

The following settings will only be used if the client creating the session did not supply them at creation.

Inactive Timeout (minutes) - session input state will close if no messages have been received in the time specified
Is Create Dataset Enabled - allows the Historian to create new DataSets when enabled. Otherwise, if a tag specifies a new DataSet, the DataSet has to be created prior to writing to it.
Is External Property Storage Enabled - allows writing properties to external SQLite database when enabled. The Views service must also be enabled to allow external property storage. Only applicable if using the SQLite option.
Is Insert or Replace Data Enabled - allows writing data prior to the last timestamp stored in the Historian. When data is not sent in order, the Historian loses some of its performance because it has been optimized for time-series data in a forward direction.
Is Timestamp Extension Enabled - extends the timestamp of the last value if the value has not changed in the last minute
Is Write No Data On Close Enabled - writes a “NoData” quality when the session is closed to mark when data logging stopped