0

Store and Forward Tile (version 25)


  • Streams - number of data streams coming through the Store and Forward (SaF) service
  • Buffer - amount of updates, or TVQs (Timestamp, Value, Quality), buffering in the SaF service. A buffer occurs whenever the Input/Sec exceeds the Output/Sec.
  • Input/Sec - amount of TVQs per second flowing into the SaF service
  • Output/Sec - amount of TVQs per second flowing out of the SaF service
  • Version - version number of the SaF service

The Store and Forward (SaF) service is responsible for both sending and receiving data from one Canary server to the next. As such, it also responsible for buffering data in the event that it cannot connect to the SaF service on the receiving end. Buffered data is cached to disk until the connection between the two SaF services is established allowing data to flow again. This connection is initiated from the "sender" side. The SaF service on the "receiving" side listens on the gRPC endpoint (55291) for any inbound connections. Data in transit is encrypted using TLS.

Prior to version 24, this process was handled with two separate services, the Sender and Receiver. Therein, each Canary Collector (e.g. MQTT Collector) is paired with a SaF service to send data to its next destination, whether it be a proxy server or a Historian, which is also paired with a SaF service to receive the data. Calcs & Events also uses the SaF service to send calc tags to the Historian.

Additionally, the SaF service contains a Write API (both gRPC and https) to allow the user to store data using a custom collector.

Status Screen

 The Status screen displays all of the sessions that are currently active.

  • PAUSE - pauses the selected session from forwarding data to its destination. Data will buffer when it is paused.
  • UNPAUSE - unpauses the paused session to resuming forwarding data to its destination
  • PURGE - opens a new window to delete the selected session(s) from the SaF service. Multiple sessions can be selected using Shift+Click or Ctrl+Click.

    A backup of purged sessions are stored indefinitely by default in C:\ProgramData\Canary\StoreAndForward\Purged.

  • MESSAGES - opens the Messages tile displaying all messages for the selected session
  • SHOW QUEUE STREAMS - filters out other sessions to only display the selected session and any other sessions waiting to be processed that come from the same data source
  • COMPARE - opens a CSV file containing the settings for the selected session(s) along with all the tags within the session(s). Multiple sessions can be selected using Shift+Click or Ctrl+Click.
  • COLUMN VISIBILITY - opens a new window where the user can decide which columns to display on the Status screen

Column Name Description
Creation Time Datetime of when the session was created
Stream Id A unique ID generated for each session. It is prefaced with a final destination endpoint and a datetime that corresponds to its creation time.
Name The session name as configured in the Canary Collector. "Calculations" if coming from the Calculation service. "DiagnosticHealthSystem", if enabled, coming from the Admin service.
Collector The name of the Canary Collector/Administrator/Calculations/API logging the data
Source The server name of the SaF service where the session originated
Hops A sequential list of 1 or more servers where the session originated up until the current SaF service.  If the session originated from source “A”, was sent through proxy “B”, then arrived at destination “C”, the field would display: “A → B → C”.
Hop Count The number of previous machines the session came from. In the example above with machines A, B, and C, the value displayed would be 2 because data was transferred across two machines before arriving at its current destination.
Destination The server name to where the data is being sent next. If localhost, the data will be stored in the Historian. If a remote machine, the data will be sent to another SaF service that resides on that machine.
DataSets Name of the DataSet where the tags will be stored in the Historian.
Tags The number of tags in the session
Input/Sec The rate of TVQs, properties, and/or annotations received each second
Input Count An accumulating amount of TVQs, properties, and/or annotations the session has received since the session was created
Input State Open - the session is active, ready to receive incoming data
Closed - the session can no longer receive incoming data
Output/Sec The rate of TVQs, properties, and/or annotations sent each second
Output Count An accumulating amount of TVQs, properties, and/or annotations the session has sent since the session was created
Output State Open - the session is actively sending data to the next destination
Closed - the session is disconnected from its destination, not sending data
Hold - the session is waiting on another session to complete sending its data
Throttled - connected to destination, but currently receiving a bad status code from the destination. The session enters a throttled state because something went wrong at the destination that probably requires user intervention to resolve.
Buffer Count The amount of TVQs, properties, and/or annotations that are currently in the SaF service waiting to be sent to the next destination
Queue Order Number of preceding sessions that must finish sending their data before this session can send its data

Configuration Screen

Endpoints

  • IPC - endpoint used by other local Canary services to write data to one or more destinations
  • gRPC Admin (55294) - endpoint used specifically by the local Admin service for configuring settings and requesting status information
  • gRPC API (55291) - endpoint used by other remote SaF services to write data to one or more destinations. It is also used by 3rd-party clients wishing to write data using the gRPC API.
  • Write API (55293) - endpoint used by 3rd-party clients to write data to one or more destinations using the Web API

CERTIFICATE (Secure/Username)

By default, Canary generates a self-signed certificate for its services that require a secure connection. These can be reconfigured if wishing to use a different certificate that has been installed on the server. Clicking the INFO button will open a new window displaying the certificate's details.

  • Kind - kind of certificate the SaF service is configured to use
    • SelfSignedCertificate - this is the default certificate that Canary uses. It is stored in the Local Computer\Personal store.
    • Certificate - use this option if wishing to upload your own certificate. Doing so will present more options to configure.

      • Store Name - local computer store name where the certificate is installed
      • Find Type - criteria by which the SaF service searches for the certificate
        • FindBySubjectName
        • FindByThumbprint
        • FindByTemplateName
  • Subject Name - set to default if using the SelfSignedCertificate option. Otherwise, the value of the Subject, Thumbprint, or Template name if using the Certificate setting.

Settings

 SERVICE SETTINGS

  • Buffer Backup (days) - stores buffer files in a backup folder (C:\ProgramData\Canary\StoreAndForward\Backup) for the specified number of days after data has successfully been sent to destination and buffer files are deleted. By default, buffer files are deleted immediately after the data is sent. We recommend turning this feature on for customers with highly sensitive data to allow a window of recovery if a disaster event occurs and data is lost or corrupted at the destination.
  • Buffer Disk Space Limit (megabytes) - will buffer data up until the specified amount of disk space is left, at which point, data will not be buffered
  • Buffer File Rollover Size (megabytes) - a new buffer file will be created when this limit is reached
  • Max Buffer Stream Flush Tasks - maximum number of tasks that will be used in parallel to send data to destinations
  • Identity Refresh Tag Permissions (minutes) - amount of time Tag Security is cached before refreshing. Used in conjunction with the Identity API Token  below if Tag Security is enabled in the Identity service.
  • Lazy Buffer Transaction Timeout (ms) - Transactions are timed when they occur. A transaction is a chunk of data that is sent through SaF. It may have 1 element, or it may have 50,000 elements. A larger transaction is expected to take longer, but this transaction timeout is a general limit that does not consider what the message size is. If the request takes longer than this amount of time to complete, the output of the session will get marked as "Slow" and will not try to write to the client, but will instead write to the buffer. This can reset when the buffer gets flushed back to 0 and it will go back into normal mode again when the transactions are under this threshold.
  • Is Lazy Buffer Enabled - when checked, data is written directly to the target (e.g. the Historian) bypassing the buffer. If this fails or there is already a buffer in the SaF service, the transaction falls back to its normal procedure of using the buffer file.
  • Throttle Concurrent DataSet Transaction Limit - used when attempting to flush buffered data. This setting controls how many transactions can attempt to write to a DataSet at the same time. If the DataSet is already being written to by the set number of transactions, the SaF service will move on to another DataSet. This allows for a more efficient way of flushing the buffer.
  • Throttle Session Creation Limit Within Time Window - number of sessions that can be created within the configured time window

  • Throttle Session Creation Time Window (seconds) - time window used in conjunction with the creation limit within time window setting

IDENTITY

  • Identity API Token - token sent to a remote Historian when writing data. If Tag Security is enabled in the Identity service, a remote SaF service must be configured with an API token to write to the Historian. This token must be linked to a Canary user who is given the proper Write permissions within Tag Security. (See How to Configure a Collector when Tag Security is Enabled for more details.)

SESSION DEFAULT SETTINGS

The following settings will only be used if the client creating the session did not supply them at creation.

  • Inactive Timeout (minutes) - session input state will close if no messages have been received in the time specified
  • Is Create Dataset Enabled - allows the Historian to create new DataSets when enabled. Otherwise, if a tag specifies a new DataSet, the DataSet has to be created prior to writing to it.
  • Is External Property Storage Enabled - allows writing properties to external SQLite database when enabled. The Views service must also be enabled to allow external property storage. Only applicable if using the SQLite option.
  • Is Insert or Replace Data Enabled - allows writing data prior to the last timestamp stored in the Historian. When data is not sent in order, the Historian loses some of its performance because it has been optimized for time-series data in a forward direction.
  • Is Timestamp Extension Enabled - extends the timestamp of the last value if the value has not changed in the last minute
  • Is Write No Data On Close Enabled - writes a “NoData” quality when the session is closed to mark when data logging stopped

Reply

null