Data Collector Overview and Methodology (version 26)

Knowledge Base / Version 26 / System Admin Duties / Data Collection

updated 11 days ago

What are Data Collectors?

Data Collectors are the components of the Canary system that connect to a data source to collect the process data to be transferred to the Canary Historian. Each Data Collector has an interface which allows the administrator to customize how data is logged as well as indicate to which Canary Historian(s) the data is sent.

Types of Data Collectors

OPC UA Collector

Built to follow the standards of the OPC Foundation for connecting to OPC UA servers.
Install with Canary Installer. From within the Canary Admin client, users can create, modify, and oversee the status of all OPC UA logging sessions through the OPC Collector tile.
Logging session can be tag-based, where users browse for specific tags they wish to log, or path-based, where users subscribe to particular node within the OPC server and log the tags that fall under that node.
Configuration settings include the application of deadbands, data transformation, adjusting the update rate (Sample Interval), adding custom metadata, and adjusting the decimal precision of timestamps using timestamp normalization.

OPC DA Collector

Built to follow the standards of the OPC Foundation for connecting to OPC DA servers.
Install with Canary Installer. From the Logger Administrator application, users can create, modify, and oversee the status of local and remote OPC DA logging sessions.
Configuration settings include the application of deadbands, adjusting update rates, configuring pre-defined metadata, creating data transformations, adjusting timestamp decimal precision, and triggering logging based on the value of another tag.

MQTT Collector

Built to follow the standards of the SparkplugB specification 2.2 and 3.0. In addition to SparkplugB payloads, it can also be configured to consume JSON payloads.
Install with Canary Installer. From the Canary Admin client users can configure, enable/disable, or monitor multiple sessions using the MQTT Collector tile.
Configuration settings include the ability to subscribe to multiple brokers and topics, the assignment of primary host to leverage store and forward, using birth and death messages for state of awareness, as well as enabling TLS security.

SQL Collector

Supports Microsoft SQL Server, MySQL, and Oracle databases.
Install with Canary Installer. From within the Canary Admin client, users can create, modify, and oversee the status of all SQL logging sessions using the SQL Collector tile.
Functionality includes the ability to convert existing historical SQL databases to Canary Historian archives and/or read new ongoing updates. The SQL Collector requires a specific queue table to stage records, imports them at a configurable interval, and then discards the records from the queue table. The performance benefits from this methodology are drastically superior to reading production tables.

Cygnet Collector

Built to utilize the CygNet .NET API.
Install with Canary Installer. From the C:\ProgramData\Canary\Logger\StoreAndForward\CygNetCollector folder, edit the SQLite database provided by the Canary team to create or modify CygNet logging sessions. From within the Canary Admin client, users can enable, monitor, or disable existing sessions using the CygNet Collector tile.
Configuration includes facility and UDC combinations, the mapping of CygNet attributes to Canary tag properties, and the auto-discovery of new CygNet facilities.

CSV Collector

Created to import flat files using the .CSV format on arrival providing a simple and easy solution for logging data from external systems without using APIs.
Install with Canary Installer. Requires the modification and creation of a JSON file used to parse the CSV files it will consume. The JSON file must reside in %ProgramData%\Canary\CsvCollector\Directories.
With over 40 parameters to choose from, the CSV Collector can be configured to parse almost any file format whether they be column-based or row-based data structures.

Canary Historian Module for Ignition

Approved by Inductive Automation and available on the Inductive Automation Third-Party Module Showcase.
From the Ignition Gateway, users can access the Platform>Modules menu to install the Canary Historian module
Configuration settings include the native historian functionality available from within the Ignition platform, including the ability to log to multiple Canary Historian instances as well as the added flexibility to query data from Canary Historians from Ignition controls.

API Collector

Offered to allow for custom gRPC or Web API Collector development.
Configuration settings include the availability of the same calls and settings as used in other Canary Collectors to empower custom Collector development.

Data Collector Licensing

The Canary Data Collectors are not licensed and all Collectors are included with the purchase of the Canary System. However, Collectors do need to be installed individually using the Canary Installer. By choosing not to license individual data Collectors, it is easier to follow best practice architectures and ensure robust solutions.

Store and Forward (SaF) Service

The Canary SaF service is the main component that works with each Collector ensuring reliable delivery to the Historian. Communication is built on gRPC framework and provides secure and encrypted data transmission. Therein, the SaF service is also responsible for:

Network/Historian machine state awareness by monitoring the connection between the SaF service on the Collector side and the SaF service on the Historian side.
Data caching to local disk should the SaF service on the Collector side lose connection to the SaF service on the Historian side. Cache will continue for as long as disk space permits. By default, the SaF service will stop writing new values to disk when only 4000 MB of space remains. (See Store and Forward Tile - Settings)
Notification of system administrators that data logging has not updated in a set period of time (customized by DataSet and managed from the Canary Admin>Historian>Configuration screen).
Upon the re-establishment of the network or connection to the Historian machine, the SaF service will automatically backfill the Canary Historian archive with all buffered values and clear the local disk cache.

The SaF service is used by all Canary Data Collectors, the Calculation service, the Admin service (when the Diagnostic Health System is enabled), as well as the Historian. Data Collectors communicate with the SaF Service via a public API. This means with some programming knowledge, a user can configure their own custom Data Collectors using the Write API (gRPC or https).

User Authentication

The SaF service supports user authentication through the use of API tokens. API tokens are generated in the Identity service and linked to an internal Canary user who then must be added to the proper node within the Identity tile>Security>Tag Security screen. When it comes to collecting data, API tokens are used

Inserting or Editing Data

The most efficient way to write data is chronologically with the Historian appending the data to the end of the archive. If data arrives out of order, the Historian needs to determine where in the historical archive it should reside which is less performant. In some cases, depending on the data collection method, data is rejected when it reaches the Historian if the timestamp moves in a backwards direction.

If needing to insert or overwrite historical data using the Write API, the user can specify the InsertReplaceData parameter when requesting a session token and set it to True.

When it comes to Canary Collectors, most support data insertions or can be configured to do so. The exception to this is the OPC DA and OPC UA Collectors. If a historical value is received from one of these Collectors an error will be thrown indicating "the timestamp moved in a backwards direction" or "the timestamp is a duplicate" and will be rejected.

Multiplexing Data Streams

The SaF service can send a stream of data to multiple destinations in the scenario where there is more than one Historian within the system architecture. This is typically configured within the Collector by adding additional Historian (or proxy) destinations separated by a comma.