Enabling Dynamic DataSets (version 26)

Some collectors (MQTT, OPC, SQL, and CygNet)* can dynamically create additional DataSets when the tag count within a single DataSet exceeds a specified limit. This helps reduce the overall size of the daily HDB files generated within each DataSet and prevents a single file from reaching the 500 MB maximum, which would otherwise force the Historian to create an additional file for that day.

Each new DataSet is incremented numerically (DataSet, DataSet2, DataSet3, etc.). Each collector maintains its own tag-to-DataSet mapping within a SQLite file.

* The OPC collector only uses dynamic DataSets for path-based sessions.
The SQL and CygNet collectors only use dynamic DataSets if an asterisk (*) is appended to the configured DataSet name.

Enabling Through Store & Forward (SaF)

If a Collector (e.g., Logger or Canary Historian Module for Ignition) does not inherently support dynamic DataSet creation, or is not configured to use it (see note* above), the Store & Forward (SaF) service on either the Collector or Historian server can be configured to handle this.

• If enabled on the Collector side, only sessions originating from that Collector will use dynamic DataSets.
• If enabled on the Historian side, all sessions—excluding those mentioned above that manage their own DataSets—will use dynamic DataSets.

Configuration Steps

1. Open %ProgramData%\Canary\StoreAndForward\store_and_forward.service.json on the appropriate server (Collector or Historian).
2. Set "DynamicDataSetIsEnabled" to true.
3. Set "DynamicDataSetTagLimit" to the desired value.
   • Default: 25,000
   • Minimum: 1,000
4. Save the file and restart the SaF service.

How It Works

When enabled, the SaF service on the Historian checks whether tags in a logging session already exist within a specific DataSet. If they do, SaF continues storing those tags in the same DataSet. This mapping is backed up in:

%ProgramData%\Canary\StoreAndForward\DynamicDataSets\DynamicDataSets.sqlite

If the tags do not exist in the Historian but a mapping exists in the backup SQLite file, a warning is logged and the tags are stored in the DataSet(s) defined in that file. Although rare, this can occur if dynamic DataSets were previously enabled and the tags were later removed.

If the tags exist in neither the Historian nor the SQLite file, the SaF service creates a new mapping and stores the tags accordingly.

Existing Tags Behavior

If tags are already being logged before dynamic DataSets are enabled—and the number of tags in the DataSet already exceeds the "DynamicDataSetTagLimit"—those tags will continue logging to the same DataSet. Any new tags will be written to a new DataSet.