Creating a New DataSet

A DataSet is a user-defined grouping of tags. DataSets play a key role in logging, storing, and reading data. Depending on the collector being used, the DataSet may need created before data can be logged.

The OPC DA Collector requires the user to pre-create the DataSet before logging begins.

The SQL Collector requires the user to pre-create the DataSet if defining it in 'Configuration' U/I. If a DataSet is NOT defined, the DataSet can be dynamically created.

The CSV Collector has the option to automatically create the DataSet based upon the <AutoCreateDataset> parameter configured in the SAF_Import.config file. By default, this is set to True.

All other collectors can automatically create the DataSet when logging begins.

To create a Dataset:

1. Open the Canary Administrator client.
2. Navigate to the Historian tile>Configuration screen then click NEW.
3. Enter the DataSet name. DataSets must have unique names, not start or end with a 'space' and cannot use the following characters:

   Colon	:
   Equals	=
   Forward Slash	/
   Asterisk	*
   Backwards Slash	\
   Percentage Sign	%
   Question Mark	?
   Greater than or equal to	< >

4. By default, all DataSets are stored in C:\Historian Data. In the Path field, type a new path or select Browse to change the directory location of the data for the DataSet.

5. Finish by selecting CREATE.
6. Adjust parameters within the SETTINGS window at right.  Each setting is explained below.

Keep the total number of DataSets below 300 and maintain less than 40,000 tags per DataSet. While there are no hard limits coded into the Historian, increasing beyond these numbers may result in performance issues or errors.

Adjusting DataSet Parameters

Path

The Path parameter displays the directory of where the HDB files are stored for the selected DataSet. See How to Change the Data Historian Location if wishing to change this directory.

DataSet Validation Configuration

Background validation checks all DataSets for unvalidated HDB files. An unvalidated HDB file is indicated by a black clock in the top left hand corner of the tile as shown on the Dec-22 HDB file below. 

By default, background validation is turned on. The background validation process may consume additional machine resources but ensures that HDB files do not contain incomplete or corrupt data values. 

A file remains unvalidated while it is still open for logging. Typically files are validated each night after they roll-over. It is not recommended to uncheck background validation.

Configure Email Notifications for DataSets Not Logging Data

The Historian can be configured to send an email to the system admin(s) if a DataSet has not received any data in the specified period of time. See Configuring an Alert when a DataSet Stops Receiving Data.

Enabling this feature is a recommended precaution for monitoring the network connection break between Store and Forward (SaF) services, monitoring the status of a logging machine, or ensuring that logging equipment maintains functionality. 

Configuring the Roll-Over Frequency for a DataSet

A Roll-Over occurs when a new HDB file is created. To ensure data integrity, it is recommended to roll-over files daily.

• When an HDB file is open for writing, it could potentially become corrupted; thus the longer an HDB file is open for writing, the greater the potential for data loss.
• Data corruption can be caused by power outages, machine restarts, and other service disruptions. 

Setting the Roll-Over to daily is a recommended best practice to take to ensure your data remains secure despite unexpected network failures.  

Configuring the Roll-Up Frequency for a DataSet

The Roll-Up setting determines how often individual HDB files get condensed into one rolled-up file. A rolled-up file is either daily, weekly, monthly, or quarterly depending on how many HDB files the system administrator wishes to condense.

HDB file roll-ups lessen storage space by deleting redundant tag names and metadata property descriptions from multiple HDB files.

• For example, instead of listing Tag0001 in thirty separate HDB files, a monthly roll-up lists Tag0001 in one monthly file while retaining all of Tag0001's TVQs from the last thirty days.

The Roll-Up feature is off by default. It is not necessary, but an Admin may want to condense many HDB files to save  storage space. The Roll-Up feature can be most effective when using large numbers of DataSets with smaller tag counts.

Canary does not allow a file to grow larger than 512 MB. If a roll-up would cause this to happen, multiple roll-up files would be created until the daily files are condensed. It is recommended to not roll-up files in this scenario then.

Delete HDB Files after a Set Period of Time

Defaulted to 'Never' this setting can be customized to automatically delete a DataSet's HDB files after a specific period of time. 

1. Set the 'Delete' section to 'Custom'. Adjust the interval and time parameter accordingly.
2. NEVER make this setting less than the Roll-Up time. Doing so will delete the entire Roll-up file that the latest HDB file is stored within.

Moving Old HDB Files Offline

If wishing to "archive" older HDB files, the user can select the 'Custom' radio button then check the box to 'Move deleted files to [filepath]'. This file path is configured in the Settings tab on the left. For more information about moving files, see How to Move Historical Data to a Temp Directory for Cold Storage.

Deleting a DataSet

A DataSet can be deleted from the system by following these steps.

1. Open the Canary Admin client and navigate to the Historian tile>Configuration screen.
2. Select the DataSet you wish to remove.
3. Click REMOVE to permanently delete the DataSet.
4. An option to also Delete All Associated Historical Data is presented.  Only check this box if you wish to permanently delete all data associated with this DataSet. This cannot be undone.
5. Click 'ok'.

Always delete DataSets from the Historian U/I. Never delete them from the file directory, doing so will create undesired errors and cause the Historian to malfunction. 

A DataSet cannot be removed if it is being actively logged to or accessed. If the user is unable to determine which client is accessing the DataSet, they can disable the SaF (writing) or Views (reading) service. Doing so will cause temporary interruption for any collector logging data and any user trying to access data in the Historian.

Importing DataSets and HDB Files

Entire DataSets or HDB files can be imported as necessary. This can serve the purpose of disaster recovery or sharing HDB files with other parts of an organization. 

Importing HDB Files

1. Open the Windows File Explorer and navigate to your DataSet directory, by default: C:\Historian Data\.
2. Place the HDB files you wish to import within the existing DataSet.
3. Open the Canary Admin and select the Historian tile. Highlight the DataSet with the new HDB files by clicking on it once, thus turning it yellow. 
4. From the DATASET window (bottom left) click the yellow down-arrow to open the DataSet menu.
5. Select REFRESH FILE LIST.  This action scans the selected DataSet and automatically discovers all new HDB files.
6. You may now browse the DataSet to confirm the files were imported.

Importing Entire DataSets

1. Open the Windows File Explorer and navigate to your DataSet directory, by default: C:\Historian Data\.
2. Place the DataSet and corresponding HDB files you wish to import within the folder.
3. Open the Canary Admin and navigate to the Historian tile>Configuration screen. Select SCAN.
4. Select BROWSE. Find and highlight the newly imported DataSet folder. Select OK.
5. Select SCAN.

You can use the 'Include Subdirectories' checkbox to instruct the scan tool to look to all subdirectories within a folder and automatically detect and create multiple DataSets.

Renaming a DataSet

1. Open the Canary Admin client and navigate to the Historian tile>Configuration screen.
2. Select a DataSet. Select RENAME. 
3. Enter the new DataSet name in the New name field. DataSets must have unique names, not start or end with a 'space' and cannot use the following characters:

   Colon	:
   Equals	=
   Forward Slash	/
   Asterisk	*
   Backwards Slash	\
   Percentage Sign	%
   Question Mark	?
   Greater than or equal to	< >

4. Select ok. This has now changed the name of all the HDB files within the DataSet.

DataSets currently in use cannot be renamed or deleted. A DataSet is in use when data is being written to or read from it. The user may stop the SaF (writing) or Views (reading) service if needing to prevent clients from accessing the data.