Data Logging

Below are instructions for creating logging sessions from both an OPC UA data feed as well as an MQTT Broker.

This lab is designed for you to connect to existing data sources hosted by Canary in our AWS instance 'datagen.canarylabs.online' and will require an internet connection to log data. 

Should you wish to re-create the data generation environment we are hosting, you may do so by following the 'Installing Local Data Feeds' instructions, otherwise, bypass this step.

Installing Local Data Feeds - Optional Not Advised

It is best to follow along with the course and connect to the AWS data sources we are demonstrating.  However, if you cannot access the AWS machine or really need a local instance of data generation, follow this section guides.  If not, skip to the next section.

If installing your entire demo on a single server, you will need the following components included.  Also, ensure you have selected 'Mosquitto MQTT Server' from the tab '3rd Party Components'.

The attached zip folder 'Bootcamp - Data Logging' will provide a series of files that will be useful in the configuration process.  Each of these files will need placed in specific directories within C:\ProgramData\Canary Labs\ as well as other locations.  

Data Generation Service

The Canary Data Generator application can run the 'PumpsAndTowers.dgf' as a service.  This will use the application 'CanaryDataGenerator.exe' to produce a simulated data stream that is published to the local Mosquitto MQTT Server.

Step One

The three files included in the Bootcamp - Data Logging.zip file will need placed in the C:\ProgramData\Canary Labs\ Data Generation folder.

Step Two

Configure the file

C:\Program Files\Canary\Collectors\Data Generator\DataGeneration.config

to point to

C:\ProgramData\Canary Labs\ Data Generation\PumpsAndTowers.dgf

<Settings>
  <Historian>localhost</Historian>
  <ClientName>DataGeneration</ClientName>
  <ClientCount>0</ClientCount>
  <PacketSize>32000</PacketSize>
  <FileName>C:\ProgramData\Canary Labs\Data Generation\PumpsAndTowers.dgf</FileName>
</Settings>

Step Three

Create a Windows Service targeting 

C:\Program Files\Canary\Collectors\Data Generator\SAF_DataGenerationService.exe

and name it 'CanaryDataGen'.

Step Four

Ensure the correct configuration of the file... 

C:\ProgramData\Canary Labs\ Data Generation\SAF_DataGenerationTarget.MQTT.xml 

Pay specific attention to the port configuration.  If using 8883 rather than 1883 in your instance, add credentials for Username and Password.  Changing this to 8883 will require you to modify steps below when connecting to the MQTT server.

<?xml version="1.0" encoding="utf-8"?>
<TargetSettingsBase xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:q1="SAF_DataGenerationCore.LogTargets.MQTT" xsi:type="q1:Settings" xmlns="SAF_DataGenerationCore.LogTargets">
  <q1:Host>localhost</q1:Host>
  <q1:Port>1883</q1:Port>
  <q1:ClientId>2a52999ee9bf445c9c7376c27ad4183a</q1:ClientId>
  <q1:Username />
  <q1:Password />
  <q1:GroupId>CanaryDataGeneration</q1:GroupId>
  <q1:NodeId>Node1</q1:NodeId>
  <q1:DeviceId>Device1</q1:DeviceId>
  <q1:SSL_Protocol>0</q1:SSL_Protocol>
  <q1:QOS_Level>0</q1:QOS_Level>
  <q1:UseTagAliases>false</q1:UseTagAliases>
  <q1:UseCompression>false</q1:UseCompression>
</TargetSettingsBase>

Step Five

Run the newly created CanaryDataGen service.

KepServer

The included files 'TRAVIS.csv' and 'HAYS.csv' can be imported into KepServer to duplicate the simulation locally if you prefer.  You will need to ensure they are imported under Simulation Examples.Functions on your KepServer.  Doing so will keep the OPC Node ID consistent with our documentation.

Logging Data via OPC UA

We will create two different OPC UA logging sessions, each targeting a different Canary DataSet.  If you are already confident in all of the steps below you can simply import the included OCF files into the ProgramData\Canary Labs\OPC Collector directory.  However, it is encouraged that you 

Logging Session #1 - Travis Water District 

1. Select OPC UA Collector tile

2. Click 'Configuration' from bottom tab menu

3. Create 'New Session'

4. Set Name to = Travis

5. Historian = localhost

6. Set DataSet to = Travis 

7. OPC Server Host name = opc.tcp://datagen.canarylabs.online:49320

8. Port = (in URL)

9. Security = false

    Create Pumps group 

1. Rename Group 1 to = Pumps

2. Browse the OPC Server > Simulation Examples > Functions > Travis 

3. Filter ‘Pump’ and add all tags

    Create Towers group 

1. Create New Group and set name to = Towers

2. Change Sample Interval to 00:00:30

3. Browse the OPC Server > Simulation Examples > Functions > Travis  

4. Filter ‘Tower’ and add all tags

    Begin Logging

1. Apply all changes
2. Select lower tab 'Status'
3. Enable logging by clicking the 'Play' button on the 'Travis' panel

Verify data is flowing from the OPC Collector to the Sender service.  By selecting the Sender tile in the Canary Admin application you should see the newly created session Localhost/Travis.  The state should be 'Active' and you should have 51 tags logging with Store/Sec and Send/Sec values changing.  The TVQs column should be at 0.  If the TVQs column is growing it means you are buffering data in Sender and not moving data into the Canary Historian you targeted in the logging session configuration.  Double-check your configuration. 

Logging Session #2 – Hays Water District 

1. Create 'New Session'

2. Set Name to = Hays

3. Historian = localhost

4. Set DataSet to = Hays

5. OPC Server Host name = opc.tcp://datagen.canarylabs.online:49320

6. Port = (in URL)

7. Security = false

    Create Pumps group 

1. Rename Group 1 to = Pumps

2. Add metadata tag properties by clicking the 'TAG PROPERTIES' button in the top right corner.  Some tag properties like 'Description' and 'Eng Units' are in the dropdown box and can be selected and then added by clicking the 'ADD' button.  Others will need typed exactly as they are listed.  Ensure you follow the same order as they appear below before clicking 'APPLY'.  Tag Properties should be in the following order and maintain case sensitivity:

   • Description

   • Eng Units

   • Model

   • Horsepower

   • Tank Height 

3. Download CSV file ‘Hays Logging Configuration’ 

4. Open in Excel and select the 'Hays Pumps' sheet.  Copy the contents to your clipboard without including the header row by selecting the entire table except row 1.  Do not select the entire rows, only the columns and rows that have data within them.

5. In OPC configuration select ‘Paste New’ 

6. Click 'APPLY' to save your changes.

    Create Towers Group 

1. Create New Group and name = Towers

2. Use the sheet ‘Hays Towers’ in your open CSV file to import the logging configuration as you just did above.

3. In OPC configuration select ‘Paste New’ 

4. Click 'APPLY' to save your changes.

    Begin Logging

1. Apply all changes
2. Select lower tab 'Status'
3. Enable logging by clicking the 'Play' button on the 'Hays' panel

Again, verify data is flowing for this newly created logging session through the Sender and into the Canary Historian you have targeted.

Logging Data Via MQTT

We will create two different MQTT logging sessions, each targeting a different Canary DataSet.  If you are already confident in all of the steps below you can simply import the included MQTTCollector.sqlite file into the C:\ProgramData\Canary Labs\Desktop\Bootcamp - Data Logging\ProgramData_Canary\Logger\StoreAndForward\MQTTCollector directory.

MQTT Collector – Comal WD

1. Open Canary Admin application
2. Select MQTT Collector tile
3. Click 'Configuration' from the lower tab menu

Group Configuration

Create new Connection Group by selecting the '+' icon and edit the following Group fields:

1. Set Name to = Comal WD
2. MQTT Client ID = generate random ID using 'generate' icon to the right of dialogue box
3. Primary Application = true
4. Request Device Births = true

Server Configuration

Note, if you are using your own local instance rather than the Canary AWS MQTT server you will use port 1883 and leave username and password blank.  If you are using 8883 locally enter the appropriate username and password rather than the ones we have provided.

Select the default Servers that was created 'tcp://localhost:1883' from the left and edit the following Server fields: 

1. URI = tcp://datagen.canarylabs.online:8883/
2. Username = Bootcamp
3. Password = SirYesSir!
4. Reconnect Interval = 00:01:00
5. Keep Alive = 30
6. Clean Session = true
7. Enable SSL/TLS = true
8. Self signed certificate = true
9. Client certificate = false

Subscriptions Configuration

Select the default Subscriptions that was created 'spBv1.0/#' from the left and edit the following Subscription fields:

1. Topic = spBv1.0/#
2. QOS = At Most Once (QoS = 0)
3. Enabled = true
4. Dataset Prefix = Comal
5. Tags Per DataSet = 25000
6. Remove tag name prefix = 3
7. Automatically Log All Tags = true
8. Sync historian tags with births = false

Sender Sessions

Select the default 'localhost' Sender Sessions that was created from the left and edit the following fields:

1. Historian = localhost
2. Enabled = true
3. External Property Storage = false

Filters

Create a filter:

1. Select the '+' icon beside 'Filters'
2. Expression = ^Comal
3. Exclude = false

Begin Logging

1. Apply all changes
2. Select lower tab 'Status'
3. Enable session
4. Select lower tab 'Configuration'
5. Select 'Tags' and confirm results

You can verify data is being logged by looking at the Sender tab in the Canary Admin.  You will see the session and will also be able to see data values updating in the Canary Historian tile.

MQTT Collector – Guadalupe WD

1. Open Canary Admin application
2. Select MQTT Collector tile
3. Click 'Configuration' from the lower tab menu

Group Configuration

Create new Connection Group by selecting the '+' icon and edit the following Group fields:

1. Set Name to = Guadalupe WD
2. MQTT Client ID = generate random ID using 'generate' icon to the right of dialogue box
3. Primary Application = true
4. Request Device Births = true

Server Configuration

Note, if you are using your own local instance rather than the Canary AWS MQTT server you will use port 1883 and leave username and password blank.If you are using 8883 locally enter the appropriate username and password rather than the ones we have provided.

Select the default Servers that was created 'tcp://localhost:1883' from the left and edit the following Server fields: 

1. URI = tcp://datagen.canarylabs.online:8883/
2. Username = Bootcamp
3. Password = SirYesSir!
4. Reconnect Interval = 00:01:00
5. Keep Alive = 30
6. Clean Session = true
7. Enable SSL/TLS = true
8. Self signed certificate = true
9. Client certificate = false

Subscriptions Configuration

Select the default Subscriptions that was created 'spBv1.0/#' from the left and edit the following Subscription fields:

1. Topic = spBv1.0/#
2. QOS = At Most Once (QoS = 0)
3. Enabled = true
4. Dataset Prefix = Guadalupe
5. Tags Per DataSet = 25000
6. Remove tag name prefix = 3
7. Automatically Log All Tags = true
8. Sync historian tags with births = false

Sender Sessions

Select the default 'localhost' Sender Sessions that was created from the left and edit the following fields:

1. Historian = localhost
2. Enabled = true
3. External Property Storage = false

Filters

Create a filter:

1. Select the '+' icon beside 'Filters'
2. Expression = ^Guadalupe
3. Exclude = false

Begin Logging

1. Apply all changes
2. Select lower tab 'Status'
3. Enable session
4. Select lower tab 'Configuration'
5. Select 'Tags' and confirm results

You can verify data is being logged by looking at the Sender tab in the Canary Admin.  You will see the session and will also be able to see data values updating in the Canary Historian tile.