Dataset storage questions
I'm trying to understand how Canary 'knows' what tags it has. As far as I can tell, if a collector sends data for a tag name then that tag is created in the relevant dataset file for that day, but if no further data is seen for that tag then it's existence is 'forgotten' in subsequent dataset files. So if I ask for the most recent value for tag 'x' but it last received a value 2 years ago, Canary must search through 2 years of archives before it even knows that the tag even exists. Is that right, or is there a separate database table logging all the tags and when they last had data?
Similarly, where are the metadata properties for a tag held? If a tag has Description and Units properties, that were imported from a text file 3 months ago, are those properties stored with each new time-series value (surely not!)? Or once to each subsequent dataset file? Or just left in the DS file they were first uploaded to? Or to a database table completely separate from the DS files?
2 replies
-
Hi ,
The Views service acts as the gateway to the Historian for any client coming in to consume data. The Views service gets its tag list from the Historian based upon the last file in each DataSet. If a tag is not in the last file of a DataSet, Views is not aware of it, and therefore clients will not be able to browse it. This is also what determines the tag count that goes against the Historian license. So a tag may exist in historical files, but it does not go against the tag count if it is not in the latest file.
When the Views service starts up, it caches all of these tags to memory so that it is aware of what it has when a client requests data.
Metadata can be stored in the Historian and/or stored in an external database. Metadata in the Historian is stored in the daily HDB3 files. When the Historian does its roll-over at midnight and creates a new file for the next day, these properties are rolled into it and carried along. So essentially, each file would have the properties. Again, the properties would have to reside in the last file of the DataSet in order for any client to see them, similar to tags. Now, if you imported properties 3 months ago, I would expect those properties to be carried along throughout the files until there was a manual intervention to force the Historian to do a roll-over. Some of our collectors (and Write API) have the ability to create a new file in the Historian. Typically, this is used if wanting to remove tags from the latest file. In order to do this, a new file must be created that only has the tags you want going forward. When this happens though, only the tags (and their properties coming from the data source) are carried into the new file as it is a break from the previous file. If the old file had properties that came from a CSV import, they would not get carried over into the new file that is created from the collector.
If metadata is stored in an external SQL database, you don't have to worry about properties disappearing from the DataSet whenever a new file is manually created in the Historian.
Hope this was helpful!
-
Thank you, that clarifies a lot. I'm currently trying to implement external metadata storage but am hitting what Canary currently think may be a bug when making the initial connection to the SQL Server instance.
Print this page