Filtering Options for JSON Message vs Topic

Dear Canary

Given the distinct structure of JSON messages (in contrast with spark message structures), it would be beneficial to provide users with the option to filter based on the content of the message rather than the topic alone.

To be more specific, I suggest adding an option that allows users to choose between filtering by the 'Topic' or the actual 'JSON Message' when constructing regular expressions. This flexibility can greatly enhance the granularity and precision of our searches, especially when dealing with diverse data structures.

To provide a clearer understanding, here's a Python example to illustrate

python:

import re

message json_string = '''{

"BatteryVoltage": 3558,

"Temperature": 20.94,

"NodeStatus": 5,

"Type": "scalar",

"SensorNodeId": "cf5f80bf",

"BatteryAlert": true,

"GatewayId": "aa423781",

"SourceAddress": "12550238",

"Timestamp": 1696519680

}'''

match = re.search(r'\bBatteryVoltage\b', json_string)

print(True if match else False)

In this example, the idea is to filter based on the content within the JSON message, specifically searching for the term 'BatteryVoltage'.

An essential point: It would be beneficial if the same subscription could process multiple filters. This capability would enhance the platform's versatility and help streamline configurations.

To give you a better perspective, consider the proposed configuration structure below:

Connection Groups:
Comal WD   
  Server:
        tcp://datagen.canarylabs.online:8083/
  Subscriptions:
        jsonMessage:
            Filters1 -> Then do 1
        jsonMessage:
            Filters2 -> Then do 2

In this structure, under the Comal WD connection group, there's the ability to set up multiple filters within a single subscription (in this case, jsonMessage). Each filter then triggers a specific action, such as "Then do 1" for Filters1.

I believe that having this kind of multi-filter support within a single subscription would greatly enhance our filtering capabilities and allow for more dynamic and responsive data processing.

Thank you once again for considering my suggestions. I'm confident that these enhancements will further solidify the canary MQTT Collector.

Best regards,

Juan de Dios