Notice: This forum has been recovered from an old backup, so some content, links, and dates may be outdated. The forum is currently read-only while we restore sign-in and registration functionality. Details

If you find this forum valuable and would like to help keep it online, donations to help cover hosting and domain costs are greatly appreciated, but never expected. You can support the forum through Buy Me a Coffee or Ko-fi. Thank you for helping preserve the EventGhost community.

ClimateDataCalculation

Questions and comments specific to a particular plugin should go here.
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

The error was my mistake, sorry for that :oops:

I think your suggestion is fine, that should work as well, provided that the names can be unique even if you would have several sensors of the same kind.

Some questions to clarify:
- the actual 'Name', like 'Sensor.Luminance', 'Sensor.Temperature' and 'Status.Level', are they user configurable? I mean can you assign different names to them if you have several sensors of the same type?
- what kind of z-wave controller do you have (vera, razberry...)?
sintei
Posts: 26
Joined: Thu Oct 09, 2014 9:46 am

Re: ClimateDataCalculation

Post by sintei »

krambriw wrote:The error was my mistake, sorry for that :oops:

I think your suggestion is fine, that should work as well, provided that the names can be unique even if you would have several sensors of the same kind.

Some questions to clarify:
- the actual 'Name', like 'Sensor.Luminance', 'Sensor.Temperature' and 'Status.Level', are they user configurable? I mean can you assign different names to them if you have several sensors of the same type?
- what kind of z-wave controller do you have (vera, razberry...)?
Yes, I think I need to clarify it to myself as well :)

I use HomeGenie as base with USB z-wave stick.
From HomeGenie, MqTT messages are sent to Eventghost.
Message comes like this:
19:58:12 Main.SilverHG/HomeAutomation.ZWave/2/event '0 {"Name":"Sensor.Luminance","Value":"11","Description":"","FieldType":"","UpdateTime":"2015-07-30T17:58:12.6733142Z","NeedsUpdate":false}'

Lets break it down!
First part: Main.SilverHG/HomeAutomation.ZWave/2/event
"Main" is normal EG.
"SilverHG/HomeAutomation.Zwave/2/" is the topic MqTT is sent on.
Here we have a unique identifier which is the "2"!
Other different sensors will have a 1, 3, 4 etc depending in what order they are added to the z-wave network.
In EG the events are unique

Then there is Payload:
'0 {"Name":"Sensor.Luminance","Value":"11","Description":"","FieldType":"","UpdateTime":"2015-07-30T17:58:12.6733142Z","NeedsUpdate":false}'
Sensors can have multiple functions, for example Sensor.Temperature, Sensor.Luminance, Sensor.Alarm, Sensor.Battery etc.
They are unique to the "2"
They are NOT configurable as they are standard for ZWave attribute.

What we type in dialog box "unique ID", I think you use the "Unique ID" in your database?
Maybe best would be if the Table Name is what is unique as you can set it for yourself. And then "unique id" is what it is supposed to search for in payload.
Shouldn't this work for all sensors, including oregon etc that you incorporated before?
Or am I just way off...?

/Nicklas
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

I see, they use a dedicated topic for each sensor, that's fine for me. This means the unique id is either the full string "SilverHG/HomeAutomation.Zwave/2/" or a part of it. I would then suggest to use the last part like '/2/' to keep it 'safe to neighbors' like '/12/' and '/21/', '/22/' etc

This means that when you type in the id in the configuration dialog, you have to type them like that, I guess that should not be a problem?
sintei
Posts: 26
Joined: Thu Oct 09, 2014 9:46 am

Re: ClimateDataCalculation

Post by sintei »

Not a problem!
And it is very good for scalability.
I am in process of setting up arduino sensors as well.
They will also communicate in same way I think.

Edit:
But if unique ID is /2/. How will it know to look for sensor.temperature? Or luminance?
Same "2" can have several different payloads. Lim. Temp. Alarm etc
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

Well, first condition to test is for sensor or device id like '/2/'. If a match, next is to check the payload to detect if Luminance or Temperature. If a match, then the value will be stored in db. Otherwise, discarded. Means that other payloads will not be handled by the plugin (until they eventually becomes supported). You can anyway use those events for opening/closing doors and similar for other purposes but they might not belong in a Climate Data chart. As example, if the sensor also provides data readings for humidity or rain, such things would then be easy and possible to support
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

OK, could you please try the new modified version I have published?

EDIT:
Forgot to mention, you just have to type in the actual number as device or sensor id

Like this example:
Skärmklipp.PNG
sintei
Posts: 26
Joined: Thu Oct 09, 2014 9:46 am

Re: ClimateDataCalculation

Post by sintei »

Now it prints the Luminance and Temperature.
But it also prints the other variables like Sensor.Alarm.
17:47:56 Main.SilverHG/HomeAutomation.ZWave/2/event '0 {"Name":"Sensor.Luminance","Value":"2790","Description":"","FieldType":"","UpdateTime":"2015-08-05T15:47:56.4506578Z","NeedsUpdate":false}'
17:47:56 ClimateDataCalculation: LightDataCapture: Light level roof
17:47:56 ClimateDataCalculation: LightDataCapture: Fibaro1SensorLuminance
17:47:56 ClimateDataCalculation: TempDataCapture: My zwave device (temp)
17:47:56 ClimateDataCalculation: TempDataCapture: Fibaro1SensorTemperature
17:47:56 lightlevel: 2790.0
17:47:56 temperature: 2790.0

Also, the values are not inserted in database as "lightData.db" is empty and there is nothing to choose in tablenames to create HTML output.

I will look into it further later, I have to be nice to the missus at the moment :)

EDIT: I looked at the code and there is no separation of lumimance or temperature, just by ID. Hence all values will be loaded into both "lightlevel" and "temperature".
It finds the devicecode good as I made a print A and B to see what was found :)

Maybe change in row 2047:
if B == deviceCode and A[3] == 'Sensor.Luminance':

And in row 1527
if B == deviceCode and A[3] == 'Sensor.Temperature':

This should separate!
Also, I am dumb.. this "version" was printing only, not logging. I must read more carefully what people write.. :lol:
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

You are correct, I just took it stepwise to see that all variables and id's was correctly parsed, otherwise I felt I could lose track since I do not have any multi-sensor for testing...

Anyway, a new version is uploaded, it should also do logging to db (printing is still there but can be commented out once we feel it is working ok)
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

Just some words about the code used to create the charts.

When you start to create a (combo) chart using the 'luminance' data, you may want to modify the python code first to fit your specific needs. The code is in the CreateComboHtml.py file. Since I am using another type of luminance sensor (a 1-wire sensor) that gives me other values, I am doing some modifications to it's value.

In line 460 (and 578) the following code can be modified or commented out:

Code: Select all

                    rd = rd*10.0 + 60.0
rd is your actual value and in my case I wanted to multiply it with 10 and also add a base value of 60

The next is line 518 (and 614) where I add some static reference levels to the chart:

Code: Select all

            for i in range (0, lgth):
                data.append(79.0)
            series['name'] = 'UpLevel'
            series['data'] = data
            series_list.append(series)
            data = []
            series = {}
            for i in range (0, lgth):
                data.append(81.0)
            series['name'] = 'DownLevel'
            series['data'] = data
This part you can modify as well, either keep it and adjust to levels your sensor provides or simply remove
sintei
Posts: 26
Joined: Thu Oct 09, 2014 9:46 am

Re: ClimateDataCalculation

Post by sintei »

Hello!

I was struck by a virus called "vacation" and was not able to test everything until now :)
It's been running fine on my computer for these past 3 weeks!
There are some questions I have:
In combined reports:
What is "use 10 min data point spread"?
Also, what is "number of additional hours"?

And in the plugin when you select the sensor:
I do not fully understand all the settings there.
Is "desired target" something that is used if "use rule" is not checked?
Hysteresis, is this used when plotting?

Why I ask is that sometimes my html pages does not plot everything always. But the plots are there if you hoover over them.
I have not changed anything in HTML.
It's reproducible when making logging intervals 30 minutes or say 60 minutes.
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

It's been running fine on my computer for these past 3 weeks!
Great !
What is "use 10 min data point spread"?
Also, what is "number of additional hours"?
'Use 10...' means that the chart is only picking a data point if time differs more than ten minutes from previous data point. This report variant is maybe only relevant if you are sampling data every minute but are happy with a report showing data with ten minutes resolution (to improve performance when generating the graph...)

'...additional hours?' means that you can add a number of additional hours to be covered by the report (as example, 2 days and 12 hours)
And in the plugin when you select the sensor:
I do not fully understand all the settings there.
Is "desired target" something that is used if "use rule" is not checked?
Hysteresis, is this used when plotting?
Yes, desired value is for example the temperature you would like to monitor. The hysteresis sets the upper and lower value where you would like to receive events when the moving average of actual temperature either goes above or below. Lets assume you set a desired temperature to 22 degrees and a hysteresis of 2 degrees. If you activate the rule you will get events when the moving average value passes 18, 22 and 24 degrees.
Desired target values, moving average values and hysteresis are only used for calculations, none is used (so far) in the graphs.
Why I ask is that sometimes my html pages does not plot everything always. But the plots are there if you hoover over them.
I have not changed anything in HTML.
It's reproducible when making logging intervals 30 minutes or say 60 minutes.
Maybe you could show this with a picture?
sintei
Posts: 26
Joined: Thu Oct 09, 2014 9:46 am

Re: ClimateDataCalculation

Post by sintei »

krambriw wrote:
Why I ask is that sometimes my html pages does not plot everything always. But the plots are there if you hoover over them.
I have not changed anything in HTML.
It's reproducible when making logging intervals 30 minutes or say 60 minutes.
Maybe you could show this with a picture?
Please see attachment.
When collecting for 1 min intervalls its OK, when doing it for 30 min or 60 min just the little points appear.
Attachments
10.PNG
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

I see and I think I know what is the problem, the python code to generate the chart is considering this as a gap in data, therefore this happen,,,when I wrote it I was working with 1 minute logging ;) The code to generate the graph is in CreateComboHtml.py and would need to be modified to handle this differently

BR
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

If you would like to try, you could modify the code in the CreateComboHtml.py file. In the current implementation I insert a 'null' when an expected data point is missing. This will then be presented as a gap (or broken line) in the graph, this is how HighCharts is handling this and I think it is good in general.

In this case where you are sampling only once per hour, it will however not look so good. The idea here is to fill the gaps with last known value instead of the 'null'

If you look at the code (the part below starting from line 188) you see the part that is looping through your data points and appending found values to the data that is going to be used in the graph. If a data point is expected but missing, a 'null' is inserted.

Code: Select all

                for row in rows:
                    rd = float("%.2f" % row[pos])
                    if bFirstDone:
                        diff, d = days_hours_minutes(
                            datetime.strptime(str(row[0][:-3]), '%Y-%m-%d %H:%M') -
                            datetime.strptime(str(row_old[0][:-3]), '%Y-%m-%d %H:%M')
                        )
                        if diff > 0:
                            for i in range(diff-1):
                               data.append('null')
                            diff = 0
                            if rd == 0.0:
                                data.append('null')
                            else:
                                data.append(rd)
                    else:
                        s, d = days_hours_minutes(
                            d1 -
                            datetime.strptime(str(rows[0][0][:-3]), '%Y-%m-%d %H:%M')
                        )
                        if sedt[0] - s > 0:
                            for i in range(sedt[0] - s):
                                data.append('null') 
                        data.append(rd)
                        bFirstDone = True
                    row_old = row


If we instead would insert the last known value, the code could look like this:

Code: Select all

                for row in rows:
                    rd = float("%.2f" % row[pos])
                    if bFirstDone:
                        rd_old = float("%.2f" % row_old[pos])
                        diff, d = days_hours_minutes(
                            datetime.strptime(str(row[0][:-3]), '%Y-%m-%d %H:%M') -
                            datetime.strptime(str(row_old[0][:-3]), '%Y-%m-%d %H:%M')
                        )
                        if diff > 0:
                            for i in range(diff-1):
                               data.append(rd_old)
                            diff = 0
                            if rd == 0.0:
                                data.append('null')
                            else:
                                data.append(rd)
                    else:
                        s, d = days_hours_minutes(
                            d1 -
                            datetime.strptime(str(rows[0][0][:-3]), '%Y-%m-%d %H:%M')
                        )
                        if sedt[0] - s > 0:
                            for i in range(sedt[0] - s):
                                data.append('null') 
                        data.append(rd)
                        bFirstDone = True
                    row_old = row

If you change, please note that this modification needs be repeated for all chart types you would like to behave like this (MakeChart. MakeChart_10 etc...)

BR
krambriw
Plugin Developer
Posts: 2570
Joined: Sat Jun 30, 2007 2:51 pm
Location: Stockholm, Sweden
Contact:

Re: ClimateDataCalculation

Post by krambriw »

Another option, if you can live with that, would be to set the logging interval to 10 minutes (or 9 minutes) and use the 10 minute reports
Post Reply