ClimateDataCalculation

[krambriw]

Just an interesting lesson learned from this whole project: when running heavy python scripts for whatever reason, run them in a thread, then EG doesn't lock up

[Mastiff]

Brilliant!

Edit: That was for finding the bug. As for the lesson I agree, this seems to be the solution. It's so annoying to do work in EG and having to wait a few seconds every 20 seconds. Not to mention that when my wife and kids (and parents, for that matter) press any button, they expect to have a reaction at once, not 5 seconds later!

[krambriw]

they expect to have a reaction at once

Same here, I have some anrdroid tablets around and when tapping with the finger on a button, you expect reaction at once. And this is working fine now even if data is being captured or the report is created!!!

[Mastiff]

Well, I still have a sort of lockup on the creation of reports in my system. If you didn't see that higher up, it stops on the first save after the report creation has started when there's a capture. So if I double click an action in a macro then, it will not open before the reports are generated. And nothing else happens either before the generation is done. Without a capture this doesn't happen. I just tried now to do somethig completely different (turn off the fan in the batroom), and I don't see anything in the logger before the reports are generated. But when that's done, the fan turns itself off from the button that was pressed a minute before. Is this because I have it running on a VM? It would be nice not to have this slowdown every hour. For now I have split the creation, so the first report (outdoors and the attic) is fired every hour. 45 seconds later it fires the indoors report, and one minute after that the ftp upload is fired. Still would be nice to avoid this lockup all together.

Edit: Maybe the soloution would be to ignore capture while reports are created?

[Mastiff]

Come to think of it that could be fixed with the events generated by report generating. One event saying that the report generation starts, which disables the folder with capture, and one when the reports are done, that enables them again. So for me it's not a big problem, I guess. But it would be nice to know the reason for this.

Edit: Can it be that generating the reports and capturing data uses the same thread, and that thread stops the rest of EG?

[Mastiff]

tested and working: When the first report macro starts, I disable the folder with all the creation, and when the ftp upload starts I enable it again. Which means that there is no lockup, only a slight slowdown on reactin when I click on stuff (a second at the most).

[krambriw]

Krambriw, when I change from 30 to 14 days, the period in the text doesn't change. The dates are the same, but the lines are flat from day 15 and out. So there's something very funky going on there.

I made a modification in the CreateComboHtml.py file so now I cannot reproduce the problem any longer. Maybe this helped? New version is uploaded.

If you didn't see that higher up, it stops on the first save after the report creation has started when there's a capture

Yes, I see that, the icon gets red (or yellow) maybe one second in my case. I will check if I can find something there. Reason for this is that the db is locked by the saving routine

Can it be that generating the reports and capturing data uses the same thread, and that thread stops the rest of EG?

No they are all having their own threads

Is this because I have it running on a VM?

No, I see it without using VM

Maybe the soloution would be to ignore capture while reports are created?

Capturing is put on hold but the db locking still has an impact the reporting so that it also blocks EG

I disable the folder with all the creation

Have you put all creations in one macro or in one folder? If you put them in the same macro, all will start to try accessing the same db's. Creation has to be serialized to be efficient

For now I have split the creation, so the first report (outdoors and the attic) is fired every hour. 45 seconds later it fires the indoors report, and one minute after that

I hope you are not using 'Wait' action for this since it stops the whole EG during the wait time

[krambriw]

I believe I found the reason for the blocking when ongoing capturing/reporting...one line of code was placed wrongly, after correction It seems to work without blockings

New version uploaded

Left 'to-do' is now a number of rain specifics, those are still 'as-is' and not using threads

Do you think we still need that separate rain report now that we have the combo?

[Mastiff]

Sorry I went dark yesterday, it was a long day of driving. First to get home, and then to drive to a place where they havethe special ceiling that I'm going to use in the main room at the cabin, so when I finally got home close to 10 P.M. I wasn't really fit for thinking! Thanks for the event, btw!

Most of these questions are now moot, since it keeps on ticking without any slowdowns even when doing the reports and capturing data at the same time! So it seems like that line of code made all the difference! Great work! I didin't use the wait, I'm kind of used to time(Sleep) in LUA, which has the same effect. I just had a trigger event with a 45 seconds delay on the first and one minute on the second, so in reality a timer.

As for creation macros I have them in one folder, not one macro. So that wasn't the problem. I'm dumb with code, but not that dumb.

For my part the combo should take care of everything I need, so no need to keep special reports for anything else. But how about wind? There is no action to capture wind, is that something that you could add?

Finally, the fix for the reports work, thanks again! No problems now with different date ranges!

So I bow to you, the master of the climate! I now have only one problem left for me (I think...), and that's the sensor called "Loftet". It has lots of data, and it shows up in the combo report creation, but I still get a flatline in the graphs. Should I mail you the database so you can have a look at it?

[Mastiff]

Btw here's a sample of that "Loftet" sensor's data:

Code: Select all

08:49:49      (u'2015-04-30 08:19:21', 1430374762, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:20:09', 1430374810, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:20:57', 1430374858, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:21:45', 1430374906, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:22:33', 1430374954, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:23:22', 1430375002, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:24:09', 1430375050, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:24:57', 1430375098, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:25:45', 1430375146, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:26:33', 1430375194, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:27:21', 1430375242, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:28:09', 1430375290, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:29:45', 1430375386, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:30:33', 1430375434, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:31:22', 1430375483, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:32:09', 1430375530, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:32:57', 1430375578, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:34:33', 1430375674, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:36:09', 1430375770, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:37:51', 1430375872, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:38:33', 1430375914, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:39:21', 1430375962, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:40:09', 1430376010, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:40:57', 1430376058, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:41:45', 1430376106, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:42:33', 1430376154, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:43:21', 1430376202, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)
08:49:49      (u'2015-04-30 08:44:09', 1430376250, u'Loftet', 0.0, u'', 10.0, 10.0, 0.0)

[Mastiff]

Btw (and now we're talking cosmetics and even cosmetic details, since the big stuff seems to be fixed, I have been running the latest version since I posted last, and there's no problems so far): Shouldn't the temp in the graph go to the very start and the very end of the graph lines? They start a bit after and stop a bit before the two ends, and I think ti would look better without that thing. Can this be a margin setting?

[krambriw]

Sorry I went dark yesterday, it was a long day of driving

I fully understand!!! No worries

There is no action to capture wind, is that something that you could add?

It is not so difficult, is just that I do not have one so I'm missing event samples.
- are we talking about capturing via RFXtrx or Duo?
- any ideas how to present data? (wind strength I guess could be a curve but direction is the problem)

So I bow to you, the master of the climate!

Thank you, you have been very helpful too and provided a lot of good advises and ideas

I now have only one problem left for me (I think...), and that's the sensor called "Loftet". It has lots of data, and it shows up in the combo report creation, but I still get a flatline in the graphs. Should I mail you the database so you can have a look at it?

Yes please, mail it to me. When I look at your sample data, I think it is presented correctly in the report, all data points are zero (0.0) so flatline is ok. Unless the rest of the data has another value. Or the capturing is not capturing the sensor data correctly. Is this a different sensor? Please provide a copy of a typical event from this sensor and I will check that we are supporting the format correctly

Can this be a margin setting?

Could very well be the case, need to study HC api

[krambriw]

HighCharts has a nice graph called Wind Rose

[Mastiff]

Thanks! I'm crap at coding, but I consider myself a rather good bug hunter!

I do wind with the RFX. Here's the format:

Code: Select all

10:21:34   RFXtrx.Type: WGR800 id: 28928 ' direction: SSE average speed: 1.9 m/s gust: 1.9 m/s signal: 6 battery: 10%'

I agree it's hard to think how to show the direction, but the average speed and gusts would really be the most interesting thing. But maybe in two different types of graphs. The average speed would be best represented in the same way as temps, but the gusts could be a column diagram (if that's what it's called - stolpediagram is the Norwegian word). Maybe the direction could be shown in a way where it's at least possible to se what direction is prevalent?

The database is being packed, and I'll mail it to you. I can't understand the 0.0 data, it's more like this:

Code: Select all

10:27:21   RFXtrx.Type: Viking 02811 id: 41216 ' temperature: +10.7 deg C signal: 7 battery: 9'

So if I was to guess I'd say that the 10.0 is the actual temp data there.

[Mastiff]

I agree, that wind rose was nice!