ClimateDataCalculation

[krambriw]

New version uploaded

[Mastiff]

Good morning! I'll wait until you have something on this setup before I do a new trialrun, ten.

[Mastiff]

Sorry, didn't see that last message... I'm downloading now and putting it to test in my VM setup.

[Mastiff]

I see no slwodown or memory leak here! This may be working well! I don't see any slowdowns or hysterical CPU usage. "Database locked" appeared a few seconds after start, but not since, so far. But I don't get any of those messages telling that the capture is finished. Did you remove those?

[Mastiff]

Still no problems, memory is under control (actually down from the versions without this optimization - from around 150 000 to 65 000), and the CPU is usually around 0 or single digit, with spikes to around 50 on temp data capture, but that only last a second or two. This looks very promising!

[krambriw]

Did you remove those?

Yes, there where so many...Since we do not run via threads it is the finish of the macro that gives an idea of time consumed (I actually added a simple 'TriggerEvent' action with text 'Done' as last in my macros)

Sounds good for the rest !!!

[Mastiff]

Understood! And yeah, still running nicely here. The memory has risen to 76 000, though. I don't know if there can be a very slow leak still. I'll check in a few hours.

[Mastiff]

Still humming along. Memory varies between around 77 000 and just over 80 000 kb. I do see "Database is locked" at times but I know now when that happens: It's when I'm running the reports and the sensors keep coming in. But maybe this happened before as well, only then it didn't report it?

[krambriw]

Most likely yes,
It's good that memory consumption has stabilized. The thing with python is that it has it's own garbage collection (freeing up memory automatically) and this happens when there is not reference left to a variable etc.

So I have spent sometime adding a number of 'del' statements just to get ride of any unwanted references. Also experimented with settings to the sqlite3 to see if performance can be boosted further

BTW I learned something new today with the highcharts: if you have many lines in the chart and the one you want to study is hided, just click on the legend on the right side to toggle the others on/off

[Mastiff]

Performance is at least a lot better, I can see that when I press a button the action happens much faster even if there's a capture going on (which there usually is, with all those sensors...). And the highest memory consumption is (just like CPU) when doing those reports. So once an hour I can take that!

As for the highcharts trick, that was very cool, thanks! Not intuitive, but very cool once I know about it! I wish there was a "click on a symbol to turn line on and off" there (especially in Norwegian!), but I'm guessing that's too far down in the Highcharts stuff to bother with. I'll put that in my user manual for the system.

[krambriw]

I made some additional optimizations, uploaded a new version

- In my case saving is now even faster than previous version
- cleaning up code and clearing variables to avoid unwanted memory consumption
- if you make changes to the CreateComboHtml.py file, you need to restart EG (it is one line, 1950, that you can un-comment if you want the reload activated again)

I do see "Database is locked" at times

How is this printed, is it printed with red text or just as a normal print statement?

[Mastiff]

Thanks, I'm downloading the new version to test that. The "database locked" is shown in both ways, both as a printed statment and a red error message. Btw it seems the previous version has fixed something I haven't remembered to mention: After exciting EG it has taken up to a minute before it's actually gone (the process and the icon in the task bar).

Edit: I have installed the new version and I just got a locked message (welll, both types again) on the first capture after startup. Also another one, but this is while the threaded reports are generated (it's always the first capture after the report action started, the next works).

[Mastiff]

Still running well, and I see in the file log (from log redirector) that the "database locked" are coming in just about every hour, when generating the reports. Which is probably to be expected with large reports. So no problems, then!

[krambriw]

Good news!

Here is also everything working fine, I have 2 systems running, one for the house automation, and the other one for stress testing. It is saving the 2 temp sensors to 10 different tables each every time the events come in and then when finished (takes around 5-6 seconds) it creates the report (so the report is actually created around every 45 seconds). Memory consumption is stable as well.

I found a minor bug since I changed the sorting order yesterday but it is only cosmetic, new version is also uploaded
# 2015-04-27 Bug fix in chart for 'selected period' presentation

[Mastiff]

Well, I didn't notice... Anyway, I've updated. So far nothing's dangerous is happening.