Page 13 of 37
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 9:37 am
by Mastiff
Well, yes and no. I don't see any erorrs this time, but EG uses 30-40 % CPU, up from about nothing without the threading of catpures. And if I try to move up and down in the log, it sort of pauses a second or two and then the text comes just like if it's scrolling auomatically. Not like when I'm usually paging up and down in it.
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 9:49 am
by Mastiff
And the lockup has happened.

Back to the init without threading.
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:22 am
by krambriw
Sorry for that m8, this is tricky since I cannot reproduce the problem, I have only a few sensors
Even when I tried to save the data from one sensor to 10 different tables , it still works fine in my system (basically running 10 threads simultaneously)
I think it could be that the code I changed has a while loop without any wait period and when it is retrying, and you have several such waiting, it could cause the cpu load. The new uploaded version has a small delay included in the while loop, could you check if it behaves better?
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:30 am
by krambriw
Finally got the error as well
In my system it works fine, it tries again and finally manage to save
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:33 am
by Mastiff
No problem, I'll be here as long as you need me, new versions only take a minute or two for me. This one doesn't give me the "Saving temp data successfully", should it do that? I have 20-40 % CPU for EG as well. Does that mean that this is not the latest version? ALso I get:
Code: Select all
13:32:26 ClimateDataCalculation: TempDataCapture: Yttergangen
13:32:31 Error database is locked:
13:32:36 Error database is locked:
But this is a printout, it's not a red error message.
Edit: This is the last revision info:
2015-04-23 Introduced retries when committing data to data bases.
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:38 am
by Mastiff
Wow, talk about retries! I deactivatede evrything and still had 20-40 % on EG, and then after a few minutes I saw one "error database is locked" and then finally saw two "Saving temp data successfully". So it's really struggling here!
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:41 am
by krambriw
Sorry missed to change the revision info...
The file date and time is 2015-04-24 13:13
Its correct, you should only get normal printouts for the moment (like I got)
If you don't get prints when saved ok, it is failing (what is happening...)
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:43 am
by krambriw
Try to increase the wait a bit
Find line 528, change the wait time from 0.1 to 0.5
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:44 am
by krambriw
Are you eventually starting to generate the report too fast?
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:47 am
by Mastiff
A bit, but not much help, I'm afraid. But I saw something I haven't noticed before. There seem to be a memory leak, because the memory useage keeps growing steadily. Closing in on 200 000 kb now. I think that's what kills it in the end. Also I rarely see the successful saving to database.
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 11:47 am
by Mastiff
I have the first report 60 seconds after startup, and then every hour.
Btw over 200 000 kb now, and still rising in the same speed.
Edit: Checked EG at home (very similar, except for the changes in this plugin), and that stays around 50 000 kb.
Edit 2: almost 400 000 kb now.
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 3:18 pm
by krambriw
Yeah, it is a mess right now... I have to find the leak
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 3:21 pm
by Mastiff
You could ask Säpo, they are pretty good at finding leaks!

Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 5:45 pm
by krambriw
You are maybe right, but only of they have a leak in their pants
Anyway, regarding the memory (leak!) I think we have a situation where there are so many threads in the queue that they never finish and new ones are created, therefore the memory consumption goes up. I base this on the fact that you rarely see the saved successfully message printed. I assume you have your data bases ion the local hard drive?
The system I am running has no problem with saving and memory usage is very stable around 69 Mb with fluctuations during save and report creation. CPU is normally around 0 but raises shortly up to 20-30 % when saving/creating report
Re: ClimateDataCalculation
Posted: Fri Apr 24, 2015 5:49 pm
by Mastiff
Well, just tell them they're looking for a submarine. Oh no, wait. They can't find that either!
I have the database on the C drive of the virtual machine, yeah. I don't know if that can be considered a local drive. But I'm guessing the theory may be right. Does that mean that it's probably not possible to thread this on a system with much traffic?