ClimateDataCalculation

[Mastiff]

Well, yes and no. I don't see any erorrs this time, but EG uses 30-40 % CPU, up from about nothing without the threading of catpures. And if I try to move up and down in the log, it sort of pauses a second or two and then the text comes just like if it's scrolling auomatically. Not like when I'm usually paging up and down in it.

[Mastiff]

And the lockup has happened. Back to the init without threading.

[krambriw]

Sorry for that m8, this is tricky since I cannot reproduce the problem, I have only a few sensors

Even when I tried to save the data from one sensor to 10 different tables , it still works fine in my system (basically running 10 threads simultaneously)

I think it could be that the code I changed has a while loop without any wait period and when it is retrying, and you have several such waiting, it could cause the cpu load. The new uploaded version has a small delay included in the while loop, could you check if it behaves better?

[krambriw]

Finally got the error as well

In my system it works fine, it tries again and finally manage to save

[Mastiff]

No problem, I'll be here as long as you need me, new versions only take a minute or two for me. This one doesn't give me the "Saving temp data successfully", should it do that? I have 20-40 % CPU for EG as well. Does that mean that this is not the latest version? ALso I get:

Code: Select all

13:32:26         ClimateDataCalculation: TempDataCapture: Yttergangen
13:32:31   Error database is locked:
13:32:36   Error database is locked:

But this is a printout, it's not a red error message.

Edit: This is the last revision info:

2015-04-23 Introduced retries when committing data to data bases.

[Mastiff]

Wow, talk about retries! I deactivatede evrything and still had 20-40 % on EG, and then after a few minutes I saw one "error database is locked" and then finally saw two "Saving temp data successfully". So it's really struggling here!

[krambriw]

Sorry missed to change the revision info...

The file date and time is 2015-04-24 13:13

Its correct, you should only get normal printouts for the moment (like I got)

If you don't get prints when saved ok, it is failing (what is happening...)

[krambriw]

Try to increase the wait a bit

Find line 528, change the wait time from 0.1 to 0.5

[krambriw]

Are you eventually starting to generate the report too fast?

[Mastiff]

A bit, but not much help, I'm afraid. But I saw something I haven't noticed before. There seem to be a memory leak, because the memory useage keeps growing steadily. Closing in on 200 000 kb now. I think that's what kills it in the end. Also I rarely see the successful saving to database.

[Mastiff]

I have the first report 60 seconds after startup, and then every hour.

Btw over 200 000 kb now, and still rising in the same speed.

Edit: Checked EG at home (very similar, except for the changes in this plugin), and that stays around 50 000 kb.

Edit 2: almost 400 000 kb now.

[krambriw]

Yeah, it is a mess right now... I have to find the leak

[Mastiff]

You could ask Säpo, they are pretty good at finding leaks!

[krambriw]

You are maybe right, but only of they have a leak in their pants

Anyway, regarding the memory (leak!) I think we have a situation where there are so many threads in the queue that they never finish and new ones are created, therefore the memory consumption goes up. I base this on the fact that you rarely see the saved successfully message printed. I assume you have your data bases ion the local hard drive?

The system I am running has no problem with saving and memory usage is very stable around 69 Mb with fluctuations during save and report creation. CPU is normally around 0 but raises shortly up to 20-30 % when saving/creating report

[Mastiff]

Well, just tell them they're looking for a submarine. Oh no, wait. They can't find that either!

I have the database on the C drive of the virtual machine, yeah. I don't know if that can be considered a local drive. But I'm guessing the theory may be right. Does that mean that it's probably not possible to thread this on a system with much traffic?