ClimateDataCalculation

[krambriw]

Hello,
So I made a new version introducing a queue for simultaneously running threads. It has been running several hours here without problems.

This version allows five threads running at the same time (this is a limitation introduced only for the plugin). Upcoming requests for new running threads will be queued and executed as soon as the total number in the queue is below the limit.

Let's see if this helps. Eventually and if needed, the limit can easily be raised. Just go to line 185 and change the value (default is 4 that will allow 5 concurrent threads)

Code: Select all

        self.qsLimit = 4

[Mastiff]

Done, thanks! I'll let you know what happens.

[Mastiff]

I can not say for sure that it is the reason, but EG locked up an hour or so after this was done yesterday, and this morning there was 36 degrees in the small bathroom! What I can say is that this has never happened before, so the immedeate instinct is that this change has done something that made it lock up. I have restarted EG now and we'll see what happens.

Edit: Locked up in a way I haven't seen before either: No processing of events or anything, but no error message, and the UI was still working. And there was nothing in the log, it only stopped getting anything.

[krambriw]

Ooops, not good

Could eventually be that resources are being tied up due to piled up requests. What I can imagine is the that the queue is growing out of boundary somehow. Let's assume an event is incoming, data being saved to db, report creation starts and this takes rather long time. Before the report is finished, an event from the same sensor is coming in. The result is that the request will be queued. Imagine this happens for several sensors...over and over

Just speculation, so therefore we need to check a bit further

To check the actual size of the queue, you can issue the following python command in the python shell

Code: Select all

eg.plugins.ClimateDataCalculation.plugin.q.qsize()

or as a python script

Code: Select all

print eg.plugins.ClimateDataCalculation.plugin.q.qsize()

If you notice the queue size not getting reduced when you run the script multiple times, I assume the whole thing will get saturated sooner or later. We could try to allow more threads by raising the upper limit (maybe set to 19 allowing 20 threads) and see if this works better.


1) How long time does it take to create your reports?
In my systems it takes
- a few seconds to generate a 10 min report covering the last 36 hours (system with SSD)
- around 15 seconds to generate a 10 min report covering the last 24 hours (system with normal HDD)

2) Do you log data every minute?

3) How long period do you cover when creating reports?

4) How many different reports are you creating?

[krambriw]

5) Do have many reports covering many tables?

6) How long time takes the most complex report to generate?

[Mastiff]

I had to go back to the previous version for now because it locked up again in the same way. Sorry, Walter. I'm at the cabin until next Sunday, and my wife drives home again this Sunday. I need to have EG running the house when I'm away, or my wife will probably kill me. So I will try to bugtest when I get back next Monday.

[krambriw]

Of course, fully understand!

In the mean time I will try to configure my system in such a way that I hopefully can reproduce the problem (so far I have not been able to). One idea I have is to save the same sensor data to multiple tables and generating multiple number of reports as a stress test

[Mastiff]

Well, I save them to only one table each, but I do save on every signal, I believe. Maybe that's why I hit some limit you didn't?

[krambriw]

I managed to recreate the problem!

I copied the macro a couple of times. One sensor event is now being saved to 10 tables, then a report is created covering all 10 tables. Everything as fast as possible. I could see that events stopped coming in.

I then made a modification of the queue handling and now it runs smoothly and I can see that threads are being queued (running with 5 threads as a limit configured in line 185). But for sure, it is possible to load the system in such a way that it gets busy. Each sensor event is coming in roughly once every 45 seconds and with a number of threads running and saving to db and some more threads waiting to start, the whole thing starts all over again.

To be on a safer side, when logging from many sensors, the logging interval should maybe be reduced. Faster hardware could also eventually help (like having a SSD).

Anyway, good news so far is that events are still coming in even though the system is pretty busy.

I suggest I let it run here in one of my non-critical systems for a while before you put it into production

If you have another system to try with, the new version is anyway uploaded now. For temperature sensors there are some additional debug printouts that I plan to remove later when we hopefully see that things are working ok.

[krambriw]

I'm a believer, I think it is looking alright now. The latest version seems to do it's work fine. The queue of threads builds up but after a while it is empty.

Starting at line 186 there are some settings that you can modify if you prefer

Code: Select all

        self.qsLimit = 15  #Will allow 15 concurrent threads
        self.q = Queue(maxsize=self.qsLimit) #Leave as is
        self.qDebug = True #Will enable debugging of the queue size, set to False to disable

Best regards, Walter

[Mastiff]

Thanks, Walter! Great work! I will try that tomorrow, when my wife goes home again, and she can keep an eye on stuff. As long as she's warned it is a test she doesn't get too mad by stuff not working!

[krambriw]

Yes, we have to be very careful, we do not want our wifes angry, right, agree
I hope this version works good but we also need to verify that your system is not getting overloaded
Have a nice day!

[Mastiff]

Thanks, it looks good so far, after a few hours in use. No complaints from the wife yet!

[Mastiff]

Still loking good here!

[krambriw]

Great to hear,

Have you noticed how the qsize develops? Is it getting larger & larger or is it stable and getting reduced down to zero when the waiting threads are finishing?