ClimateDataCalculation

[krambriw]

New version uploaded, all data capturing are handled via threads. It looks fine for me I think, I clearly see that EG is not blocked during data capture

Your last post: did you see a problem there with the latest version? First try this and then check.

[krambriw]

I did test to see what happens when I start creation of a report with many data points

From the log below it is seen that EG is not locked up anymore (I manually started the report creation on a nexa remote event, AC address: 004542b2 unit: 04 command: on ), events and other macros/actions are executed during the 10 seconds while the report is being created by the thread (I temporary added a simple print when the report is finished)

Code: Select all

13:43:10   RFXtrx.Type: AC address: 004542b2 unit: 04 command: on ' level: 0 signal: 6'
13:43:10      ClimateDataCalculation: CombinedReport: ()
13:43:10         ClimateDataCalculation: CombinedReport: EventGhost Combo Report
13:43:10         Python Script
13:43:10   MQTT./zwave u"Device_6_Instance_3_temperature, 21.68"
13:43:10      WebUpdaterSuite: WebCapture
13:43:10         WebUpdaterSuite: WebCapture
13:43:11   RFXtrx.Type: AC address: 004542b2 unit: 03 command: on ' level: 0 signal: 6'
13:43:11   MQTT./zwave u"Device_6_Instance_4_temperature, 23.18"
13:43:11      WebUpdaterSuite: WebCapture
13:43:11         WebUpdaterSuite: WebCapture
13:43:12   RFXtrx.Type: AC address: 004542b2 unit: 03 command: on ' level: 0 signal: 7'
13:43:13   RFXtrx.Type: AC address: 004542b2 unit: 03 command: on ' level: 0 signal: 6'
13:43:14   RFXtrx.Type: AC address: 004542b2 unit: 03 command: on ' level: 0 signal: 7'
13:43:15   RFXtrx.Type: AC address: 004542b2 unit: 03 command: on ' level: 0 signal: 6'
13:43:15   RFXtrx.Type: AC address: 004542b2 unit: 03 command: on ' level: 0 signal: 6'
13:43:17   RFXtrx.Type: AC address: 004542b2 unit: 02 command: on ' level: 0 signal: 7'
13:43:20   ComboReport finished
13:43:22   System.Idle

[Mastiff]

Great, thanks! Now it ticks on, even if I'm creating the reports! But I got an error. I think this is because the thread is locking up the database:

Code: Select all

14:05:21   RFXtrx.Type: PCR800 id: 25856 ' rainrate: 0 mm/hr rainTotal: 269 mm signal: 7 battery: 100%'
14:05:21   Exception in thread Thread-21:
14:05:21   Traceback (most recent call last):
14:05:21     File "threading.pyc", line 532, in __bootstrap_inner
14:05:21     File "threading.pyc", line 484, in run
14:05:21     File "C:\Program Files (x86)\EventGhost\plugins\ClimaDataCalculation\__init__.py", line 1149, in CaptureTemp
14:05:21       rule
14:05:21     File "C:\Program Files (x86)\EventGhost\plugins\ClimaDataCalculation\__init__.py", line 500, in SaveTempData
14:05:21       conn.commit()
14:05:21   OperationalError: database is locked
14:05:21   

[Mastiff]

It seems this happens on regular saves as well, not just when there's a report being generated.

[Mastiff]

Now I had to kill EG because it had locked up. When I restarted I got this error:

Code: Select all

14:54:35      Exception in thread AsyncoreThread:
14:54:35         Traceback (most recent call last):
14:54:35           File "threading.pyc", line 532, in __bootstrap_inner
14:54:35           File "threading.pyc", line 484, in run
14:54:35           File "asyncore.pyc", line 202, in loop
14:54:35           File "asyncore.pyc", line 132, in poll
14:54:35         error: (10038, 'Det ble fors\xf8kt en operasjon p\xe5 noe som ikke er en kontakt')
14:54:35         

This is not consistent, though. Next restart it wasn't there.

[krambriw]

I think this is because the thread is locking up the database

Yes, if one saving is on the way and the next starts before it has finished, this will happen...

How shall we handle this, a first idea is if it would be possible to check if the sql is open and if so, wait a while and try again. Having a global variable for each type of data base and use as a semaphore could be a workaround. Needs to be investigated a bit

[Mastiff]

Well, your guess is as good as mine. Yeah, right...giving myself way too much credit. But I like the que principle, so it waits and tries again. There may be a long line there since I have stuff coming in almost constantly from 14 temp sensors, one wind sensor, one rain sensor. And that's just at the cabin! At home I have more temp sensors. So if you could make sure it would work with a ridiculous amount of sensors that would be great!

Would it be possible to have a separate database for each sensor added? That would help so the database doesn't grow so large as well.

[krambriw]

New version uploaded where I introduced retries when committing data to data bases. This seems to work for me, but please test since you have many more sensors

Would it be possible to have a separate database for each sensor added?

Not easily (means hell of a lot of work) to change in current design and I wonder, the size of one big would be equal to the sum of a number of small, and for read & write, well instead of open one, you would have to open several

[Mastiff]

I'm afraid not. Started after four minutes, even had a double.

Code: Select all

08:10:01   Exception in thread Thread-134:
08:10:01   Traceback (most recent call last):
08:10:01     File "threading.pyc", line 532, in __bootstrap_inner
08:10:01     File "threading.pyc", line 484, in run
08:10:01     File "C:\Program Files (x86)\EventGhost\plugins\ClimaDataCalculation\__init__.py", line 1180, in CaptureTemp
08:10:01       rule
08:10:01     File "C:\Program Files (x86)\EventGhost\plugins\ClimaDataCalculation\__init__.py", line 473, in SaveTempData
08:10:01       c.execute(qs+prm)
08:10:01   OperationalError: database is locked
08:10:01   
08:10:03   Exception in thread Thread-136:
08:10:03   Traceback (most recent call last):
08:10:03     File "threading.pyc", line 532, in __bootstrap_inner
08:10:03     File "threading.pyc", line 484, in run
08:10:03     File "C:\Program Files (x86)\EventGhost\plugins\ClimaDataCalculation\__init__.py", line 1180, in CaptureTemp
08:10:03       rule
08:10:03     File "C:\Program Files (x86)\EventGhost\plugins\ClimaDataCalculation\__init__.py", line 473, in SaveTempData
08:10:03       c.execute(qs+prm)
08:10:03   OperationalError: database is locked
08:10:03   

As for the second question, like I always say I have no clue. I just shoot ideas that sound smart when they're coming out. Not always as smart when they come back!

[Mastiff]

It doesn't seem to lock up EG this time, but the errors keep flowing.

[krambriw]

I do not recognize this:

Code: Select all

c.execute(qs+prm)

The structure should be

Code: Select all

c.execute(ss,lst)

Have you modified the code???

[Mastiff]

Nope. Downloaded and overwritten what was there. I only modify code if I know exactly what line and characters I should modify, and in those instances I have asked you first!

[krambriw]

OK, thanks, just wanted to align with you

I think I have to improve the complete sqlite3 handling with error detection in the save routines...still with retries if it fails. Maybe in the first new version, I will add a print statement when things goes wrong, hopefully we will catch the problem

[Mastiff]

OK, thanks! Just let me know when something's ready, and I'll test it. Unless it's between 15 and 21 tomorrow. There's a family gathering, and I would prefer to betatest, but I'm afraid my parents will give my part of the inheritance to my brothers if I don't attend...

[krambriw]

I made the modification to the saving of temperatures so far, please test and see if it helps

You get a printed line in the log when it succeeds and it should also print when it fails. It will keep on retrying until it succeeds

If this works, I will modify for the other data types as well