ClimateDataCalculation

[Mastiff]

Well...no. This morning the threading had locked up (but not the execution of other stuff, luckily), so I have now gone to logging every 5 minutes in the action settings for each room. But it seems like the queue is rising and falling (quite fast, but I don't know if that's just my imagination) even if it shouldn't log very often now. Is a thread started even when it's less than five minutes since the last logging on a sensor?

[krambriw]

Is a thread started even when it's less than five minutes since the last logging on a sensor?

Yes, the check if it shall be logged is starting on each event. A check up of last saved event in the database is then performed. If a save is not needed, the thread is terminated. So your observation that the Q is being reduced faster is correct.

[Mastiff]

OK, I'm going to do some work video conversion today, so I'll see if it helps to have them on 5 minutes.

[Mastiff]

Nope, there doesen't seem to be any change in how fast the queue (terrible word, I prefer the Norwegian k├© or swedish k├Â!) rises. It should have been rising 1/5 as fast when I have them set to every 5 minutes instead of every minute, right? But if there is a change it's a lot smaller then that.

[krambriw]

The rise of the Q would be the same (is driven by how often you receive an event from the sensors, and this is unchanged). The fall of the Q should however be faster.

The thing is that I do NOT store the last event in memory for each sensor. Instead I do a check-up in the database to see if it is time to do it or not. To do this, with the current implementation, I need to start a thread...
This check-up will therefore happen EVERY time a sensor event is received. It was not designed to be fail-safe against a Norwegian author rendering video in the same machine

EDIT: Does it crash or survives?

[krambriw]

@Mastiff

Hello,
I could not stop thinking of how the saving of data could be further improved and made more efficient. What I wanted was to avoid starting a thread just to check the database if it was time to save or not. So, following the idea of keeping the last event data in run-time, I made a new test version that does exactly that. It keeps the data from the last events in memory and when a new event comes, it checks if the time delta is within or outside the configured logging interval. If outside, the thread will be started as earlier and data will be saved. This change should reduce the number of threads being started.

Please note: in this test version, the change is only for capturing temperature data events!

If you wanna check the event data in run-time (for debugging), you can use this python script:

Code: Select all

print eg.plugins.ClimateDataCalculation.plugin.lastTempData

The output typically looks like:

12:24:49 {'1796': [1452511444.4130001, 4.7000000000000002], '8194': [1452511452.483, 0.90000000000000002]}

where you find the id of the sensors, the time stamps and temperature readings when they where saved

This test version is available in the normal place in a separate folder "Test version 0.4.3.beta_1"

Best regards, Walter

[krambriw]

Applied the same changes to cover also humidity, wind and rain data.

This test version is available in the normal place in a separate folder "Test version 0.4.3.beta_2"

[Mastiff]

Hi, Walter! Sorry I fell out a bit there... Well, I guess I can drive any system to the brink, can't I? Yesterday it had actually messed up so badly after filling up the queue (with 2000 queue size!) that my wife was unable to turn up the temp in the home gym. Which is typical, stuff like that always appens when she uses it! But it's brilliant that you could change it this way! I will try to install the new version now and see what happens, thanks! I'll let you know later today or tomorrow!

Edit: Well, I hardly see a queue at all here at least in the beginning! I'm guessing that's a good sign. I will convert video in a while, so we'll see what happens then.

Edit 2: Btw what you have done is essentially what I was thinking of doing, in a script! And what I have done with turning on and off the heaters, so signals doesn't fly around all the time.

[Mastiff]

Preliminary results: Perfect! Queue up to 10 at the max when running video conversion and reports at the same time! I have a larger conversion later today (about one hour), so I'll see what that does, but I think you hit the spot!

[krambriw]

Dear Mastiff,
Thank you for your efforts, I am really happy that you try it to the limits! This is a very good way to test it, it is the only way we know if it is stable enough or not. The modifications I have made during the recent versions was requiring me to enter new fields of competence, I learned something new with every challenge!

Thank you again,
Walter

[Mastiff]

No, really! I should, and do, thank you for putting up with all my weird requests! And I can tell you that during the heavy, one hour long video conversion (I convert from one type of MJPEG codec to another, with resize, bluring and sharpening, and it uses both hard drive and CPU ressourses) I never saw 2 digits in the queue, even if I ran the reports twice in that time! So I think we can officially call this version working! When not running reports I usually only see a queue at all every five minutes (since all logging is set to five minutes they tend to all come in within the same minute, which is a factor of 5 from when I restarted last).

Btw is the logging dependant on a specific number of digits in the sensor ID? Because next weekend I'm going to be home, not at the cabin, and then I am going to implement the Python version of logging as part of my main temperature control script. I was thinking of something like this, after the first part of the script has converted the sensor ID to both a zone name (Rom) corresponding to the table name and a zone number (RomID) corresponding to the device ID:

Code: Select all

eg.plugins.ClimateDataCalculation.TempDataCapture(Rom, RomID, 5, 22.0, 5, False, 2.0, False, u'equal-less-greater')

With the variables put in, that would be:

Code: Select all

eg.plugins.ClimateDataCalculation.TempDataCapture(WC, 1, 5, 22.0, 5, False, 2.0, False, u'equal-less-greater')

Would this work? Also I don't really change anything else than the table name and the the device ID between the rooms, so is it possible to remove the rest of the stuff, or does it have to be there?

[krambriw]

Just to update myself

What you wanna achieve? Are you somehow thinking of building up new database tables using the actions of the plugin? If so, yes, I think that is possible but not as in your example using the action. Indicate if this is your target, then I will check if it is possible and how to do it.

[Mastiff]

Good morning, sir! There is really only one goal: consolidation. Let me tell you a little story, and please note that most of this has been done with your great help (and a bit of help from others)! I started with one macro for each room, with the signal from the sensor in that room as the trigger for the RFXtrx and the high and low temperature (I usually use .5 degrees in the room, for instance 19.5-20.5) and a small script executing. Then I doubled that because I got a Tellstick as a backup. Next step was to make one script for all rooms triggered by "RFXtrx.*" and code extracting the sensor ID, checking it against a conversion table and controlling the temperature from that, with tables for temperature, and after a while different temperature modes (day, night, no frost, off) controlled from a webpage in Girder, sending the changes to EG. Then I consolidated the two different systems by adding "Tellstick.*" as an event and using a second table to convert the sensor ID to the room. I also added code that stopped execution of the temperature control script if the zone/room was the same as the previous run, so I didn't run the full script and sendt on/off to the heater for both the RFXtrx and the Tellstick Duo and code that only sendt the control of the oven every five minutes if there was no change from the previuos signal (so two consecutive on signals were only sendt every five minutes, but if the signal went from on to off it was sendt at once). Finally I added code for when one of the Viking sensors had to be changed or I had to get new batteries, so that any new sensors were printed in the logg with lines of **** over and under, so it was easy to see them. The only thing missing is to integrate the logging of temperatures, which still is one macro for each room.

So what I want to have, is a way to keep using the same data tables (which should be easy because the name of the tables is in most instances the same as the name of the zone), but putting the data in from the same main script. Here is the code that I use to pull the sensorID, convert that to a zone ID and zone name for the RFXtrx, maybe it's clearer if you see it that way (and remember that this reacts to any signal in to the RFX, so it has to filter out stuff that isn't temperature sensors by using the full event string, but I think the commenting in Norwegian should be understandeable). The final line here is where I would like to put the logging, if possible.

Code: Select all

#1: Sjekke om det er RFX-en
if (eg.eventString.split(' ')[0]) == "RFXtrx.Type:":
#2: Sjekke om det er en Viking-sensor 
    if (eg.eventString.split(' ')[1]) == "Viking":
#Finne rommet sensoren kommer fra ved hjelp av tabellen i oppstartsskriptet
        key = int(eg.eventString.split(' ')[4])
        Rom = ""
        #print("Dette er en Viking-sensor fra RFXtrx-en")
    else:
        #print("Dette er RFXtrx-en, men ikke en Viking-sensor")
        eg.Exit()
 #3: Sjekke om det er en sensor som står i listen, og i så fall ta ut temperatur og signalstyrke
    if key in eg.globals.Romkonvertering:
        Rom = eg.globals.Romkonvertering[key]
        #print (Rom)
        #Skille ut temperatur og signal fra payload
        eg.globals.Romtemperatur[Rom] = float((eg.event.payload.split(' ')[2]))
        eg.globals.Signal[Rom] = (eg.event.payload.split(' ')[6])

#4: Hvis det ikke er en sensor som står i listen, skrives dette i loggen, så den er lett å finne igjen:
    else:
        print('**********************************')
        print("**Ny Viking-sensor i RFXtrx-en**")
        print('**********************************')
        print('************* '+str(key)+' *************')
        print('**********************************')
        eg.Exit()
#5: Logge temperatur til ClimateDataCalculation (eg.globals.Rom[int(Rom)] er oppslag i tabellen over romID og romnavn):
eg.plugins.ClimateDataCalculation.TempDataCapture(eg.globals.Rom[int(Rom)], Rom, 5, 22.0, 5, False, 2.0, False, u'equal-less-greater')

Was that understandable, or do I need to explain something better?

[krambriw]

OK, well, the way you can save data to the database using the plugin is supported mainly with the action named 'TempDataCapture'. The action however needs an event that fullfills certain criteria to work properly. Supported events are those that are listed in the plugin code (see from line 1609).

If you instead are going to manipulate the data, you can trick the plugin in (at least) two ways to save the data:

1) In your python script, you can combine & change whatever data you have and construct and trigger a "compatible event" that you in next step use in a macro to capture the temperature with.

2) This would be my choice. You combine your data likewise but instead save it directly without creating a new event. The trick here is to use a function that is already defined in the plugin, it is named 'SaveTempData'. You would still use all the good parts introduced in the plugin with putting threads into queues etc. The required syntax for this to work is

Code: Select all

SaveTempData(tableId, temperature, sInterval, setPoint, movAverage, useHysteresis, hysteresis, useRule, rule)

This code in a python script would save the temperature 15.0 to the tableId 'eg.globals.Rom[int(Rom)]' if the last data saved is at least 10 minutes older.

Code: Select all

eg.plugins.ClimateDataCalculation.plugin.SaveTempData(eg.globals.Rom[int(Rom)], 15.0, 10, 15.0, 1, False, 2.0, False, u'')

[Mastiff]

I knew you would have something, master! Number two works almost perfectly! Almost may be because of the padawan menace (me, of course...), but unlike the regular method it seems like this doesn't work the same way as the regular way, with checking when the signal came last. Am I right there? Because with this method I suddenly see a queue again, even without the video conversion running on the server, and that doesn't happen with the regular way. Or is it something else I may have messed up? This is the code I'm using for testing (I only use a trigger for one room for now, I have to double check that all the zone names are the same as the database table names):

Code: Select all

#Finne rommet sensoren kommer fra ved hjelp av tabellen i oppstartsskriptet
key = int(eg.eventString.split(' ')[4])
Rom = ""
print("Dette er en Viking-sensor fra RFXtrx-en")

if key in eg.globals.Romkonvertering:
    Rom = eg.globals.Romkonvertering[key]
    print('******Her er rom og temperatur****')
    print (eg.globals.Rom[int(Rom)])
    #Skille ut temperatur og signal fra payload
    eg.globals.Romtemperatur[Rom] = float((eg.event.payload.split(' ')[2]))
    print(eg.globals.Romtemperatur[Rom])

eg.plugins.ClimateDataCalculation.plugin.SaveTempData(eg.globals.Rom[int(Rom)], eg.globals.Romtemperatur[Rom], 5, 20.0, 1, False, 2.0, False, u'')

Edit: Weird. I'm wondering if I can be wrong. I do see queues all the time, but they don't grow on video conversion like they did before. Any idea why?

Edit 2: I am running the heaviest and longest type of conversion, and the queue surfaces here and there (actually very often), but it never gets large. Only 1-5, or so. Usually only 1 or 2. Obviously it is not like before. Another thing: Is it possible in any way to change the name of a table that's already saved? I have a couple that doesn't correspond with the final naming of zones, like "stua" instead of "stue" (the closest in English would be "the living room" instead of "a living room"). Can I edit that in any way? I would like not to loose the data I already have.