ClimateDataCalculation

[krambriw]

Hello, I have made a new version supporting gaps (it was actually a supported HighCharts api feature)

[Mastiff]

How nice, thanks! I'll get to that tomorrow!

[Mastiff]

Weird, I still have those lines down to zero. Also it seems to be very sensitive to the gaps, probably too sensitive (a few minutes, and there's a gap). But worse: Opening the graph locks up Firefox for a minute or two. Any idea what that can be? I'm PM'ing you the address to one of my graphs.

[krambriw]

So here is a new version that supports making charts with 10 minutes data point interval (default). To update, replace with the new files, restart EG and open/save existing actions for Combo reports.

To select/deselect the new setting, tick the check box as below

[Mastiff]

Wow, than you for that one! 1/10 of the data really sped up the loading of those graphs! Very nice, sir! A couple of things:

1. What program do you use to look at the database? I have zero experience with databases (well, close to zero - I was doing a bit of dBase work in the early 90's, but I would imagine stuff has changed a bit since that...). I was looking for the reason for a gap in the graph, and I printed the full database for "Loftet". That locked up EG for abour 10 minutes...

2. I found the area with data for the gap (in the file log, it was long gone in the log window because of the mass of data, which is why I'd like to have a program to look at this in a better way), it's like this:

Code: Select all

2015-06-02 09:26:36     INFO: (u'2015-05-29 18:18:45', 1432916325, u'Loftet', 25.600000000000001, u'', 10.0, 10.0, 25.579999999999998)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:21:09', 1432916469, u'Loftet', 25.600000000000001, u'', 10.0, 10.0, 25.579999999999998)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:21:57', 1432916517, u'Loftet', 25.5, u'', 10.0, 10.0, 25.57)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:22:45', 1432916565, u'Loftet', 25.5, u'', 10.0, 10.0, 25.550000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:24:21', 1432916661, u'Loftet', 25.5, u'', 10.0, 10.0, 25.550000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:25:09', 1432916709, u'Loftet', 25.5, u'', 10.0, 10.0, 25.530000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:25:57', 1432916757, u'Loftet', 25.5, u'', 10.0, 10.0, 25.52)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:26:45', 1432916805, u'Loftet', 25.5, u'', 10.0, 10.0, 25.5)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:27:33', 1432916853, u'Loftet', 25.5, u'', 10.0, 10.0, 25.5)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:28:21', 1432916901, u'Loftet', 25.5, u'', 10.0, 10.0, 25.5)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:29:09', 1432916949, u'Loftet', 25.5, u'', 10.0, 10.0, 25.5)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:29:57', 1432916997, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.48)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:30:45', 1432917045, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.469999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:31:33', 1432917093, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.449999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:32:21', 1432917141, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.43)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:33:09', 1432917189, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.420000000000002)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:33:57', 1432917237, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.399999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:34:45', 1432917285, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.399999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:35:33', 1432917333, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.399999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:36:21', 1432917381, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.399999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:37:09', 1432917429, u'Loftet', 25.399999999999999, u'', 10.0, 10.0, 25.399999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:37:57', 1432917477, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.379999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:38:45', 1432917525, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.370000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:39:33', 1432917573, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.350000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:40:21', 1432917622, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.329999999999998)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:41:09', 1432917669, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.32)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:42:45', 1432917765, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.300000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:43:33', 1432917813, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.300000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:44:21', 1432917861, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.300000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:45:09', 1432917909, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.300000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:46:45', 1432918005, u'Loftet', 25.300000000000001, u'', 10.0, 10.0, 25.300000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:47:33', 1432918054, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.280000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:48:21', 1432918101, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.27)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:49:09', 1432918150, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.25)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:49:57', 1432918198, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.23)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:51:33', 1432918293, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.219999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:52:21', 1432918341, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.199999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:53:09', 1432918389, u'Loftet', 25.199999999999999, u'', 10.0, 10.0, 25.199999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:54:45', 1432918486, u'Loftet', 25.100000000000001, u'', 10.0, 10.0, 25.18)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:55:33', 1432918534, u'Loftet', 25.100000000000001, u'', 10.0, 10.0, 25.170000000000002)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:56:21', 1432918581, u'Loftet', 25.100000000000001, u'', 10.0, 10.0, 25.149999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:57:09', 1432918629, u'Loftet', 25.100000000000001, u'', 10.0, 10.0, 25.129999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:57:57', 1432918677, u'Loftet', 25.100000000000001, u'', 10.0, 10.0, 25.120000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:58:45', 1432918726, u'Loftet', 25.0, u'', 10.0, 10.0, 25.079999999999998)
2015-06-02 09:26:36     INFO: (u'2015-05-29 18:59:33', 1432918774, u'Loftet', 25.0, u'', 10.0, 10.0, 25.07)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:00:21', 1432918822, u'Loftet', 25.0, u'', 10.0, 10.0, 25.050000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:01:09', 1432918869, u'Loftet', 25.0, u'', 10.0, 10.0, 25.030000000000001)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:01:57', 1432918917, u'Loftet', 25.0, u'', 10.0, 10.0, 25.02)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:02:45', 1432918966, u'Loftet', 25.0, u'', 10.0, 10.0, 25.0)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:03:33', 1432919014, u'Loftet', 25.0, u'', 10.0, 10.0, 25.0)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:04:21', 1432919062, u'Loftet', 25.0, u'', 10.0, 10.0, 25.0)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:05:09', 1432919110, u'Loftet', 25.0, u'', 10.0, 10.0, 25.0)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:05:57', 1432919158, u'Loftet', 25.0, u'', 10.0, 10.0, 25.0)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:08:21', 1432919301, u'Loftet', 24.899999999999999, u'', 10.0, 10.0, 24.98)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:09:09', 1432919350, u'Loftet', 24.899999999999999, u'', 10.0, 10.0, 24.969999999999999)
2015-06-02 09:26:36     INFO: (u'2015-05-29 19:09:57', 1432919398, u'Loftet', 24.899999999999999, u'', 10.0, 10.0, 24.949999999999999)

This looks like this in the graph, zoomed in:

So there's a gap between 18.19 and 19.09, but there is a lot of data, just not on the exact 10 minute. Is it possible in any way to read the closest data to the ten minutes instead of getting that gap, or is that impossible? I'm just asking, I have no idea about this.

Oh, btw: Do you get mails warning you about new posts in your threads? I don't get that anymore and I'm wondering if it's my account or a general error on the board.

[krambriw]

1. What program do you use to look at the database?

I have nothing special, normally I just write a suitable python script if we are talking about the databases for this plugin. But I guess there could be many other alternatives. Have you tried with excel?

2. I found the area with data for the gap

Hmm, it looks like there should not be any gap here...

Is it possible in any way to read the closest data to the ten minutes instead of getting that gap

It works that way (or should). In the python code I am looking at the time when it is changing. Lets say it is now 18:19, I will then pick the first value found recorded at that time. The next will be the first value found at 18:20. And so on

Maybe you could PM me the link to download your db where this error is happening and I will have a look at it

Oh, btw: Do you get mails warning you about new posts in your threads?

Sorry, I don't use that feature

[Mastiff]

1. No, Excel does not want to open them. But it would help to limit the scope. Here's your code I use to read the database, maybe you could show me how to limit on time/date as well as sensor, or even how to export it to a text file, if you do that:

Code: Select all

import sqlite3

def GetAllData(
    dbName,
    dbTableId
):
    dbName = str(dbName)
    conn = sqlite3.connect(dbName)
    c = conn.cursor()
    t = ((dbTableId),)
    try:
        c.execute('SELECT * FROM '+dbName.split('.')[0]+' WHERE tableId=?', t)
        coll = c.fetchall()
        conn.close()
        return coll
    except:
        conn.close()
        return None

rows = GetAllData('tempData.db', 'Loftet')

for row in rows:
    print row

2. I'm glad you already had thought of that, I was scared thinking that only I had that idea! You'll find the file tempData.zip at the same URL as you got the previous temp database of mine. I zipped it this to shave off some time with the crappy upload I have here.

As for the mail thing I actually got a mail warning for this one! Maybe because I was into my control panel, changed and changed back and saved a couple of settings. They're just like they were before, but maybe they were garbled in the internal system at the forum.

[krambriw]

In your db for 'Loftet', I can see the gap from '2015-05-29 21:52:21' until '2015-05-29 23:44:22'.

I realize however that I'm not handling gaps larger than one hour...so maybe this is messing up the graph (I'll have to fix this)

As for exporting data to a file, I have adjusted the script a bit and here you can set the starting/ending date & hour

Code: Select all

import sqlite3

def GetAllData(
 dbName,
 dbTableId
):
    dbName = str(dbName)
    conn = sqlite3.connect(dbName)
    c = conn.cursor()
    t = ((dbTableId),)
    try:
        c.execute('SELECT * FROM '+dbName.split('.')[0]+' WHERE tableId=?', t)
        coll = c.fetchall()
        conn.close()
        return coll
    except:
        conn.close()
        return None

rows = GetAllData('tempData.db', 'Loftet')

fileHandle = None
fileHandle = open('dbExport.txt', 'a')
for row in rows:
    if row[0].find('2015-05-29 21')> -1 or row[0].find('2015-05-29 23')> -1:
        fileHandle.write(str(row)+'\r\n')
fileHandle.close()
print 'Export done'

[Mastiff]

Nice, thanks for the script! As for the gap the missing bit is from 18:18 to 19:08 in the graph, and there seems to be data in the graph in the period where there actually are no data in the database. So you're probably right, as usual.

[krambriw]

I found the problem (handling more than 1 hour gaps). Updated the CreateComboHtml.py file

Now I think it looks correct

Skärmklipp.PNG (7.71 KiB) Viewed 3891 times

[Mastiff]

Are you certain? On my system the gap seems to be between 19.13 and 21.03 now, not 21.52 and 23.44 (when zooming in and holding the mouse over the data points).

[Mastiff]

[Mastiff]

OK, it seems to be something weirder. It saves a new report, but the data in the new report stops at 14.51 today.

[krambriw]

I think I see your point, you are correct, something is wrong in the calculation & gap compensation I am trying to achieve. It is easy to see if you generate a chart for five days, then it should match ok but for 30 days it gets corrupted...

I need to see if I can improve the gap compensation

[Mastiff]

OK, thanks! So this time it's not just me being crazy!