Drawing graphs in EG

[HTPCanwender]

I want to create and send a picture (.png or .jpg) that contains a graph of measured temperature values to my MQTT Dashboard. Is it possible to create the picture with EG? Witch library do I need? How can I install it? Sending it with the MQTT plugin is possible.

Thank you in advance.

[kgschlosser]

Ohhhhh this is a complicated thing to have to deal with..

you need to be using the latest release of EventGhost 0.5
you will need to install python 2.7 x86 (32 bit). can be gotten from HERE.

once you have that installed python 2.7 there are a couple of libraries needed. if you used the default installation location for all users.. you will need to run the commands below. if you installed to some other location then you will need to change the path accordingly.

Code: Select all

c:\python27\scripts\pip install matplotlib

Code: Select all

c:\python27\scripts\pip install numpy

now restart EG if you had it running.. You can test if the modules are available by creating a macro and adding a python script action. paste the code below into the script then press the OK button then the Test button. if no errors occur then all is good.

Code: Select all

import matplotlib
import numpy

Now depending on the type of graph you want is going to depend on what we do with the code.HERE is a good website that shows 50 different graphs and provides example code of each of them.

[HTPCanwender]

Thank you for your quick answer. I think, this is a project for my next holydays.

Yours

Arnd

[kgschlosser]

if you give me an example of the data you want to dump into a graph and also what kind of a graph you want to use I can help in giving you some direction.

It can grow to be more then a simple graph if there is a lot of data you may opt to do something 3d and add in some navigation ability to navigate through the graph. possibly providing tool tips for various components to it. You can get quite complex. You also have different ways to offer the graph up to be viewed.. You can do a native Windows frame (the application box), or you can do javascript or html. you can also save it as any number of image files.

[dan Edens]

if you give me an example of the data you want to dump into a graph and also what kind of a graph you want to use I can help in giving you some direction.

Can i jump in on this, I would love to do this in a EG plugin so I can use eg.globals and my eg tools in my scriptsssss.
actually I need to think through what I would leave to pythonscript and what I'd want as an Action.
I'll obsess over mathploglib for a few days and go from there. I've been needing to master it for long enough.

Heres an example dataset tho if that helps.
Lets pick a fun one, Contour graph!

https://matplotlib.org/3.1.1/gallery/im ... er-mask-py

Code: Select all

Units: gons 

x		y		z

358.2081, 77.98767, 8.64653

358.2075, 77.98698, 8.64647

329.8109, 81.79859, 9.72455

296.444, 86.00778.01008

280.8094, 88.12063, 39.302

278.1551, 88.65435, 49.81701

274.3098, 89.03193, 60.17438

273.4284, 89.28923, 70.66862

272.7982, 89.36868, 81.34909

245.8424, 00.5406, 20.76551

[kgschlosser]

I was asking HTPCanwender for an example data set of what is to be graphed/plotted.

depending on the type of graph or plot is going to dictate what is needed for a data set and vise versa

Are you providing me with an example to hammer out some code for you?

[HTPCanwender]

Please give me some time. At the moment I am "playing" with the hardware. The source of my data is a Sonoff basic WLAN switch with TASMOTA firmware with an additional temperature and humidity sensor AM2301. The Sonoff reports the data periodically with MQTT messages. They are in JSON format:

Code: Select all

{"Time":"2019-11-10T16:03:21","AM2301":{"Temperature":20.8,"Humidity":48.3},"TempUnit":"C"}
{"Time":"2019-11-10T16:13:21","AM2301":{"Temperature":20.8,"Humidity":48.3},"TempUnit":"C"}
{"Time":"2019-11-10T16:18:21","AM2301":{"Temperature":20.8,"Humidity":48.3},"TempUnit":"C"}
{"Time":"2019-11-10T16:23:21","AM2301":{"Temperature":20.7,"Humidity":48.4},"TempUnit":"C"}
{"Time":"2019-11-10T16:28:21","AM2301":{"Temperature":20.7,"Humidity":48.4},"TempUnit":"C"}
{"Time":"2019-11-10T16:33:21","AM2301":{"Temperature":20.7,"Humidity":48.5},"TempUnit":"C"}
{"Time":"2019-11-10T16:38:21","AM2301":{"Temperature":20.7,"Humidity":48.6},"TempUnit":"C"}
{"Time":"2019-11-10T16:43:21","AM2301":{"Temperature":20.7,"Humidity":48.7},"TempUnit":"C"}

At the moment I am storing only the last actual value of each device in a dictionary and send it with the MQTT plugin to my android dashboard. This is necessary because my dashboard is not allways online and I need the possibility to refresh it with the actual data from EG. Storing the MQTT messages with the mosquitto MQTT broker (persistent session) is not an option for me.

Here is an example of the definition of the dictionary. Every device (e.g. Sonoff-33) has several components and every component has several properties (e.g. status of a relay, temperature, humidity or the URL of a picture).

Code: Select all

eg.globals.Geraete = {"Sonoff-31": {         "Switch1": {      "Power": None}},    # ON / OFF   
                      "Sonoff-32": {          "Relay1": {      "Power": None},     # ON / OFF   
                                              "Relay2": {      "Power": None}},    # ON / OFF   
                      "Sonoff-33": {          "Relay1": {      "Power": None},     # ON / OFF   
                                              "AM2301": {"Temperature": None,      # Number    
                                                            "Humidity": None}},    # Number  
                      "FS20TFI-1": {"FHT80TF-K00-4211": {     "Status": None,      # ON / OFF 
                                                             "Picture": None}}}    # URL 

The dashboard can show a picture provided by a webserver if you send the URL as payload to the dashboard. I am using the webserver plugin of EG for this. I am right at the beginning. I did not decide where to store the periodical data (in a file or in memory?) and in which format. At the end I want to create a picture with a line graph of the development of the temperature in jpg format and store it on the webserver. From there it can be accessed from the dashboard. There is no plan for an interactive picture on the windows PC. There are many questions not answered or even asked. E.g. How long should be the time period at the x-axis? How is the resolution at the y-axis? Which resolution the whole picture should have? But I am not in hurry.

[kgschlosser]

we can do whatever it is you want as far as the graph goes. It likes like what you are wanting is historical data graphs. That can be represented as a line or bars. It all depends on how long you want to cache the information for.

if this is a single entry for a single device

Code: Select all

{"Time":"2019-11-10T16:03:21","AM2301":{"Temperature":20.8,"Humidity":48.3},"TempUnit":"C"}

it looks like there is a poll per device that is supposed to be every 5 seconds. when dealing with temperatures a 5 second poll is way to fat. you want between 30 seconds at the fastest and probable every minute is going to be ideal. Temp sensors are usually thermisters and they are not the fastest in terms of temp changes. A great example would be a digital meat thermometer. when you stick that thing into the meat how long does it take before it stops moving. close to a minute and a half or 2 minutes. so running a temperature poll every 30 seconds should be fine.

so you are looking at 91 characters per entry at 30 seconds = 10920 per hour * 24 hours = 262080 per day * 365 days a year = 95659200 per year.

1 character is a single byte. so 1 device for a whole year of logging would take up 95659200 bytes or 91.22772216796875MB worth of drive space a year. That is if you only write the data to a flat file as JSON. So the length of time is completely up to you. You can have several button on the graph to show average per hour, average per week, average per month or you can look at it as a line over the whole year taking into account every sample.

Now in terms of using the EG webserver for displaying the graph. That is fine to do I am sure we can set up some kind of a property that would get called when a specific webpage was accessed and at that point the graph would be built.

There is no need to have a 3d graph at all because the data is only 2 axis... x = time , y = temp
you can also show the humidity in a separate graph. as well as a "feel likes" temperature which is a marrying of the humidity and temperature.

hammering out a graph for this is super easy to do for what you are needing. JSON is fine, if you wanted to get all kinds of professional we can also use MySQL to hold the data if you wanted (less chance of corruption). it all depends on how you want to go about it.

[HTPCanwender]

Thank you for your advice. At the moment the temperature is measured every 5 minutes, not every 5 seconds. The time is in military format: 16h 23 min 21 sec.

I want to keep the project as simple as possible. The Android MQTT dashboard itself is able to show a jpg-file located on a webserver. E.g. http://IPadress:port/mypicture.jpg. That's all I want to use. It's like an old fashioned webcam with periodically actualized pictures. So I have to update the picture on the webserver every 5 minutes when new data arrive.

The dashboard is also able to open an external browser like chrome to show a complete website. But this is not my goal. I want only to see the development of the temperature or any other time dependent value, let's say for the last 24 hours.

[kgschlosser]

sorry about that my brain was cooked. I am not sure why I was reading the minutes as tho they were the seconds. that's my bad. It was a long day yesterday.....


so to recalculate that based on 5 minutes.

91 * 12 times an hour = 1092 bytes * 24 hours a day = 26208 bytes * 365 days a year = 9565920 bytes, 9.12MB a year to data log a single device.

we can do that as well. The pyplot module has the ability to build an actual interactive type of webpage pyplot relies on matplot lib. or we can use matplot lib directly and simply dump an image into a directly. either way it's still easy to do. The only thing is that dumping the image to a folder every 5 minutes is kind of a waste of resources if it is not being used. That is your call to make.


here is an example of what you are kind of looking for I think

Code: Select all

import matplotlib.pyplot as plt
import numpy as np

params = {
    'legend.fontsize': 16,
    'figure.figsize': (16, 10),
    'axes.labelsize': 16,
    'axes.titlesize': 16,
    'xtick.labelsize': 16,
    'ytick.labelsize': 16,
    'figure.titlesize': 22
}

plt.rcParams.update(params)
# plt.style.use('seaborn-whitegrid')

# this section is only to build a mock up of your data
import datetime
import random

dt = datetime.datetime.today()
data = []

temp_increment = 0.1
humidity_increment = 0.1

humidity = float(random.randrange(20, 80)) + (float(random.randrange(0, 9)) / 10)
temp = float(random.randrange(10, 50)) + (float(random.randrange(0, 9)) / 10)

count = 0

while len(data) < 500:
    count += 1
    temp += temp_increment
    humidity -= humidity_increment
    dt = dt - datetime.timedelta(minutes=5)

    entry = dict(Time=dt.strftime('%c'), AM2301=dict(Temperature=temp, Humidity=humidity, TempUnit='C'))
    data += [entry]
    temp_random_flip = random.randrange(1, random.randrange(2, 20))
    humidity_random_flip = random.randrange(1, random.randrange(2, 20))

    if temp >= 50.0 or temp <= 10.0 or count >= temp_random_flip:
        temp_increment = -temp_increment
        count = 0

    if humidity <= 20.0 or humidity >= 80.0 or count >= humidity_random_flip:
        humidity_increment = -humidity_increment
        count = 0


# here we are going to parse your data into the sections needed for the graph

x = []
y1 = []
y2 = []

for entry in data:
    x += [entry['Time']]
    y1 += [entry['AM2301']['Temperature']]
    y2 += [entry['AM2301']['Humidity']]


# Plot Line1 (Left Y Axis)
fig, ax1 = plt.subplots(1, 1, figsize=(16,9), dpi= 80)
ax1.plot(x, y1, color='tab:red')

# Plot Line2 (Right Y Axis)
ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis
ax2.plot(x, y2, color='tab:blue')

# Decorations
# ax1 (left Y axis)
ax1.set_xlabel('Sensor Sample Time', fontsize=20)
ax1.tick_params(axis='x', rotation=45, labelsize=12)
ax1.set_ylabel('Temperature (C)', color='tab:red', fontsize=20)
ax1.tick_params(axis='y', rotation=0, labelcolor='tab:red' )
ax1.grid(alpha=.4)

# ax2 (right Y axis)
ax2.set_ylabel("Humidity", color='tab:blue', fontsize=20)
ax2.tick_params(axis='y', labelcolor='tab:blue')
ax2.set_xticks(np.arange(0, len(x), 60))
ax2.set_xticklabels(x[::60], rotation=90, fontdict={'fontsize':10})
ax2.set_title("AM2301 Temperature and Humidity Readings", fontsize=22)
fig.tight_layout()

plt.show()

I coded in a random generator to create the samples. that graph is 500 samples

and here is one with 1000 samples.

[kgschlosser]

if you wanted to show to graph at the same time. This is an example where the second graph is a subset of the first.

image is 1000 samples

image is the same 1000 samples as above but only the first 500 samples are being shown.. call this is "zoomed in"

Code: Select all

import matplotlib.pyplot as plt
import numpy as np

params = {
    'legend.fontsize': 16,
    'figure.figsize': (16, 10),
    'axes.labelsize': 16,
    'axes.titlesize': 16,
    'xtick.labelsize': 16,
    'ytick.labelsize': 16,
    'figure.titlesize': 22
}

plt.rcParams.update(params)
# plt.style.use('seaborn-whitegrid')

# this section is only to build a mock up of your data
import datetime
import random

dt = datetime.datetime.today()
data = []

temp_increment = 0.1
humidity_increment = 0.1

humidity = float(random.randrange(20, 80)) + (float(random.randrange(0, 9)) / 10)
temp = float(random.randrange(10, 50)) + (float(random.randrange(0, 9)) / 10)

count = 0

while len(data) < 1000:
    count += 1
    temp += temp_increment
    humidity -= humidity_increment
    dt = dt - datetime.timedelta(minutes=5)

    entry = dict(Time=dt.strftime('%c'), AM2301=dict(Temperature=temp, Humidity=humidity, TempUnit='C'))
    data += [entry]
    temp_random_flip = random.randrange(1, random.randrange(2, 20))
    humidity_random_flip = random.randrange(1, random.randrange(2, 20))

    if temp >= 50.0 or temp <= 10.0 or count >= temp_random_flip:
        temp_increment = -temp_increment
        count = 0

    if humidity <= 20.0 or humidity >= 80.0 or count >= humidity_random_flip:
        humidity_increment = -humidity_increment
        count = 0


# here we are going to parse your data into the sections needed for the graph

x = []
y1 = []
y2 = []

for entry in data:
    x += [entry['Time']]
    y1 += [entry['AM2301']['Temperature']]
    y2 += [entry['AM2301']['Humidity']]


plt.figure(1)
# Plot Line1 (Left Y Axis)
fig, ax1 = plt.subplots(1, 1, figsize=(16,9), dpi= 80)
ax1.plot(x, y1, color='tab:red')

# Plot Line2 (Right Y Axis)
ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis
ax2.plot(x, y2, color='tab:blue')

# Decorations
# ax1 (left Y axis)
ax1.set_xlabel('Sensor Sample Time', fontsize=20)
ax1.tick_params(axis='x', rotation=45, labelsize=12)
ax1.set_ylabel('Temperature (C)', color='tab:red', fontsize=20)
ax1.tick_params(axis='y', rotation=0, labelcolor='tab:red' )
ax1.grid(alpha=.4)

# ax2 (right Y axis)
ax2.set_ylabel("Humidity", color='tab:blue', fontsize=20)
ax2.tick_params(axis='y', labelcolor='tab:blue')
ax2.set_xticks(np.arange(0, len(x), 60))
ax2.set_xticklabels(x[::60], rotation=90, fontdict={'fontsize':10})
ax2.set_title("AM2301 Temperature and Humidity Readings", fontsize=22)
fig.tight_layout()


plt.figure(2)
# Plot Line1 (Left Y Axis)
fig, ax1 = plt.subplots(1, 1, figsize=(16,9), dpi= 80)
ax1.plot(x[:-500], y1[:-500], color='tab:red')

# Plot Line2 (Right Y Axis)
ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis
ax2.plot(x[:-500], y2[:-500], color='tab:blue')

# Decorations
# ax1 (left Y axis)
ax1.set_xlabel('Sensor Sample Time', fontsize=20)
ax1.tick_params(axis='x', rotation=45, labelsize=12)
ax1.set_ylabel('Temperature (C)', color='tab:red', fontsize=20)
ax1.tick_params(axis='y', rotation=0, labelcolor='tab:red' )
ax1.grid(alpha=.4)

# ax2 (right Y axis)
ax2.set_ylabel("Humidity", color='tab:blue', fontsize=20)
ax2.tick_params(axis='y', labelcolor='tab:blue')
ax2.set_xticks(np.arange(0, len(x[:-500]), 60))
ax2.set_xticklabels(x[:-500][::60], rotation=90, fontdict={'fontsize':10})
ax2.set_title("AM2301 Temperature and Humidity Readings", fontsize=22)
fig.tight_layout()


plt.show()

[kgschlosser]

here is an example using the plotly library

Code: Select all

import plotly.graph_objects as go
import plotly
import plotly.io as pio
pio.renderers.default = "browser"

# this section is only to build a mock up of your data
import datetime
import random

dt = datetime.datetime.today()
data = []

temp_increment = 0.1
humidity_increment = 0.1

humidity = float(random.randrange(20, 80)) + (float(random.randrange(0, 9)) / 10)
temp = float(random.randrange(10, 50)) + (float(random.randrange(0, 9)) / 10)

count = 0

while len(data) < 105120 / 4:
    count += 1
    temp += temp_increment
    humidity -= humidity_increment
    dt = dt - datetime.timedelta(minutes=5)

    entry = dict(Time=dt.strftime('%c'), AM2301=dict(Temperature=temp, Humidity=humidity, TempUnit='C'))
    data += [entry]
    temp_random_flip = random.randrange(1, random.randrange(2, 20))
    humidity_random_flip = random.randrange(1, random.randrange(2, 20))

    if temp >= 50.0 or temp <= 10.0 or count >= temp_random_flip:
        temp_increment = -temp_increment
        count = 0

    if humidity <= 20.0 or humidity >= 80.0 or count >= humidity_random_flip:
        humidity_increment = -humidity_increment
        count = 0

date = []
temperature = []
humidity = []

for entry in data:
    date += [entry['Time']]
    temperature += [entry['AM2301']['Temperature']]
    humidity += [entry['AM2301']['Humidity']]

fig = go.Figure()
fig.add_trace(
    go.Scatter(
        x=date,
        y=temperature,
        name="Temperature",
        line_color='red'
    )
)

fig.add_trace(
    go.Scatter(
        x=date,
        y=humidity,
        name="Humidity",
        line_color='blue'
    )
)

fig.update_layout(
    title_text='AM2301 Temperature and Humidity',
    xaxis_rangeslider_visible=True
)

plotly.offline.plot(
    fig,
    filename=r'C:\some\output\path\index.html',
    auto_open=False
)

it produces a nice JavaScript enabled HTML document that is interactive including over overs and a range slider




note: This applies to everyone. If you want to be able to embed HTML code into a post you will need to send me a PM along with the zipped up code. I am the only one that can do it as the HTML code will need to be uploaded to the server to do it. This is so that I can review it and make sure that nothing malicious exists in the code. I am going to reserve this feature specifically for the intent of displaying things like shown above. So the user will not have to deal with dealing with archives and extracting files. It is also going to be a foundation for being able to set up a live EventGhost as an example of how to use EventGhost and the webserver plugin and also what the possibilities are.

[kgschlosser]

i forgot to mention. on that graph example. pay close attention to the bottom graph there are 2 sliders one at either side of the graph. this allow you to set the range to display in the top graph.

[HTPCanwender]

Oh, you are so diligent. Thank you very much. I have no time left this week. I will have a look at your code next weekend.

[HTPCanwender]

I couldn't wait until the weekend and tried your code creating the png image and spent a night for it. I am collecting the sensor data in a separate file and regenerate the image periodically from it. But I think the html solution you suggested is actually better. Since I have no experience with webservers, I will certainly have more questions in the future. But first I will adapt the design of the png image to my needs. I think, I will find all explanations on the internet.

Please be patient with me. I learned programming in the early 80s with BASIC and FORTRAN. At this time python was even not invented. Object oriented programming is absolutely new for me and really a hard job.

Thanks again.