EventGhost

Posted: **Tue Mar 08, 2016 5:37 pm**

Sorry for the previous post. It is all working fine. I used the wrong IP.

Posted: **Tue Mar 15, 2016 12:42 pm**

Hi krambriw,

I'm having problems with updating to the Paho MQTT plugin using step 2. I create a new configuration in EG ok, but when I then try and add the new plugin to the config EG just hangs, so it looks like even doing the less fun step 1 method wouldn't work for me either. I've reverted temporarily, but wondered if you could give me some advice on what might be causing the issue please?

TIA,

Luke

Posted: **Tue Mar 15, 2016 5:08 pm**

You did this?

If you get some error message that Paho is missing or something related, try to copy the paho subfolder to /lib26/site-packages folder and restart EG
EDIT: Actually, I think you need to do this, copy or move the paho subfolder with all content as mentioned

Posted: **Tue Mar 15, 2016 11:02 pm**

Nope, I missed (missunderstood on first read) that step. It's now working. Thanks for the reply.

Posted: **Thu May 05, 2016 1:15 pm**

Hello,

I have just upgraded to the Paho-based plugin and my log is now spammed with MQTT Connection messages every 10 secs (see attachment). Is this normal behaviour?

Thank you

Posted: **Thu May 05, 2016 2:11 pm**

Hi again Walter,

By the way, the reason why I tried upgrading my old MQTT plugin is because the plugin listener seems to miss messages or not to trigger eg events when sent repeateadly

See the sample code below:

Code: Select all

print 'START SENDING'
for i in range(5):
    eg.plugins.MQTTthreads.publishMQTT(u'MQTT test', 
                                        u'localhost', 
                                        1883, 
                                        u'/openHAB/in/ECL_mN2_Chambre_chevetp/state', 
                                        str(i), 
                                        0, 
                                        False)
print 'COMPLETED'

When hitting the [Test] button several times in the Python script window, events are triggered but time to time only:

In the meantime, the openHAB listener receives them all with no issue.
Is there anything I am doing wrong or is this a real bug?
I attached the XML test file for testing on your side. It is configured with the 'old' MQTT plugin but it behaves the same with the new Paho plugin.

Thank you in advance for your help!

Posted: **Fri May 06, 2016 6:24 am**

Is this normal behaviour?

No, it is not, it looks like the client is being kicked out every twenty seconds

Posted: **Fri May 06, 2016 6:38 am**

not to trigger eg events when sent repeateadly

You cannot send the events that fast I think, try to insert a short time.sleep(0.1) in you script (modification below is for the paho version and my configuration but you get the idea)

Code: Select all

import time
print 'START SENDING'
for i in range(5):
    eg.plugins.MQTTthreads.publishMQTT(u'MQTT test', 
                                        u'192.168.10.248', 
                                        1883, 
                                        u'/zwave', 
                                        str(i), 
                                        0, 
                                        False,
                                        '691760406161'
    )
    time.sleep(0.1)
print 'COMPLETED'

08:35:52 MQTT./zwave u"4"
08:35:53 MQTT./zwave u"Device_6_Instance_3_temperature, 21.3700016"
08:35:53 START SENDING
08:35:54 COMPLETED
08:35:54 MQTT./zwave u"0"
08:35:54 MQTT./zwave u"1"
08:35:54 MQTT./zwave u"2"
08:35:54 MQTT./zwave u"3"
08:35:54 MQTT./zwave u"Device_6_Instance_4_temperature, 22.43"
08:35:54 MQTT./zwave u"4"
08:35:55 START SENDING
08:35:56 COMPLETED
08:35:56 MQTT./zwave u"0"
08:35:56 MQTT./zwave u"1"
08:35:56 MQTT./zwave u"2"
08:35:56 MQTT./zwave u"3"
08:35:56 MQTT./zwave u"4"
08:35:56 RFXtrx.Type: RTHN318 id: 7425 ' temperature: +13.2 deg C signal: 6 battery: 9'
08:35:57 START SENDING
08:35:58 COMPLETED
08:35:58 MQTT./zwave u"0"
08:35:58 MQTT./zwave u"1"
08:35:58 MQTT./zwave u"2"
08:35:58 MQTT./zwave u"3"
08:35:58 MQTT./zwave u"4"
08:35:59 START SENDING
08:36:00 COMPLETED
08:36:00 MQTT./zwave u"0"
08:36:00 MQTT./zwave u"1"
08:36:00 MQTT./zwave u"2"
08:36:00 MQTT./zwave u"3"
08:36:00 MQTT./zwave u"4"
08:36:01 SunTracker.Weather Condition: 'Fair'
08:36:06 RFXtrx.Type: THGN122/123, THGN132, THGR122/228/238/268 id: 1796 ' temperature: +10.0 deg C humidity: 50 %RH status: normal signal: 6 battery: 9'
08:36:06 MQTT.out u"['event', 'Oregon_Message', 'Direction', 'In', 'Oregon.Channel', '4', 'Oregon.Id', '112', 'Oregon.LowBattery', '0', 'Oregon.Moisture', '50', 'Oregon.SensorId', '7456', 'Oregon.Temp', '100', 'Value', '']"

Posted: **Fri May 06, 2016 9:34 am**

krambriw wrote:
not to trigger eg events when sent repeateadly
You cannot send the events that fast I think, try to insert a short time.sleep(0.1) in you script (modification below is for the paho version and my configuration but you get the idea)

Hello Walter,
Thank you for replying so fast.

Adding a pause does not fix the issue on my side. Even sending one single message (removing the for... loop from the test script) results in messages not translated into events by MQTT listener. And in the meantime, my openHAB server receives all events published by these tests like a charm, with no loss (even without the time.sleep() )

I also run the Eg test script on 3 different computer with all the same result: messages lost ramdomly when sending repeatedly.
I run the Eg test script on the 3 computers simultaneously (1 "publisher" and 3 "listeners): it shows that the only Eg instance that misses message is the one that is both sending and receiving.

So the code involved in this issue is rather the code handling subscribed messages rather than the code handling publishing messages
And publishing messages seem to interfere with reception and event triggering under some circumstances

I tried to look at your code but my only conclusion was that I need to get more experienced in Python coding before I can suggest anything

I hope this help to troubleshoot the issue.

Posted: **Fri May 06, 2016 10:31 am**

Do you have to publish and subscribe to the same topic? Maybe that is conflicting in your setup with openHAB? You could eventually try to have separate topics for sending & receiving. On the other hand, as you saw, I could without problem publish to the same topic as I subscribed to. I do not use openHAB so I cannot reproduce the errors and I have never heard other users having the same problem..

Regarding the code under the hood:

- there is a separate thread handling subscriptions (receiving of events) so it should not be possible to interfere
- when publishing /(sending a message) a new client connection is opened and then closed

I think the problem is related to that your connection only lasts 20 seconds, this is not a normal behavior. The normal is that the connection is kept steady until it is closed on purpose.

Posted: **Tue May 10, 2016 9:24 pm**

Hello Walter,

krambriw wrote:Do you have to publish and subscribe to the same topic? Maybe that is conflicting in your setup with openHAB? You could eventually try to have separate topics for sending & receiving.

You are right: openHab uses 2 topics for MQTT communication. One openhab/in/# for inbound traffic and one openhab/out/# for outbound traffic.
The issue came when I created Eg Python scripts for testing purpose: I was trying to simulate messages as if they were published by openHAB. So I was posting them to openhab/out/# from within Eg which was also listening to openhab/out/#

Based on the MQTT "topic" organization, I do not see any reason why I could not listen and post to the same topic from within Eg.

As you probably understood, I am using Eg to handle my entire automation system. OpenHAB only provides zwave hardware interfacing and a convenient built-in Web UI. All sets of automation functionalities (lights, air flow control, heating, multiroom notification, automated doors, watering, alarm, zone management, user authentification...) are implemented as Eg scripts and I want each of these "modules" to communicate using MQTT rather than plain Eg events.
The reason is that MQTT messages allow distributing my Eg home automation system seamlessly over several hardware and allow redundancy and watchdog process over several computers.
For example, the module "Notification" is activated on my laptop, on the main home automation PC and on the mediacenter PC in the living-room. When the system need to notify us, it just publishes a MQTT message and each of the running PC decides of the strategy to deliver the notification (OSD display, text-to-speech, push message to phone, light blink, email...)

krambriw wrote:On the other hand, as you saw, I could without problem publish to the same topic as I subscribed to. I do not use openHAB so I cannot reproduce the errors and I have never heard other users having the same problem..

Walter, the issue is independant from OpenHAB. Just see the xml test I sent earlier. If you click repeatedly on the [TEST] button for that script, you do not see all the corresponding Eg events display in the log : some are missed. On my side, this happens on all the PC on which I run that script.
I just used openHAB as a MQTT client to check that all messages were published by your plugin when clicking repeatedly on the [TEST] button (I could have used any other MQTT client than OpenHab)
This was to understand whether messages were not sent by Eg or or if they were not received by Eg. At this point, my conclusion is either that they are not received by the plugin or that the plugin does not manage to create Eg event for all of them.

krambriw wrote: Regarding the code under the hood:

- there is a separate thread handling subscriptions (receiving of events) so it should not be possible to interfere
- when publishing /(sending a message) a new client connection is opened and then closed

I understand the general threading layout which sounds to be the robust construction for this plugin but there should be some side effect we miss that is causing the behaviour of "lost messages"

krambriw wrote: I think the problem is related to that your connection only lasts 20 seconds, this is not a normal behavior. The normal is that the connection is kept steady until it is closed on purpose.

I don't think so: it behaves the same when the MQTT server is running on the same computer as the MQTT clients so there is no connection drop effect in that case. Or should there be any OS or network adapter mechanism that drops connection?

Posted: **Wed May 11, 2016 4:16 am**

Hello,
I agree & have come to the same conclusion regarding MQTT, it is an excellent way of integrating systems and devices. It is the IoT de-facto standard.

It is for sure not normal that the connection drops every 20 seconds, that is wrong!

I think we need to start isolate where the problem is so I have some questions & tests to suggest

1) Where do you run the MQTT broker and what OS is used in that environment?

2) Could you try to connect and test to another broker, try the public available and see if you experience the same problem when running your test script from EG?
test.mosquitto.org. Below is from my test using that broker, I do not miss any event.

Myself, I have also my system completely built up using several MQTT brokers but all are running in Raspberry Pi's under Linux

Code: Select all

06:19:47   START SENDING
06:19:48   COMPLETED
06:19:48   MQTT.openhab/out u"0"
06:19:48   MQTT.openhab/out u"1"
06:19:48   MQTT.openhab/out u"2"
06:19:48   MQTT.openhab/out u"3"
06:19:48   MQTT.openhab/out u"4"
06:19:51   START SENDING
06:19:52   COMPLETED
06:19:52   MQTT.openhab/out u"0"
06:19:52   MQTT.openhab/out u"1"
06:19:52   MQTT.openhab/out u"2"
06:19:52   MQTT.openhab/out u"3"
06:19:52   MQTT.openhab/out u"4"
06:19:54   START SENDING
06:19:55   COMPLETED
06:19:55   MQTT.openhab/out u"0"
06:19:55   MQTT.openhab/out u"1"
06:19:55   MQTT.openhab/out u"2"
06:19:55   MQTT.openhab/out u"3"
06:19:55   MQTT.openhab/out u"4"
06:19:58   START SENDING
06:19:59   COMPLETED
06:19:59   RFXtrx.Type: RTHN318 id: 7425 ' temperature: +6.4 deg C signal: 6 battery: 9'
06:19:59   MQTT.openhab/out u"0"
06:19:59   MQTT.openhab/out u"1"
06:19:59   MQTT.openhab/out u"2"
06:19:59   MQTT.openhab/out u"3"
06:19:59   MQTT.openhab/out u"4"

Posted: **Sat May 14, 2016 1:08 pm**

krambriw wrote:Hello,
1) Where do you run the MQTT broker and what OS is used in that environment?

I am running Mosquitto broker on the same computer as the one running Eg. It is a Win10 OS

But again, for me the broker is not the cause as any other subscriber to the test topic correctly receives all messages published during the test.
How can I add some clever 'print' command in your plugin to see if Mqtt messages are actually received and if it is only a matter of event generation in Eg?

krambriw wrote:2) Could you try to connect and test to another broker, try the public available and see if you experience the same problem when running your test script from EG?
test.mosquitto.org. Below is from my test using that broker, I do not miss any event.

I tested with test.mosquitto.org and got same results: Eg generating events ramdomly when received back from the broker.

Posted: **Sat May 14, 2016 3:02 pm**

How can I add some clever 'print' command in your plugin to see if Mqtt messages are actually received and if it is only a matter of event generation in Eg?

Since the simple test script you provided works for me but not for you, I for the moment believe there must something in your environment that is causing this problem

To add a print statement you could try (search for line 222) and add the print statement as below:

Code: Select all

        def on_message(client, userdata, msg):
            print client, userdata, msg.payload

Posted: **Sat May 14, 2016 6:33 pm**

krambriw wrote: To add a print statement you could try (search for line 222) and add the print statement as below:
Code: Select all
        def on_message(client, userdata, msg):
            print client, userdata, msg.payload

Below the result of the test with the logging you suggested.
As you can see, there are more messages received by your Plugin (the <paho.mqtt... > lines) than Eg events triggered by it (<the MQTT./openHAB/in... lines.
Indeed, there are as many <paho.mqtt...> lines as messages published. And it also matches the count of messages received by other subscribers

Code: Select all



19:57:02   MQTT./openHAB/in/AV_mRDC_Salon_media_album/state u"02"
19:57:02      <paho.mqtt.client.Client object at 0x0500D7B0> None 02
19:57:03      <paho.mqtt.client.Client object at 0x0500D7B0> None 03
19:57:04   MQTT./openHAB/in/AV_mRDC_Salon_media_album/state u"03"
19:57:04      <paho.mqtt.client.Client object at 0x0500D7B0> None 04
19:57:04   MQTT./openHAB/in/AV_mRDC_Salon_media_album/state u"04"
19:57:04      <paho.mqtt.client.Client object at 0x0500D7B0> None 04
19:57:04      <paho.mqtt.client.Client object at 0x0500D7B0> None 04
19:57:04      <paho.mqtt.client.Client object at 0x0500D7B0> None 04
19:57:05      <paho.mqtt.client.Client object at 0x0500D7B0> None 05
19:57:06   MQTT./openHAB/in/AV_mRDC_Salon_media_album/state u"05"

I have added additional log to the ProcessEvent() call. Eg event is triggered when self.eventCollection[res_key] is empty (log at 20:17:30) but on repeated calls (log at 20:17:33 to :34), self.eventCollection[res_key] keeps the value of previous res_base on the second MQTT message received and therefore does not trigger the event.

20:17:30 <paho.mqtt.client.Client object at 0x082C0FB0> None 30
20:17:30 ProcessEvent(self.eventCollection[res_key]=, res_base= ['30'])
20:17:30 MQTT./openHAB/in/AV_mRDC_Salon_media_album/state u"30" (eg event)
>>> first message triggers Eg event

20:17:33 <paho.mqtt.client.Client object at 0x082C0FB0> None 33
20:17:33 ProcessEvent(self.eventCollection[res_key]=, res_base= ['33'])
20:17:33 MQTT./openHAB/in/AV_mRDC_Salon_media_album/state u"33" (eg event)
>>> second message triggers Eg event

20:17:34 <paho.mqtt.client.Client object at 0x082C0FB0> None 34
20:17:34 ProcessEvent( self.eventCollection[res_key]= ['34']), res_base= ['34'])
>>> third message DOES NOT triggers Eg event

I hope this can help to go further. Tell me what other testing should I do.

EventGhost

MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client

Re: MQTT Client