EventGhost

Is there any chance we could get rid of the requirement for the leading slash? It is 100% non-standard and recommended against in the MQTT spec. Requiring the slash breaks integration with several other platforms.

References:
http://docs.oasis-open.org/mqtt/mqtt/v3 ... c398718106
http://www.hivemq.com/blog/mqtt-essenti ... -practices

edit: I've looked at this a little further and it appears that the implementation here kind of supports standard topics (no leading slash), but only if there is at least 2 slashes in the full topic string. Take a look at the following examples:

MQTT Client subscription: /test/#
Test publish command w/ Mosquitto: mosquitto_pub -h mqtt -t "/test" -m "message"
Result: message received

MQTT Client subscription: /test/#
Test publish command w/ Mosquitto: mosquitto_pub -h mqtt -t "/test/1/2/3" -m "message"
Result: message received

Now let's change to a standard topic:
MQTT Client subscription: test/#
Test publish command w/ Mosquitto: mosquitto_pub -h mqtt -t "test" -m "message"
Result: message not received

MQTT Client subscription: test/#
Test publish command w/ Mosquitto: mosquitto_pub -h mqtt -t "test/1" -m "message"
Result: message not received

MQTT Client subscription: test/1/#
Test publish command w/ Mosquitto: mosquitto_pub -h mqtt -t "test/1" -m "message"
Result: message received

MQTT Client subscription: test/1/#
Test publish command w/ Mosquitto: mosquitto_pub -h mqtt -t "test/1/2" -m "message"
Result: message received

So it appears to actually work with the leading slash but... only if the topic is at least 2 layers deep. In all test cases above the standard Mosquitto client successfully subscribes to and receives the messages. All test cases are utilizing standard notation and are acceptable publish/subscribe strings, but only some of them work with this plugin.

Thanks for analyzing and testing!

I have made an updated version that should handle all this (including the compatibility with existing configurations)

Available to download from link below

Hi,

Is there any chance you can add authentication into this plugin? I'm currently running it with a MQTT broker running in my LAN, but I would like to use it with adafruit.io too, and this requires authentication.

Kind regards,

Luke

Well, actually, after looking at the adafruit.io a bit, I think it would be better to create a separate plugin instead of adding too much to the existing one. There is a nice Adafruit IO Python Client Library available at github that I think can be used
https://github.com/adafruit/io-client-python

I do not use this myself so currently I have no plans for the effort required

What is the advantage/benefit of using adafruit.io compared to having your own broker?

Kind regards, Walter

lbretth wrote:Hi,

Is there any chance you can add authentication into this plugin? I'm currently running it with a MQTT broker running in my LAN, but I would like to use it with adafruit.io too, and this requires authentication.

Kind regards,

Luke

You could bridge your home MQTT broker to adafruit so you could exchange messages back and forth. Here's an example config for Mosquitto: https://gist.github.com/tdack/45dd356d9271a87914ce

This way you get the low-latency local access and offline capability that comes with a local broker, while allowing direct integration of external devices utilizing the adafruit broker. While I do feel like MQTT auth should probably be included here (it's part of the spec and the Mosquitto libraries, see "username_pw_set()" in the included Mosquitto.py), for your specific use case the bridge is probably what you want. The only advantage to using an external broker for all of your internal communication would be where you cannot deploy a local broker. In your case, you've already done so. Bridge them and you get the best of both worlds.

krambriw wrote:Thanks for analyzing and testing!

I have made an updated version that should handle all this (including the compatibility with existing configurations)

Available to download from link below

HOLY COW THAT'S FAST SERVICE!

I can confirm that all test cases outlined above (and my own use cases) all now work as expected using MQTT standard namespace conventions. Thank you so much!

edit: OK I did find one more problem. If you want to subscribe to ALL topics on a given broker, the subscription string is a simple "#". The plugin still won't accept this value, and as a result you cannot use EG to listen to all topics - only some subset of them. It'd be great if I could simply point it to "#" and then grab every topic/message that my broker is sending for possible handling by EG.

OK, new version available supporting subscription to # (this is however not recommended due to possible client overload if the broker handles many messages). See here: http://www.hivemq.com/blog/mqtt-essenti ... -practices

Still, I do not support single level wildcard (+) either, maybe not needed for the moment

Then something about authentication: I have understood that just using username/password is not recommended for internet connections, you need certificate based authentication. Implementing this goes anyway beyond my willingness spending substantial time on this. I did the MQTT Client plugin originally for the local LAN, not for connections over internet.

To overcome, one thing you could easily do is to use a RPi as a bridge. Install a MQTT broker. From your local LAN, use the EG MQTT Client plugin to connect and subscribe. Then use the library mentioned above (Adafruit IO Python Client Library) to connect and subscribe. With very little Python code running in the RPi you could subscribe and re-publish messages both ways between the MQTT broker and the Adafruit IO.

(Currently the Adafruit IO Python Client Library requires at least Python 2.7 (EG is using 2.6) so writing a plugin is for the moment a no-go)

I would like to use it with adafruit.io

Some good news on the way, I have a first stumbling plugin running, Adafruit MQTT Client

It is working but io.adafruit.com is not really yet what it should be (requires multiple connection attempts before succeeding etc etc). Is maybe good since it forces me to improve exception handling...

I just started using this plugin for communication with my Raspberry Pi, and it's working marvellously. Thank you!

Is it possible to make a new subscription persistent? I would like to send messages to my PC that arrive after it has booted, if it happens to be off when the message is published. If this is impossible, are there any workarounds? For my specific application I could probably send a message to the RPi after boot, and only send messages from the RPi after that, but a persistent subscription would be really neat.

Best regards,

Martin

Is it possible to make a new subscription persistent?

Hi there, great to hear you got it working with the RPi !

To your questions, just to clarify that I did understand you correctly, you would like to save messages persistent when & where they are created, so that they can be sent "later" to EG when EG is up and running?

First of all, the MQTT Client in EG should automatically try to re-connect to the broker if the connection is lost for some reason.

In MQTT you can publish events/messages flagged as 'retained'. This means that the MQTT broker will keep the last message flagged as such and re-send it to clients once they connect and subscribes. This is however only supported for one message per topic. If you are only interested in the the last message per sensor/device/whatever, this could be enough. To make it "as safe as possible", you have to implement it as close to the source as possible. Means you have to have this running in the RPi.

Assume you create a topic structure like "my_messages/subtopic_per_type". In EG you can configure your MQTT Client to subscribe to "my_messages/#" and you would receive all messages for the topic and the subtopics. A small python script running in your RPi that is listening to your messages and publishing them 'retained' to individual subtopics in the local broker will then do.

If you instead have a number of sequential messages for each type that you need to keep, the above is maybe not enough, you have to add a local repository that can be re-published on request. Again, a python script running in the RPi is very well suited for the task
- it has be able to connect and subscribe to the same topic/topics as the MQTT Client in EG (listening in parallel)
- uses shelve or pickle to maintain a list or dictionary of messages that arrives while EG is off line
- support commands via MQTT to publish all saved messages so that EG can synchronize (at startup or whenever...)
- support some kind of heartbeat from EG so the script knows when EG is off-line and it should start saving messages
- deleting the saved messages once EG is on-line again and saved messages have been published
....

Thank you, krambriw, for such quick and helpful support!

By a persistent subscription, I mean one where the subscription isn't terminated regardless of whether the client stays connected or not. If the client that established the persistent subscription is unavailable, any messages sent to the relevant topic will be logged by the broker to be sent to the client upon reconnection. I believe the term used in the MQTT documentation is a "clean session" (where a non-persistent subscription would be considered clean), and I think it's referred to by that name in the mosquitto.py file that's available on your plugin download page. My apologies if my explanation of the matter is lacking; I'm quite new to MQTT (read: started yesterday). My source of information, along with a better explanation, can be found here: http://www.hivemq.com/blog/mqtt-essenti ... g-messages

Initially, I didn't think a 'retained' message would work for my purposes, but reading your post, I realise it probably would. I'm going to have my RPi start the computer every now and then for file transfers. I'll want it to shut down again afterwards. I could probably implement this by publishing a retained message from the RPi to the topic "eventghost/last_wol_startup" or somesuch, with the time and date of the last WOL startup command. I could react to this message in EventGhost and compare the time and date with when the computer was started to determine whether that WOL command was the source of the boot. If it was, then I could shut down the computer after the file transfers are finished (these file transfers could also happen when the computer is on from the start, so I can't just shut it down after they're finished without checking the startup source). I will also need to interrupt said shutdown if I start using the computer after it's been automatically started, but that's a separate issue.

Your tips on sequential messages are very helpful! If comparing startup time with WOL command time does not work well, I'm considering having the RPi check whether the PC is on using MQTT (or maybe just pinging), and waiting for an answer. If it isn't on, a WOL startup command is sent, and the RPi can tell the PC to shutdown when it's done. This is where I thought a persistent subscription would be useful, as I could send that message without worrying whether the PC had had time to start yet. However, I can just wait for a startup message triggered by the event Main.OnInitAfterBoot in EG, before sending the shutdown message. I will look into using a python script to react to messages if I end up doing something like this.

Thank you again for your help!

Hello,
I get your point, it would of course be very convenient using clean_sessions. Honestly, I did not know too much of all MQTT features when I once wrote the plugin. There is always room for improvements.
No promises made...

If I really need them, I could probably bridge a mosquitto broker on the PC to the one on the RPi, make that connection persistent, and then send all messages through there, so no problem.

Again, thank you for your help and the great plugin!

I took the challenge and migrated the MQTT Client plugin to support also persistent connections. At the same time I also used the newer library named Paho (Mosquitto is since a couple of year superseded by Paho)

After testing quite a lot, I think it works well (no bugs what I can find). When you are creating a subscription, you now have the possibility to select that a persistent connection shall be established. This works pretty nice when messages are published to the broker in case the client happen to be off-line (works for messages published with QoS = 1,2). When you re-connect, you will receive those messages that happened while you where off-line.

So now how to upgrade from previous version?? I think there are two ways you can do it

1) Not so fun way
- Delete the older plugin and all actions you have configured
- Close down EG
- Replace the plugin files, restart EG
- Add the plugin and actions again and configure them

2) If you know what you are doing, this is a clever way that will save you time and effort.
- Replace the plugin files, restart EG (you will get a number of errors in your log)
- Make a new configuration holding only the MQTT Client plugin and one of each action types
- Save and close down EG
- Open both the new configuration and the previous one (xml-files) in a good editor
- Replace the entry for the plugin in the old file with the new
- Look at the action structures and compare the old with the new
- Edit each old accordingly so the structure becomes like in the new
- When done, save and close the files
- Restart EG and you should be fine, your configuration should work as before
- If you like, you can change subscription types to persistent connections and try it out

If you get some error message that Paho is missing or something related, try to copy the paho subfolder to /lib26/site-packages folder and restart EG
EDIT: Actually, I think you need to do this, copy or move the paho subfolder with all content as mentioned

This new version should have all features of the old one. I have anyway also kept the old version available for download for those who do not want to migrate. But I will target future maintenance to the new one.


Good luck & best regards

Thank you krambriw for the plugin. I am trying to install the new paho plugin. First when I was adding the plugin the eg was crushing . Then I moved the paho directory to sit-pakages and I was able to add the plugin.
Now when I want to subscribe this happens:
09:25:21 Exception in thread malinka:
09:25:21 Traceback (most recent call last):
09:25:21 File "threading.pyc", line 532, in __bootstrap_inner
09:25:21 File "C:\Program Files (x86)\EventGhost\plugins\MQTT Client\__init__.py", line 309, in run
09:25:21 resp = mqttc.connect(self.host, self.port, 60)
09:25:21 File "C:\Program Files (x86)\EventGhost\lib26\site-packages\paho\mqtt\client.py", line 612, in connect
09:25:21 return self.reconnect()
09:25:21 File "C:\Program Files (x86)\EventGhost\lib26\site-packages\paho\mqtt\client.py", line 732, in reconnect
09:25:21 sock = socket.create_connection((self._host, self._port))
09:25:21 File "socket.pyc", line 514, in create_connection
09:25:21 error: [Errno 10061] No connection could be made because the target machine actively refused it
Any help is appreciated.