EventGhost

This is a fix for the broadcaster plugin. the plugin when receiving data doesn't check to see if the data is within the decimal range of 32 - 127, and by not checking this when triggering an event throws an unhandled exception and drops the bound port it's listening on stopping the plugin from working. wich you would either have to disable and then enable the plugin or restart EG to get the plugin listening again.

I have added handling for the exception as well as having an error printed stating that the data is outside of printable range and also includes the senders IP address. it also searches the data for the bad character(s) and converts them to hex values. the traceback and reformatted message and the senders ip are sent off to be printed in the debug log file.

by catching this error the plugin is able to continue along like nothing ever happened.

I have attached the modified plugin.

this code has not been fully tested. as it would take some time for me to cause this problem intentionally . there is a forum member who now has the new code and will hopefully tell me that it works properly. this problem can take weeks to show in their setup. so i figured i would post it in the off chance someone else may have had the same issue and might shed some light on if it's working or now. and hopefully have a solution to their problem.

this code does compile without error.

the file is attached. just download it and copy it into the C:\Program Files (x86)\EventGhost\plugins\Broadcaster if that is where your installation of EG is located. overwrite the existing file. and restart EG. Everything functions the same save for the plugin dropping connection if bad data is received.

This workaround is just treating the symptoms of an encoding snafu. See related issue with the Web server.

viewtopic.php?f=4&t=9455&p=43562#p43562

Thanks!

And I apologize, but I haven't had a chance to replace the Broadcaster plugin. My workaround that monitors UDP activity and restarts the plugin has been working fine and the related Web server issue was far more important. I'll get to the Broadcaster eventually.

Instead of catching the exception, a better solution for the moment would be to either replace offending characters through transliteration or simply remove them. Of course, unlike HTTP, I can't think of a particularly compelling reason to pass non-ASCII characters to EG via UDP. Not even sure why VC was passing them; expect it misheard a voice command or just barfed for some reason. AFAIK, there's nothing else on my network sending UDP on the monitored port. As you mentioned, it could go weeks or months between occurrences.

Side note: recommend against VC, particularly if sending UDP from EG. It's an absolute disaster in that regard (ironic as that was half of its initial reason for being). Appears to have threading issues, at least when sending emails and have stopped using it for that. But the receiving end snafu appears to be down to EG's encoding/decoding woes.

I had to do the transliteration trick to deal with the Serial plug-in talking to alphanumeric signs that can't display characters > 127.

The simplest answer I found worked to get rid of accents and such and added some regular expression replacements for curly quotes, hyphens, etc.

import unicodedata
def strip_accents(s):
return ''.join(c for c in unicodedata.normalize('NFD', s)
if unicodedata.category(c) != 'Mn')

http://stackoverflow.com/questions/5179 ... ode-string

Of course, there's a lot more discussion after that simple example, but not sure if I need it for English. Just wanted to make sure that "se├▒or" displayed on the signs as "senor" and that worked. Before this workaround, the Serial plug-in would just throw an exception for non-ASCII characters.

HTH

After reviewing the stack trace from the original thread and the EventGhostEvent.py script on Github, I believe this is the line that is throwing the exception:

self.string = prefix + "." + suffix

Like the fix for the Web server, I believe this will do it:
...You'll have to fix the case for commandSize > 2 as I'm not yet fluent enough in Python. After that you should be able to remove the previously added exception handling. Easiest way to test it is to send a broadcast with an offending character. If you can fix the last case I'll try it out.

If I replace the PY and restart the plug-in, will it pick up your changes? I'd prefer not to restart EG itself unless I have to (usually about once per year). It's basically part of my house at this point; typically Windows dies and reboots before EG runs into any problems; last time took about 6 months.

http://stackoverflow.com/questions/3177 ... C2%A3-u1-t

Good luck!

I just searched the forum for "Unicode" and it's strangest thing, kgschlosser: you posted a similar fix back in 2015, except you used str() instead of decode(). Did you forget about that?

Neither of your fixes are in GitHub. Will add if you can fix that last case (where payload length > 2). Also, in your str() fix you only did the first argument in each TriggerEvent call; wouldn't all of the arguments need the same treatment?
viewtopic.php?f=9&t=9453&p=43542&hilit=unicode#p43542

Actually, not a year ago. I must have been tired when I read that other thread. November 15 of *this* year, so just a few days ago.

I just commented there as well. This can be wrapped up entirely by finishing what I started above.
What's the last line?

I am going to run some tests on the trigger event.. and yeah, you just up and vanished so never really got o know if it solved the problem or not...


and decode won't work. we have to use encode.

and i just posted something in the other thread about it.. I am going to ask in the other thread if i can merge these 2 since they are duplicates of one another

actually i am going to merge this thread with the other because it's a plugin related issue and shouldn't be in this thread...


but i also attached a new version of the plugin as well

viewtopic.php?f=9&t=9453&p=43733#p43733

and there is a link in the other thread with a tutorial about unicode you may want to breeze through... it made the light bulb go on for me..

kgschlosser wrote:I am going to run some tests on the trigger event.. and yeah, you just up and vanished so never really got o know if it solved the problem or not...

That was a workaround anyway. This is the fix.

kgschlosser wrote:and decode won't work. we have to use encode.

As discussed in another recent thread, you've got that backwards.

kgschlosser wrote: and i just posted something in the other thread about it.. I am going to ask in the other thread if i can merge these 2 since they are duplicates of one another

Perhaps that was the one I just read. In any event, the solution here requires decode not encode. See my reply in the other thread.

Okay, I have finished the last bit of this after a little Google searching to find a "forEach" in Python and testing the resulting solution in a Python terminal.
One huge problem with this project is that the documentation for Python programmers is virtually nonexistent.

For example, what is self.payDelim supposed to be? Note that I removed the str() call around that as it would prevent the use of non-Ascii characters. Seems like any Unicode string should be allowed for that property. Before removing str(), any non-ascii characters would cause an exception similar to what we've been seeing throughout.

By the same token, what are the event name and payload allowed to be? Again, I suggest any Unicode string. Finally, what are the raw bytes received from UDP allowed to be? I suggest UTF-8 encoded Unicode. Same as with HTTP, except in the case of UDP there is no way to specify an alternate encoding. We need to set down these rules and follow them. Any code that breaks (if any) needs to be updated to follow these rules. As we've seen, without rules we get complete chaos and confusion.

The main point is that once bytes are received via HTTP, UDP or whatever, they must be decoded to Unicode strings. It follows that we must not pass anything other than Unicode strings around EG for event names, payloads, etc. I'm not suggesting we ever did or allowed for that, but sure somebody somewhere has tried it and may well have "succeeded" due to a lack of rules and consistency in how the core and plugins interact with user code. In my case, the opposite happened in that I "failed" repeatedly because my code had no clue what to expect from the plugins. The only way I could "fix" the issues was to hack around with encoding and decoding (things users should never need to do).

Understand that my own workarounds will break once these rules are in place for received UDP and HTTP. I'll have to remove the workarounds and all will be well again. Others may have to do the same, but this will not indicate a failing in the suggested rules and fixes.

I would welcome exception handling where appropriate, but should not do anything but print errors. For example, decoding steps should be wrapped in a try-except that print an error and bypasses the TriggerEvent on the event of an exception. Once we have successfully decoded the bytes to Unicode strings, we know that TriggerEvent will work, so no try-except is needed for that call. On the other side, TriggerEvent should have its own exception handling.

Using try-except as a probe to see if the result "worked" is a non-starter. We must define and document what TriggerEvent expects for event names and payloads (e.g. Unicode strings) and plug-ins should pass nothing else (e.g. byte arrays). We aren't going to get anything but Unicode strings out of HTTP after the next update, so might as well make UDP consistent with that. Why we would ever want to pass around anything else for event name and payload (or delimiter) is beyond me; just makes things much more difficult (and less flexible in case of the delimiter) than they should be.

Suggestion for exception handling: ...You can expand on that to make it print the offending "bits" (though take care that printing doesn't cause another exception) and exiting before the TriggerEvent call. As mentioned, I'm not really a Python programmer, so not sure of the best way to accomplish this. Perhaps setting an "errored" flag and only proceeding with the TriggerEvent call if it is not set?

On the TriggerEvent side, the only line I know of that will throw an exception when passed raw bytes is the concatenation between suffix and prefix. You might look at that to see if there are other cases as code besides this plug-in could pass incorrect arguments to that method. That's why I think wrapping the TriggerEvent call in a try-except is the wrong way to go here. For this plug-in, we just need to make sure we pass it Unicode strings.

As with HTTP, created a pull request for this on GitHub:

https://github.com/EventGhost/EventGhost/pull/119

...And a companion for the send bit:

https://github.com/EventGhost/EventGhost/pull/120

That should do it. If non-Ascii characters are in the event name and/or payload, they are encoded with UTF-8 on send and decoded on receive. Seems impossible to break existing installations (unless they've developed "scabs" to deal with undocumented behavior as described above) as we already know the receiver blew up on reception of non-Ascii characters. Users (e.g. myself) may have added their own encoding/decoding workarounds to deal with the previous issues (and lack of documentation) and they will have to remove such workarounds on upgrading EG. Make sense?

PS. As noted on the pull requests, we may want to use eg.systemEncoding for this, rather than hard-coding UTF-8. This was not an option for the Web server fixes, as HTTP specifies the encoding with a standard default of UTF-8.

HTH

for you specific use case this should be a specific way. i am thinking about all of the various things that can send data and that the user may not have any control over. and using the system encoding wouldn't help for instance if the system encoding is for utf8 and the incoming data is coming in as latin1

it will still fail. just because someone's system encoding is for place doesn't mean that anything they may want to have come into eg is going to be encoded the same way and not from another place.


there are way to many devices/software out there to make that assumption

kgschlosser wrote:for you specific use case this should be a specific way.

Not sure what you mean, but my ugly workarounds for similar HTTP issues are going to be deleted, as will the Broadcaster monitor I built to restart it after a timeout. These were in place to deal with bugs in EG and won't be needed once the pull requests are accepted (assuming they ever are).

kgschlosser wrote: i am thinking about all of the various things that can send data and that the user may not have any control over.

Well, it's really simple at the moment: EG blows up (without silent failure workarounds like we've suggested in the past) on any non-Ascii characters and my fixes don't affect Ascii characters.

kgschlosser wrote: and using the system encoding wouldn't help for instance if the system encoding is for utf8 and the incoming data is coming in as latin1

If the user specifies the system encoding to be UTF-8 and uses some other sender (or even another EG) that is configured to use an encoding of "latin1", then they have a problem (and an easy one to solve). There's nothing we can do about that (other than to log errors).

kgschlosser wrote: it will still fail. just because someone's system encoding is for place doesn't mean that anything they may want to have come into eg is going to be encoded the same way and not from another place. there are way to many devices/software out there to make that assumption

Yes, and that's a very easy problem to solve (see above). Remember, that none of this stuff worked at all with non-Ascii characters before these fixes. In fact, all non-Ascii characters caused exceptions that took down the Broadcaster. Now that we have a fix, let's not worry too much about other things that we can't control. Furthermore, this is in line with what Network Receiver and Sender do. If you think about it, there's nothing else that we can do.

davidmark wrote:Okay, I have finished the last bit of this after a little Google searching to find a "forEach" in Python and testing the resulting solution in a Python terminal.

Code: Select all

if (not my_addr) or self.selfBroadcast:
            bits = data.split(self.payDelim)
            bits = [bit.decode('utf-8') for bit in bits]

            commandSize=len(bits)
            if commandSize==1:
                self.plugin.TriggerEvent(bits[0])
            if commandSize==2:
                self.plugin.TriggerEvent(bits[0],bits[1])
            if commandSize>2:
                self.plugin.TriggerEvent(bits[0],bits[1:])

H

I just got a new flavor of error msg: EG is choking on a new byte! I'll try your workaround snippet for this... restarting the plugin didn't work for me unfortunately


edit: your workaround quoted above didn't work for me. I immediately got the same error when trying to receive a UDP message just now. Same byte 0xef

Okay, there's no exception handling in there yet and there's no exception handling in TriggerEvent either (clearly needs at least one try-except as Broadcaster is not the only code that calls it).

What exactly did you send for the event name and payload? And what did you send it from? I created one pull request for receiving events (the one you quoted) and one for sending. Both need to be in place, else I have no idea what a round trip will do.

Looking at the code in EventGhostEvent.py, it certainly looks like this line (87 on GitHub's version, not 84 as in the quoted stack trace).

self.string = prefix + "." + suffix

Post exactly what was sent (and how) and I'll run some more tests...

Thanks!

This article may contain a clue:

http://stackoverflow.com/questions/8481 ... or-on-join

Edit:
...And actually just confirms that the decoding worked as expected: u'\xf6' is the same as u'├Â'.

Testing in the Python console, I'm not able to duplicate the exception by concatenating with '.', so maybe I've got the wrong line in TriggerEvent (line numbers are not the same on GitHub as 84 is the function declaration). May just be reading the stack trace wrong as I'm not a Python expert. But no question this is a problem in TriggerEvent now...

Edit again:

I see that it is actually \xef that you have (not \xf6). Let me try again. Can somebody confirm what line it is that's throwing the exception. Clearly it is at some point after the TriggerEvent call in the Broadcaster plug-in...

And again:

u'\xef' is 'ï', so can you confirm that your text had that character in it? If so, that confirms that the decoding went as expected and passed a proper Unicode string to TriggerEvent. Question is what happens after that, which is unrelated to the Broadcaster code.

Here are lines 84-87 in EventGhostEvent.py on GitHub:
I incorrectly stated above that 84 was the function declaration. It's actually the first line above, which couldn't be the culprit. I assumed that the line numbers were a bit off (and seems they must be) and figured that line 87 was the culprit. Can you confirm what line 84 in EventGhostEvent.py is on your system? If I'm reading the stack trace correctly, that's the line that is having a problem with the Unicode string. Can't speculate as to why at this point, other than some line somewhere in that module (assuming the stack trace is accurate) is causing Python to do an implicit decode using the Ascii default.

Can create similar errors this in the console like this:

u'\xef'.decode('ascii')

...Or with the default:

u'\xef'.decode()

...Or even:

u'\xef'.decode('utf-8')

...As trying to decode a Unicode string (as opposed to encoded bytes) causes an implicit Ascii encode (see linked article).

HTH