Broadcaster Plugin Received Data Fix

[skribb]

self.string = prefix + "." + suffix

Post exactly what was sent (and how) and I'll run some more tests...

Thanks!

I will do it tomorrow.

The message is from one of my tablets which I call Origin.... and i have devised Origin in such a way that right now I can't access anything on it, I'm waiting for it to reboot (I have a reboot on a timer).
So once it's rebooted I can see what message it sent because I don't remember it by heart. Will keep you posted.

[davidmark]

Aha! I see that my console is Python 2 and IIRC, EG uses Python 3. That makes a big difference, so perhaps that concatenation line in EventGhostEvent is the culprit after all. Not sure what else it could be, but would be nice to know exactly which line in there is throwing that exception (and I'm not near EG at the moment to check).

[davidmark]

Can I ask what version of EG you have as I just took a moment and ran a few tests on mine and these all worked as expected:

eg.TriggerEvent(u'\xef');

eg.TriggerEvent(u'\xef', [u'\xef']);

eg.TriggerEvent(u'\xef', [u'\xef', u'\xef']);

eg.TriggerEvent(u'\xef', [u'\xef', u'\xef', u'\xef']);

...Even tried this:

eg.TriggerEvent(b'\xef', [b'\xef', b'\xef', b'\xef']);

...And this:

eg.TriggerEvent('\xef', ['\xef', '\xef', '\xef']);

...No issues. As expected, I see Main.ï in the log for the event name in each case and the appropriate payloads.

Can you do the same and confirm there are no issues? Going to also need to know which line in EventGhostEvent.py is throwing the exception. Please post whatever you find at line 84. I'd look in mine, but don't want to assume anything at this point. I suspect you'll find it is the concatenation I mentioned before. I played around trying to reproduce that in EG and only way I could do it was to change it to something like:

print b'\xef' + u'.';

...And there's no way that's happening as the '.' in that line (at least on GitHub) is not a Unicode literal and if you are using my update to Broadcaster, it is definitely not passing any bytes. Very strange, but then I have no idea exactly what is going on in your installation. Let me know...

Once we are sure what line 84 is, I want to add a print right above it, for example:

Code: Select all

print suffix

That, along with the above tests will tell the tale. Seems like we are making progress (assuming you sent something with "ï") as at least the correct character made it to TriggerEvent. Before you would have been getting the dreaded A with a hat or some such.

Thanks!

[skribb]

I thought EG used Python2

anyway I checked and the message sent from my tabelt is t1.reboot_in_2hours

now, I tried sending it manually from my tablet and it was received in EG without error now, so I dunno what the hell that was.

But as with my other message that choked on 0xc3, the UDP messages that I send to EG do not contain that actual character...
Funny thing is if you search for 0xef you get 111 000 results pointing to very similar Python problems https://www.google.com/search?q=byte+0x ... e&ie=UTF-8 so I'm sure this isn't just EG being cranky

Is the crux of the matter that EG/asyncore tries to encode the received message and then decode it again?


edit: here is the entire Def thingy from my EventGhostEvent.py:

Code: Select all

    def __init__(self, suffix="", payload=None, prefix="Main", source=eg):
        self.string = prefix + "." + suffix
        self.prefix = prefix
        self.suffix = suffix
        self.payload = payload
        self.source = source
        self.time = clock()
        self.isEnded = False
        self.shouldEnd = Event()
        self.upFuncList = []

[davidmark]

I thought EG used Python2

Maybe it does. Not sure.

anyway I checked and the message sent from my tabelt is t1.reboot_in_2hours

Well, that's not the one that caused the issue.

now, I tried sending it manually from my tablet and it was received in EG without error now, so I dunno what the hell that was.

I would expect not.

But as with my other message that choked on 0xc3, the UDP messages that I send to EG do not contain that actual character...

The 0xc3 indicates that the special character was not decoded. We've fixed that now, which is why you get the correct 0xef character, which is ï. Are you never intentionally sending "'ï'" to that UDP port? Something is, just as something periodically sent some screwy characters to mine (causing identical exceptions). Do you run VoxCommando by any chance?

Funny thing is if you search for 0xef you get 111 000 results pointing to very similar Python problems https://www.google.com/search?q=byte+0x ... e&ie=UTF-8 so I'm sure this isn't just EG being cranky

Cranky has no technical meaning. As for the 111,000 results, they are mostly related to Python programmers having trouble with Unicode encoding and decoding. It's a common problem, but there's always a reasonable explanation. I'm going to swap out by Broadcaster plugin tomorrow and try sending some non-Ascii characters, but after testing TriggerEvent manually, I'm not expecting an exceptions on passing those characters for the event name or the payload. Please try out the tests I posted on your system.

Is the crux of the matter that EG/asyncore tries to encode the received message and then decode it again?

No, your second exception (with the 0xef) confirms that Broadcaster is correctly decoding the character and passing it to TriggerEvent, which is choking on the non-Ascii character, just as it did the encoded version.

Consider this console output:

Code: Select all

>>> u"\xfe".encode("utf8")
'\xc3\xbe'

...It's what you were receiving before (0xc3 being the first non-Ascii character).

Now consider the reverse:

Code: Select all

>>> b"\xc3\xbe".decode("utf8")
u'\xfe'

...That's what TriggerEvent is getting now, which is correct. However, if you didn't send the "'ï'" on purpose, that's one mystery, particularly if you only ever send ASCII characters. I suppose it's possible Asynccore has some crazy bug that periodically turns plain old Ascii into bytes representing UTF-8 encoded Unicode, but I doubt it. The other mystery is why TriggerEvent is throwing that exception. That's beyond the scope of Broadcaster and we need to concentrate on implementing logging and exception handling in EventGhostEvent.py at this point.

edit: here is the entire Def thingy from my EventGhostEvent.py:

Code: Select all

    def __init__(self, suffix="", payload=None, prefix="Main", source=eg):
        self.string = prefix + "." + suffix
        self.prefix = prefix
        self.suffix = suffix
        self.payload = payload
        self.source = source
        self.time = clock()
        self.isEnded = False
        self.shouldEnd = Event()
        self.upFuncList = []

[/quote]

On mine, line 84 (which also throws exceptions once in a blue moon for no apparent reason) is:

Code: Select all

self.prefix = prefix

...Which doesn't seem like it could be the culprit. Can anyone in a forum read a Python stack trace with confidence? If so, how in hell could that line throw such an exception? Could the line count be off somehow as I would think the concatenation would be the only possible cause (assuming the error happened here at all). Maybe I miscounted as I had it in an editor without line numbers. I'll check again tomorrow, but it's got to be this:

Code: Select all

self.string = prefix + "." + suffix

...So we need to add this above it:

Code: Select all

print suffix;

...As I can't come up with anything for suffix (or prefix) that would cause that line to fail in any way. I've tried repeatedly calling TriggerEvent with all sorts of Unicode and byte combinations and nothing breaks it at all. Have also played around in the console with that expression and gotten nowhere. Only clue is that if you change it to:

Code: Select all

self.string = prefix + u"." + suffix

...It's easy to get the very exception we are looking for, but only if you had NOT applied my fix as prefix or suffix would have to be bytes rather than a Unicode string (for obvious reasons). That is what would happen if you edited the file but didn't restart the plugin, but then there's no "u" prefix in the above line. Just grasping for straws there.

Maybe this has something to do with Python's default system encoding or some such. Afraid I'm no expert in this area. Apparently nobody involved with this project is.

Regardless, that line is going to need a try-except until we can pinpoint just what can be sent to TriggerEvent to cause it to throw an exception (and fix it). Near as I can tell, there's nothing wrong with Broadcaster at this point, just that the fixes that decoded the characters didn't appear to fix the TriggerEvent error.

I'll know more once I try all this myself tomorrow. I'll make VC send various non-Ascii characters and see what happens. At least that thing is good for something.

[kgschlosser]

i am going to bet you dollars and cents tht this issue is in the core of EG.. i bet it uses the system encoding in triggerevent which will cause all kinds of hairy problems. like if you encode/decode something in utf8 and the system encoding is set to something different and then trigger event tries to encode/decode it that is why this is happening..

would have to look at the core code to see if that is the case

[kgschlosser]

issues like this will show up in situations like this...

this is an example.
you live in spain. but you are english speaking... and you have a spanish version of windows but the language is set to english

because there are 2 different ways to obtain system encoding and language. and using locale will report wrong information in this specific scenario and EG does use locale

the 2nd way is by use of ctypes

i do not know if your setup is like this skribb

but i do also find it very odd that if you manually send the event it works just fine. what are you using to send the event??? automatically and manually???

[skribb]

Are you never intentionally sending "'ï'" to that UDP port? Something is, just as something periodically sent some screwy characters to mine (causing identical exceptions). Do you run VoxCommando by any chance?

That's what I'm saying. So the question is where is that character coming from.

I do not use voxcommando. could it be that the ï is some kind of byproduct of the UDP message? Maybe the UDP Sender plugin in Tasker (which is waht i'm using) could be at fault?

Is it possible at all that it could be some kind of drive-by message? Such as when I set up my EG webserver, I noticed that a bunch of unauthorized IP addresses tried to gain access to my server, so it does seem as soon as you set up a networking service, net sniffers get the scent of your machine..... could it be a similar phenom? Random machines just spitting out UDP messages hoping that something catches it?


@KG: in Tasker automatically and manually is the same thing, at least programmatically I think. when it's automatic it is acting on an autoremote message from my EG by sending back a UDP message. if it's manual I launch that specific task by hand by using the "Test" button in Tasker. and it is UDP Sender plugin in Tasker .

Bottom line is I am not sending any non-english characters and EG thinks I am. The ï isn't even a character used in Swedish so I wouldn't even be able to type it in somewhere by accident.


I am not knowledgeable enough to know how relevant it is, but my system language is english and my locale is Sweden according to "region and language" in Win7

[davidmark]

i am going to bet you dollars and cents tht this issue is in the core of EG.. i bet it uses the system encoding in triggerevent which will cause all kinds of hairy problems. like if you encode/decode something in utf8 and the system encoding is set to something different and then trigger event tries to encode/decode it that is why this is happening..

would have to look at the core code to see if that is the case

There's no sense guessing as we have a stack trace (unless Python stack traces are unreliable).

I've been all over the TriggerEvent code and it doesn't encode or decode anything explicitly. And, as mentioned, we can use the eg.systemEncoding instead of UTF-8 for this (but not HTTP).

Furthermore, as I expected, the Network Sender/Receiver are encoding and decoding as I have proposed. For some reason Broadcaster left this out. Putting it in is a good thing, but it may not solve whatever TriggerEvent's problem is. We'll know more once Skribb runs the TriggerEvent tests I posted (and perhaps adds some print statements in EventGhostEvent.py so we can see just what exactly is causing the exception in there).

Here is the relevant code from Network Sender:

Code: Select all

# now just pipe those commands to the server
            if (payload is not None) and (len(payload) > 0):
                for pld in payload:
                    sock.sendall(
                        "payload %s\n" % pld.encode(eg.systemEncoding)
                    )

            sock.sendall("payload withoutRelease\n")
            sock.sendall(eventString.encode(eg.systemEncoding) + "\n")

...I don't use that plug-in, but it's a good confirmation that the Broadcaster fixes are appropriate (perhaps using eg.systemEncoding instead of UTF-8). Regardless, I can just about guarantee that no matter what encoding we use it isn't going to stop TriggerEvent from throwing its exception. Whatever it is can be recreated simply by calling TriggerEvent and passing whatever data causes the exception. We need to concentrate on finding out what that data is and then we can put a stop to this issue for all plug-ins. After that, I'd like to find out where this random gibberish is coming from; I expect a bug in asyncore and/or EG's interface with it.

[davidmark]

issues like this will show up in situations like this...

this is an example.
you live in spain. but you are english speaking... and you have a spanish version of windows but the language is set to english

because there are 2 different ways to obtain system encoding and language. and using locale will report wrong information in this specific scenario and EG does use locale

the 2nd way is by use of ctypes

i do not know if your setup is like this skribb

but i do also find it very odd that if you manually send the event it works just fine. what are you using to send the event??? automatically and manually???

As mentioned, we can use eg.systemEncoding (instead of hard-cording UTF-8) and we don't yet know skribb's results from sending the various test arguments to TriggerEvent. No sense guessing as will just drive you crazy.

I sent the event from a Python module by calling eg.TriggerEvent, same as the plug-in does. I don't know what you mean by automatically and manually. (?)

[davidmark]

Are you never intentionally sending "'ï'" to that UDP port? Something is, just as something periodically sent some screwy characters to mine (causing identical exceptions). Do you run VoxCommando by any chance?

That's what I'm saying. So the question is where is that character coming from.

Mine too, but that's beside the point as anything could send anything to TriggerEvent. We have to make sure it has exception handling for any harmful cases or fix whatever it is in there that fails for these cases. I've been all over the thing and I don't see any issues. Adding prints in there will tell the tale; at this point, we are just guessing.

I do not use voxcommando. could it be that the ï is some kind of byproduct of the UDP message? Maybe the UDP Sender plugin in Tasker (which is waht i'm using) could be at fault?

Highly unlikely as I certainly don't use that and we are both experiencing random gibberish in received UDP packets. I'm starting to lean towards the asyncore and/or EG's interface with it. Regardless, we might as well forget about UDP for the moment; we solved one glaring issue with it (decoding the received bytes to Unicode strings), but there's still something wrong in TriggerEvent. Please try the tests I posted that call TriggerEvent from EG. Just create a Python action item and run it with the code provided.

Is it possible at all that it could be some kind of drive-by message? Such as when I set up my EG webserver, I noticed that a bunch of unauthorized IP addresses tried to gain access to my server, so it does seem as soon as you set up a networking service, net sniffers get the scent of your machine..... could it be a similar phenom? Random machines just spitting out UDP messages hoping that something catches it?

It could be anything, but all we should be concerned about at this point is logging the offending event name and/or payload and avoiding the uncaught exception in TriggerEvent that takes down the Broadcaster (or does God knows what to other plug-ins unfortunate enough to try to trigger events with similar data).

@KG: in Tasker automatically and manually is the same thing, at least programmatically I think. when it's automatic it is acting on an autoremote message from my EG by sending back a UDP message. if it's manual I launch that specific task by hand by using the "Test" button in Tasker. and it is UDP Sender plugin in Tasker .

I didn't understand that question from KG. But it's all beside the point. An exception in TriggerEvent is the problem now as anything could call TriggerEvent with any data. It shouldn't have uncaught exceptions in any event (no pun intended).

Bottom line is I am not sending any non-english characters and EG thinks I am. The ï isn't even a character used in Swedish so I wouldn't even be able to type it in somewhere by accident.

I am not knowledgeable enough to know how relevant it is, but my system language is english and my locale is Sweden according to "region and language" in Win7

And your eg.systemEncoding is likely cp-1252 just like mine and that is likely just another red herring as TriggerEvent doesn't do any sort of explicit encoding and decoding that I can find.

Please try the TriggerEvent tests from EG and let me know if any of them throw exceptions. Also, if you could add some print statements to EventGhostEvent.py, that will help us see just what in hell is being passed that causes TriggerEvent to throw an exception. Something like:

Code: Select all

print suffix;
print prefix;

...at the top of the init method. I'll do some more experimenting here in the meantime. Thanks!

PS. I use Tasker extensively and always send everything via HTTP. Far more reliable as UDP doesn't guarantee delivery and I've never seen the Web server do anything that caused asyncore to shut down the socket. I've only ever used UDP with VC; works fine when receiving from that, but sending to it is a disaster (VC seems to have threading issues, at least when trying to send emails). I've since added Python code to EG to send emails and all is well.

[skribb]

Highly unlikely as I certainly don't use that and we are both experiencing random gibberish in received UDP packets. I'm starting to lean towards the asyncore and/or EG's interface with it. Regardless, we might as well forget about UDP for the moment; we solved one glaring issue with it (decoding the received bytes to Unicode strings), but there's still something wrong in TriggerEvent. Please try the tests I posted that call TriggerEvent from EG. Just create a Python action item and run it with the code provided.

PS. I use Tasker extensively and always send everything via HTTP. For more reliable as UDP doesn't guarantee delivery and I've never seen the Web server do anything that caused asyncore to shut down the socket. I've only ever used UDP with VC; works fine when receiving from that, but sending to it is a disaster (VC seems to have threading issues, at least when trying to send emails). I've since added Python code to EG to send emails and all is well.

Sorry, I wasn't on the same frequency as you. now I understand that you're trying to collaborate a solution. My head is too far up my ass lol.

I will run the tests ASAP, expect a day or two. I will post follow up questions if i'm having any trouble.

I started out using HTTP Get from Tasker but since I had all those unahtorized IPs trying to get to my server I freaked out and stopped using it should probably get a proper hardware firewall for that

[davidmark]

Highly unlikely as I certainly don't use that and we are both experiencing random gibberish in received UDP packets. I'm starting to lean towards the asyncore and/or EG's interface with it. Regardless, we might as well forget about UDP for the moment; we solved one glaring issue with it (decoding the received bytes to Unicode strings), but there's still something wrong in TriggerEvent. Please try the tests I posted that call TriggerEvent from EG. Just create a Python action item and run it with the code provided.

PS. I use Tasker extensively and always send everything via HTTP. Far more reliable as UDP doesn't guarantee delivery and I've never seen the Web server do anything that caused asyncore to shut down the socket. I've only ever used UDP with VC; works fine when receiving from that, but sending to it is a disaster (VC seems to have threading issues, at least when trying to send emails). I've since added Python code to EG to send emails and all is well.

Sorry, I wasn't on the same frequency as you. now I understand that you're trying to collaborate a solution. My head is too far up my ass lol.

No problem. BTW, here's a better way to track this by printing the received bits so you don't clog up your EG log too badly:

Code: Select all

if (not my_addr) or self.selfBroadcast:
            bits = data.split(self.payDelim)
            bits = [bit.decode('utf-8') for bit in bits]

            print bits

            commandSize=len(bits)
            if commandSize==1:
                self.plugin.TriggerEvent(bits[0])
            if commandSize==2:
                self.plugin.TriggerEvent(bits[0],bits[1])
            if commandSize>2:
                self.plugin.TriggerEvent(bits[0],bits[1:])

I will run the tests ASAP, expect a day or two. I will post follow up questions if i'm having any trouble.

Thanks!

I started out using HTTP Get from Tasker but since I had all those unahtorized IPs trying to get to my server I freaked out and stopped using it should probably get a proper hardware firewall for that

Yes, you should configure your router so that your local network is secure (e.g. no port forwarding). Without that, all bets are off.

Thanks again! We'll fix it one way or the other. It's just a matter of finding the best place to put the try-except to prevent the uncaught exceptions that kill the socket. I'm sure that should be inside TriggerEvent as anything could send anything to that method at any time (the Broadcaster just happens to be the one that is causing it in such a way that it the problem becomes glaring).

I'm currently using a monitor that disables and then re-enables the plug-in and would like to jettison that myself. I have no doubt that I still share the same issue as you, though perhaps not as frequently (and BTW, VC sends UDP all day long for X10 received via AHP and occasional voice commands received through the PC microphone). Whether VC or Tasker or whatever, something is causing asyncore to puke periodically (at least that's my theory at this point). As I now suspect asyncore, I really should change VC to use HTTP as well, but near as I can tell there is no simple way to send HTTP requests from that PoS (short of creating a PY file and having it load and execute it each time). Suppose I should ultimately look into doing the AHP and voice commands through EG and dump VC altogether, but one step at at time.

[skribb]

stuff

No port forwarding is active, and I actually have this set on my router: Blocked - No access to local network from Internet, except as configured in the Port Forwarding.
but it doesn't seem to actually secure my LAN to a sufficient degree.

man I gotta sleep now.......

[davidmark]

stuff

No port forwarding is active, and I actually have this set on my router: Blocked - No access to local network from Internet, except as configured in the Port Forwarding.
but it doesn't seem to actually secure my LAN to a sufficient degree.

man I gotta sleep now.......

Then either you need a new router (or updated firmware) or some person or device is hacking around your local network looking for HTTP servers. Hard to say.

I know the feeling on sleep. I'll post an update tomorrow that has a (temporary) exception handler and prints any errors so that you won't have to restart the Broadcaster each time something bad happens on its TriggerEvent call. Just want to stress that it should be temporary until one of use in this thread (or forum, hello?!) figures out what should be trivial to track down: what line in EventGhostEvent.py could possibly be throwing an uncaught exception and under what circumstances.

Thanks again!