EventGhost

Posted: **Mon Nov 21, 2016 6:54 am**

and the encoding does not take place in the Trigger event...

it takes place in the log. where it is displayed

i do have a question tho.. are you running EG in debugging mode???

because of the use of python codecs and 'ascii' when it writes to the log file. and it specifically states ascii in the traceback. that is the only place that i know of in EG that any kind of codec is use with ascii that is in the chain for TriggerEvent.

and following a stack trace that comes out of asyncore leads you to a dead end because of how it completely butchers it

and UDP is a broadcast message and is not able to pass through a router. so it has to be coming from your internal network. now it can be many different things. but the fact that it is always the same character is suspect of something doing it and not some garbage getting in.

i personally do not like asyncore as there are known issues with it. and it's not a true thread it's some kind of voodoo code. now what i find very odd is the fact it states unhandled exception. when in the version i made it is handled. i am not sure if you also tried the one i made...

now if you want to get the true traceback and just have it bypass the error totally

then you will have to add this is the subclass of asyncore

Code: Select all

def handle_error(self):
    import traceback
    traceback.print_exc()

this will override the handle_error method for asyncore and stop it from raising an exception where it has modified the original exception

this will print the exception but not raise it. and should allow the broadcaster plugin to continue along it's way.

but the simple fact is that the event never gets fired if this takes place..

but the printed traceback could be more revealing

Posted: **Mon Nov 21, 2016 8:41 am**

davidmark wrote:Aha! I see that my console is Python 2 and IIRC, EG uses Python 3.

EventGhost 0.4 uses Python 2.6, and 0.5 beta uses 2.7.

jonib

Posted: **Mon Nov 21, 2016 2:54 pm**

Fixed! After extensive testing with fresh eyes (really needed to sleep after the marathon yesterday), I've figured out exactly what causes that exception. In short, you have to pass something wrong to TriggerEvent to make it throw an exception. BTW, sucks that those Python messages aren't red in the log.

https://github.com/EventGhost/EventGhost/pull/122

There's no exception handling as shouldn't need any. Near as I can tell, TriggerEvent doesn't really need any either as you have to really try to break it. If you pass the wrong thing, you get what you deserve. As expected, the concatenation was the culprit and some well-placed print statements would have revealed that we passed something unexpected.

I don't have time to explain it all at the moment, but the final code is very simple. I'll post some console examples later after I've had some time away from this stuff. In short, I can see why Python programmers hate everything to do with Unicode. Though, to be fair, there's always a documented reason for everything that happens.

Still not sure which version of Python EG uses, but have heard 3 is much more strict than 2. I may be able to tell after another session with the Mac Python console, which is definitely 2.

Would have been a lot easier if I had bothered to test every bit of the problem with EG from the start, but the initial changes seemed simple enough to vet with the console. Pretty stupid of me considering that I'm barely a Python programmer at this point. Getting better though.

Note: The latest version uses eg.systemEncoding (cp1252 on my Windows 8 box). Didn't cause me any issues sending round trip from EG to EG. Now, if using some other program to send UDP, will have to make sure they both agree on the encoding by either changing eg.systemEncoding or adjusting the other program. I would expect most use UTF-8. However, this will not matter unless using non-ASCII characters and I don't expect any exceptions in any event. Worst case scenario should be munged data on an encoding mismatch.

Posted: **Mon Nov 21, 2016 2:55 pm**

jonib wrote:
davidmark wrote:Aha! I see that my console is Python 2 and IIRC, EG uses Python 3.
EventGhost 0.4 uses Python 2.6, and 0.5 beta uses 2.7.

jonib

Okay, that settles that then.

Posted: **Mon Nov 21, 2016 2:58 pm**

kgschlosser wrote:and the encoding does not take place in the Trigger event...

it takes place in the log. where it is displayed

i do have a question tho.. are you running EG in debugging mode???

because of the use of python codecs and 'ascii' when it writes to the log file. and it specifically states ascii in the traceback. that is the only place that i know of in EG that any kind of codec is use with ascii that is in the chain for TriggerEvent.

and following a stack trace that comes out of asyncore leads you to a dead end because of how it completely butchers it

and UDP is a broadcast message and is not able to pass through a router. so it has to be coming from your internal network. now it can be many different things. but the fact that it is always the same character is suspect of something doing it and not some garbage getting in.

i personally do not like asyncore as there are known issues with it. and it's not a true thread it's some kind of voodoo code. now what i find very odd is the fact it states unhandled exception. when in the version i made it is handled. i am not sure if you also tried the one i made...

now if you want to get the true traceback and just have it bypass the error totally

then you will have to add this is the subclass of asyncore
Code: Select all
def handle_error(self):
    import traceback
    traceback.print_exc()
this will override the handle_error method for asyncore and stop it from raising an exception where it has modified the original exception

this will print the exception but not raise it. and should allow the broadcaster plugin to continue along it's way.

but the simple fact is that the event never gets fired if this takes place..

but the printed traceback could be more revealing

Thanks for the info; will file for future use, but this issue should be settled with my latest pull request. Tested every which way with round trip send and receive of all sorts of non-Ascii Unicode characters, as well as plain old Ascii.

Posted: **Mon Nov 21, 2016 3:22 pm**

Here some console results that will help illuminate the problem:

Code: Select all

>>> '\xf1&&\xf1'.split(u'&&')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 0: ordinal not in range(128)
>>> u'&&'.join('\xf1\xf1');
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 0: ordinal not in range(128)

...My previous pull request threw one of these on sending non-ASCII characters. Once I saw that, I knew I had it. Note that the delimiter is always a Unicode string.

Also note these:

Code: Select all

>>> '\xf1&&\xf1'.split('&&')
['\xf1', '\xf1']

>>> '\xf1' + u'&&' + '\xf1'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 0: ordinal not in range(128)

In short, if a non-ASCII character is in a "str" object (as opposed to a Unicode string) and is used with split, join, concatenation or whatever, that error comes up every time (as would be expected after reading through hundreds of posts and finally resorting to RTFM). The manual was a tough read for a non-Python programmer and really didn't get what was happening until doing some experiments, logging the received and sent values (in arrays so I could see the quotes and "u" prefixes when present). Live and learn.

Posted: **Mon Nov 21, 2016 8:50 pm**

Here is the full fix:

https://github.com/david-mark/EventGhos ... _init__.py

...It hasn't been merged into the official trunk as of yet and occurred to me that it's not the easiest journey from the pull request I linked earlier and this file.

And just in case we do run into trouble when asyncore barfs up the mystery datagram, here's a debug version of handle_read:

Code: Select all

def handle_read(self):
        data, addr = self.recvfrom(1024)

        # Check if the sending address is any of our interfaces
        my_addr = addr[0] in self.addresses

        if (not my_addr) or self.selfBroadcast:

            print 'Datagram received';
            print 'Raw data:';
            print [data];

            data = data.decode(eg.systemEncoding)

            print 'Decoded data:';
            print [data];

            bits = data.split(self.payDelim);

            print 'Split data:';
            print bits;

            commandSize=len(bits)

            print 'Command size:';
            print commandSize;

            if commandSize==1:
                self.plugin.TriggerEvent(bits[0])
            if commandSize==2:
                self.plugin.TriggerEvent(bits[0],bits[1])
            if commandSize>2:
                self.plugin.TriggerEvent(bits[0],bits[1:])

...Note that I wrap every output in an array; for whatever reason, it helps to see whether the strings are Unicode objects or plain old "str" objects.

In case anyone wants to give up and have the mysterious (corrupted?) datagrams fail with just a log entry (and without taking down asyncore), this should do the trick:

Code: Select all

            try:
                if commandSize==1:
                    self.plugin.TriggerEvent(bits[0])
                if commandSize==2:
                    self.plugin.TriggerEvent(bits[0],bits[1])
                if commandSize>2:
                    self.plugin.TriggerEvent(bits[0],bits[1:])
            except:
                eg.PrintError('Failed to trigger event!');

KG, perhaps you can expand on the except clause to print the stack trace and anything else that would help? I know you had that in one of your earlier posts, but too tired of this stuff to go back and dig it up. I recall you had recently posted about stack traces that come out of asyncore and how to get the most out of them. Also, I didn't see any inaccurate stack traces when I had exceptions in my original fixes; they always led me straight to the issue. That being said, though I got the same error messages, I never duplicated the exact stack trace in my testing earlier today. Though have seen it many times over the years, always indicating line 84 of EventGhostEvent.py; which, unless I miscounted yesterday, is off by a few lines from the only culprit in the vicinity (the concatenation I mentioned yesterday).

Could also do something like this on exception:

Code: Select all

               eg.TriggerEvent('asyncore.sucks');

...And have it speak or play a sound or whatever as a reminder to check the log for any clues. Mine only had the mystery datagrams every few weeks IIRC; And asyncore logged them in such a way that they were as good as invisible (unlike the red PrintError results), so I didn't event notice this was going on for months. Just noticed that my voice commands and X10 reception from AHP stopped working as those were coming from VC on the same box as EG (in which case I assume UDP is as close to guaranteed delivery as it could be). The whole experience was so mysterious and maddening (as everything else worked virtually 100%), I almost gave up on EventGhost entirely.

Hopefully we won't need any of that exception handling and I will finally be able to remove my workaround that disables and enables the plug-in after a period of inactivity. Thanks to everyone that helped and looking forward to status reports on the latest (last?) fixes, regardless of the outcome.

Posted: **Tue Nov 22, 2016 3:43 am**

davidmark wrote:Here is the full fix:

https://github.com/david-mark/EventGhos ... _init__.py.

Does this mean you don't need me to run the tests?

Posted: **Tue Nov 22, 2016 5:20 am**

skribb wrote:
davidmark wrote:Here is the full fix:

https://github.com/david-mark/EventGhos ... _init__.py.
Does this mean you don't need me to run the tests?

Not at this time. What I'd appreciate is if you could try the new version of the plug-in and add the suggested print statements to help debug if there are any more problems. Can optionally add the try-catch, but please wait until we see whether we will have to deal with any exceptions from the mystery datagrams that seem to pop out of asyncore. I used to get them about once every few weeks; for whatever reason, you seem to get them more often. Let's see what happens with those now...

Thanks!

EventGhost

Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix

Re: Broadcaster Plugin Received Data Fix