Stops receiving UDP once per week - restart fixes after hang

Found a bug in EventGhost? Report it here.

Re: Stops receiving UDP once per week - restart fixes after

Postby Pako » Wed Jun 08, 2016 4:49 am

davidmark wrote:How to do that from a macro?
This might work, if you use the action EventGhost-Disable an item (item = Broadcaster plugin), and then the action EventGhost-Enable an item.

Pako
You know flattr ? You can Image
User avatar
Pako
Plugin Developer
 
Posts: 2231
Joined: Sat Nov 11, 2006 1:31 pm
Location: Czech Republic

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Wed Jun 08, 2016 5:04 am

Thanks!

Will try that next time the Broadcaster goes AWOL. Will also try the -debug command, but imagine it will either be a needle in a haystack or the plugin simply stops logging at some point (as it does in the application). Whatever the issue is, the plugin should report *something* to the UI when such catastrophic failures occur.
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Wed Jun 08, 2016 6:31 am

it depends on how the plugin is coded. there might be en exception catching statement put in that part of the plugin but they didn't have it generate a message, that sometimes happens if you put layered exception catching in sometimes you miss one to have print something up.

i will take a look at the code to see if that kind of thing is done.

and the EG -debug will show you things not reported in the normal log and it may become a large file. and there is some exception catching going on. and that could be why there is no log of it in eventghost

you can restart the core by sending a python command

Code: Select all
eg.RestartAsyncore()


i modified the broadcaster plugin to try to restart if it just closes. i added a clause in there so if something goes awry it won't get caught in a loop. so for the moment it's only going to restart once.

it's not throwing any errors for you so i am suspecting for some reason you computer is losing it's network card/connection maybe during a sleep. i am not sure. i know there are a lot of power saving settings for NIC cards and WiFi cards. so you may want to check those settings as well. it's in the device manager under properties for the card.

but this will tell us if that's what is occurring. and if it successfully does the restart i can add something that will allow for unlimited restarts but with clause to stop looping. or to slow it down anyways. that way you don't get an eg log full of notifications. but before that gets done lets narrow down the problem, and make sure that it's closing the connection.

and if your computer is going to sleep and that is in fact what may be causing the problem i can add something that will close and restart the server side of the broadcaster plugin on resume. it could be it's getting stuck somehow and might need a good kick to get it going again

there are a lot of things that could cause this kind of problem. we just want to narrow down what way to attack it and provide the proper solution

attached is the original plugin with some added code to let us know what is happening as well as that restart of the socket closes.
i have made sure the plugin compiles and installs but didn't test the restart bit of it (have to many things going on my computer to disable the network card and restart it. and that is all you need to do to see if it works. then we wait.

this is a direct replacement of the plugin __init__.py file. so no need to do anything else except close EG and put it into place and restart EG. backup your original first that way if there is something wrong you can put the original back in place

K
Attachments
__init__.py
(9.57 KiB) Downloaded 70 times
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Wed Jun 08, 2016 6:37 am

pako's solution would work if tied to the schedulghost plugin so you can have it restart once a day.

K
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Wed Jun 08, 2016 7:43 am

kgschlosser wrote:pako's solution would work if tied to the schedulghost plugin so you can have it restart once a day.

K


I figure I'll set a timer for an hour to restart the Broadcaster, then stop and restart it every time I get a broadcast. Have to figure disabling and re-enabling the plugin will have the same effect as executing it again. Hope so anyway. :)

Thanks guys! Support for this application is very good. Expect my current project will generate numerous plugins for it.
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Wed Jun 08, 2016 7:49 am

kgschlosser wrote:it depends on how the plugin is coded. there might be en exception catching statement put in that part of the plugin but they didn't have it generate a message, that sometimes happens if you put layered exception catching in sometimes you miss one to have print something up.

i will take a look at the code to see if that kind of thing is done.

and the EG -debug will show you things not reported in the normal log and it may become a large file. and there is some exception catching going on. and that could be why there is no log of it in eventghost

you can restart the core by sending a python command

Code: Select all
eg.RestartAsyncore()


i modified the broadcaster plugin to try to restart if it just closes. i added a clause in there so if something goes awry it won't get caught in a loop. so for the moment it's only going to restart once.

it's not throwing any errors for you so i am suspecting for some reason you computer is losing it's network card/connection maybe during a sleep. i am not sure. i know there are a lot of power saving settings for NIC cards and WiFi cards. so you may want to check those settings as well. it's in the device manager under properties for the card.

but this will tell us if that's what is occurring. and if it successfully does the restart i can add something that will allow for unlimited restarts but with clause to stop looping. or to slow it down anyways. that way you don't get an eg log full of notifications. but before that gets done lets narrow down the problem, and make sure that it's closing the connection.

and if your computer is going to sleep and that is in fact what may be causing the problem i can add something that will close and restart the server side of the broadcaster plugin on resume. it could be it's getting stuck somehow and might need a good kick to get it going again

there are a lot of things that could cause this kind of problem. we just want to narrow down what way to attack it and provide the proper solution

attached is the original plugin with some added code to let us know what is happening as well as that restart of the socket closes.
i have made sure the plugin compiles and installs but didn't test the restart bit of it (have to many things going on my computer to disable the network card and restart it. and that is all you need to do to see if it works. then we wait.

this is a direct replacement of the plugin __init__.py file. so no need to do anything else except close EG and put it into place and restart EG. backup your original first that way if there is something wrong you can put the original back in place

K


Thanks for the tips. Not entirely sure that there are no exceptions logged when things go awry, just that it doesn't do anything on broadcast reception after the fact. Haven't noticed any red text on scrolling through the log anyway.

Quite sure the PC in question never sleeps and it sends emails constantly, so I'd notice quickly if it got cut off from the Internet.

Will definitely try your new plugin, but am going to wait for the next failure (and add my timer to disable/enable the plugin) before swapping out the script. Am concerned that restarting the Asynccore thing may have side effects with other modules that use it. (?)

Understand that there may just be a spot where an exception is caught, but no error printed to the log. Will look at that as well. IIRC, the Broadcaster plugin code is pretty brief.

Thanks again!
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Wed Jun 08, 2016 4:22 pm

don't use that plugin i modified. i did it in a wrong way. i mis understood what the handle_close is. that is a close of an actual connection and not a close of the actual port being bound.

my goof.
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Thu Jun 09, 2016 5:50 am

kgschlosser wrote:don't use that plugin i modified. i did it in a wrong way. i mis understood what the handle_close is. that is a close of an actual connection and not a close of the actual port being bound.

my goof.


No problem. I suspect my timer-based restart of the Broadcaster (or Asynccore) will do the trick. Will have it alert me when there is an outage and then I can check the debug log to see how it went off the rails.

Thanks again!
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Thu Jun 09, 2016 10:19 pm

Didn't have to wait long that time. Good news is the disable/enable plugin trick worked like a charm. Set up a 1 hour timer (X10 door sensors send status about every 50 minutes) to restart the timer every time I get a signal via UDP. That should make it work indefinitely without any manual intervention.

Also added a notification when it restarts the timer, so I can check the log for suspicious entries. Will let you know if I find anything to indicate the cause of the outages.

Thanks again!
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Fri Jun 10, 2016 12:40 am

you can also if needed use the

Code: Select all
eg.app.Restart()


and that will restart Eventghost

but here is a better option

Code: Select all
eg.document.Save()
eg.app.Restart()



that will force the save first then restart so the "Do you want to save?" dialog (modal) doesn't come up stopping the restart

i am going to see if i can make a watchdog timer for EG so it will autorestart if it freezes. i am thinking something as simple as 2 threads that run as timers and one just resets the second one. and if the second one doesn't get reset in time if will do the restart.

with providing a checkbox to either default to saving or to not save.

i am curious to know if that would work

but it would have to be like a 10 second thing at most i am thinking. so every 9 seconds it will restart the reboot timer.

the plugin would have to be placed at the top of the plugins list to function properly.

maybe a way to disable the plugin that is causing the problem as well. if it gets caught in a loop. and log the last traceback erros too.


at any rate that could also solve some problems if they are intermittent like the broadcaster one.
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Fri Jun 10, 2016 4:24 am

kgschlosser wrote:you can also if needed use the

Code: Select all
eg.app.Restart()


and that will restart Eventghost


but here is a better option

Code: Select all
eg.document.Save()
eg.app.Restart()



Cool. Is the complete object model documented somewhere?


that will force the save first then restart so the "Do you want to save?" dialog (modal) doesn't come up stopping the restart

i am going to see if i can make a watchdog timer for EG so it will autorestart if it freezes. i am thinking something as simple as 2 threads that run as timers and one just resets the second one. and if the second one doesn't get reset in time if will do the restart.


Have never had EG freeze entirely (on Windows 8.1 anyway). Could be that I am just avoiding problematic plugins.

with providing a checkbox to either default to saving or to not save.

i am curious to know if that would work

but it would have to be like a 10 second thing at most i am thinking. so every 9 seconds it will restart the reboot timer.

the plugin would have to be placed at the top of the plugins list to function properly.

maybe a way to disable the plugin that is causing the problem as well. if it gets caught in a loop. and log the last traceback erros too.

at any rate that could also solve some problems if they are intermittent like the broadcaster one.


Sounds interesting; but, in my experience, the Broadcaster plugin has never frozen EG or interfered with anything else. It just goes AWOL. May have a clue as to why on the next failure.

A ten second timer wouldn't work for me in this case, but once I have enough signals coming in through UDP, can probably drop it down to ten minutes. Possibly less, but don't want to restart the plugin for no reason. Most of the signals come in over HTTP for redundancy as well, so may be able to compare the incoming traffic and watch for discrepancies.

Speaking of HTTP, it occurs to me that I have two Web server plugins running. Don't remember the details, but wanted the persistent variables of the enhanced version and recall having trouble getting the values to appear in responses. IIRC, all request go to the vanilla version and the persistent values are mirrored as EG globals, which are easily inserted in responses. Something like that. Will check the details and make another post.

Thanks again!
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Sat Jun 11, 2016 11:57 pm

I was mentioning the reboot thing because I think you asked about it in an earlier post. And then I went on to babbling. Just thinking of something I should code just in case EG hangs. To have it automatically reboot if that happens. Which at the moment the stock EG does not do. Just something to put on a to-do list
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Stops receiving UDP once per week - restart fixes after

Postby davidmark » Mon Jul 11, 2016 11:55 am

Haven't had to restart EG since the last countermeasure. Also got lucky and caught the failure in the act this weekend. Really lucky as EG just printed the exception message like any other info, so was hard to pick out from the rest of the event log. Think it should be using PrintError. (?)

This is the uncaught error that shuts down the Broadcaster plugin. Appears - handle_read - does not implement exception handling when calling ayncore's - read - method. Apparently some mysterious UDP message received with 0xc0 causes decoding to choke, but simply catching the error in the plugin would at least stop it from closing the "channel". Then I should be able to remove the timer monitoring broadcasts.

Code: Select all
error: uncaptured python exception, closing channel <eg.CorePluginModule.Broadcaster.Server connected 192.168.1.30:33333 at 0x4c001c0> (<type 'exceptions.UnicodeDecodeError'>:'ascii' codec can't decode byte 0xc0 in position 57: ordinal not in range(128) [asyncore.pyc|read|76] [asyncore.pyc|handle_read_event|416] [C:\Program Files (x86)\EventGhost\plugins\Broadcaster\__init__.py|handle_read|95] [C:\Program Files (x86)\EventGhost\eg\Classes\PluginBase.py|TriggerEvent|133] [C:\Program Files (x86)\EventGhost\eg\Classes\EventThread.py|TriggerEvent|78] [C:\Program Files (x86)\EventGhost\eg\Classes\EventGhostEvent.py|__init__|84])


...Actually, on further review, the stack trace appears to be reversed (from what I'm used to anyway). Line 95 in the Broadcaster plugin is:
Code: Select all
self.plugin.TriggerEvent(bits[0])


...So the exception is being thrown (and not caught) in the process of EG triggering an event based on the received "bits". Implementing exception handling on calling TriggerEvent from Broadcaster would seem to be the fix. Make sense?
davidmark
Experienced User
 
Posts: 85
Joined: Thu Jan 01, 2015 5:25 pm

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Mon Jul 11, 2016 8:06 pm

remember how i told you about the readable character thing.

only characters that are decimal 32 or higher and 127 or lower are allowed. because those are human readable.

that is what is causing the problem. something is sneaking past.


so what needs to be done is a for loop to check the ordinal of each byte coming in.

pretty simple actually.

and then just remove the one byte. because it could have been trash or some kind of noise. but you still want the rest of the message as it could be in tact.

or you could still use the event if it remains a constant.

if it is always the same message and the character comes up always in the same spot then i would check with whatever is sending it because there is an error on that end.

let be make up that looping check for ya..

i just have to copy it out of another plugin i wrote because i had to do the same thing. because the avr i have for some reason would mess up. and i would get the same thing. so i had to come up with a way of filtering that stuff out.

with my AVR it only happens when there is a lot of sending and receiving going on between it and eg. like rapidly moving the volume. something gets tweaked on the receivers end of things. could be the same kind of a deal with whatever it is you are receiving the messages from.

but give me like 15 or so to get the code together. and you will be able to just copy and past it right into the current plugin. i will tell you what lines need to be added and at what point in the script. if you don't have a text editor that shows line numbers would be a good thing to download one.

i personally like sublime text for this.
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: Stops receiving UDP once per week - restart fixes after

Postby kgschlosser » Mon Jul 11, 2016 8:35 pm

ok i added it and it compiles, don't really have the means to test it. so give it a shot and see if it works properly.


the code is below
this is what the code was. starting at line 91
Code: Select all
        if (not my_addr) or self.selfBroadcast:
            bits = data.split(str(self.payDelim))
            commandSize=len(bits)
            if commandSize==1:
                self.plugin.TriggerEvent(bits[0])
            if commandSize==2:
                self.plugin.TriggerEvent(bits[0],bits[1])
            if commandSize>2:
                self.plugin.TriggerEvent(bits[0],bits[1:]) 


and you want to change it to this. what it will do is it will just remove the bad characters. and trigger an event for what is remaining.



Code: Select all
        if (not my_addr) or self.selfBroadcast:
            bits = data.split(str(self.payDelim))
            commandSize=len(bits)
            for item in bits:
                itemlen = len(item)
                for i in range(itemlen):
                    if i > len(item) - 1:
                        break
                    if ord(item[i]) > 127 or ord(item[i]) < 32:
                        item.pop(i)
            if commandSize==1:
                self.plugin.TriggerEvent(bits[0])
            if commandSize==2:
                self.plugin.TriggerEvent(bits[0],bits[1])
            if commandSize>2:
                self.plugin.TriggerEvent(bits[0],bits[1:])



we could do a simple exception catching but as i said in the last post it could just be garbage and the message could be ok.

but on the chance you want to use that method i have included the code below for that purpose.


Code: Select all
        if (not my_addr) or self.selfBroadcast:
            bits = data.split(str(self.payDelim))
            commandSize=len(bits)
            try:
                if commandSize==1:
                    self.plugin.TriggerEvent(bits[0])
                if commandSize==2:
                    self.plugin.TriggerEvent(bits[0],bits[1])
                if commandSize>2:
                    self.plugin.TriggerEvent(bits[0],bits[1:])
            except Exception as err:
                errdata = ''
                errlist = ['suffix: ', 'payload: ']
                for i, item in enumerate(bits):
                    if i < 2:
                        errdata += errlist[i]
                    itemlen = len(item)
                    for j in range(itemlen):
                        if j > len(item) - 1:
                            break
                        if ord(item[j]) > 127 or ord(item[j]) < 32:
                            errdata += '  **' + hex(ord(item.pop(j))) + '**  '
                        else:
                            errdata += item[j]
                    if i < 2:
                        errdata += '\n'
                eg.PrintError(str(err))
                eg.PrintError(errdata)
                eg.PrintError('sender address: ' + str(addr[0]))



again the plugin compiles but until you run it real world to let me know if it works or not.

the above code catches the exception and prints an error for it. i also have it printing out the senders IP address to narrow down the source

the code below catches it and moves on so no indication of it happening.

but me personally i would want to know what is causing it.

Code: Select all
        if (not my_addr) or self.selfBroadcast:
            bits = data.split(str(self.payDelim))
            commandSize=len(bits)
            try:
                if commandSize==1:
                    self.plugin.TriggerEvent(bits[0])
                if commandSize==2:
                    self.plugin.TriggerEvent(bits[0],bits[1])
                if commandSize>2:
                    self.plugin.TriggerEvent(bits[0],bits[1:])
            except:
                pass


so there are 3 ways to handle it. i hope one of them is the solution you need.

K
A loved one and Time, The 2 things that can never be replaced.

Family, The only thing you don't get to choose in life.
User avatar
kgschlosser
Site Admin
 
Posts: 1467
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

PreviousNext

Return to Bug Reports

Who is online

Users browsing this forum: No registered users and 3 guests