When receiving UDP with nordic characters?

[kgschlosser]

hey skribb, you mind if i make a new topic called Broadcaster Unicode Error and merge this one with another one that exists... and is the same issue... and i have also rebuilt the event triggering component of the plugin.


I now FINALLY understand Unicode encoding...

http://farmdev.com/talks/unicode/

this has to be the most demeaning kind of tutorial.. but i get it now... LOL


you decode a string to unicode. and you encode unicode to a string that was the magical tidbit that helped me to understand it better

so s.decode
and u.encode

so now this thing should work.

i also added a little backup just in case there is still an issue.. it will spit out the error but not halt the asyncore and if you are running eg 0.4 the -debug can be used and it will also print the error there as well.

attached is the plugin.. in the new .egplugin format... so you should be able to just double click away on it

PLUGIN HAS NOT BEEN TESTED

[davidmark]

thanks David for catching that dumbass messup.. LOL

happens with me all the time tho.

No problem.

yeah this is for nordic characters. so it could be latin1 that's why i stated that..

There's no telling what the bytes received via UDP will represent, that's why it is important to standardize on UTF-8 and document that it is the only acceptable encoding to send to EG. Same for HTTP, though it is possible for that protocol to specify the encoding (UTF-8 is just the default).

i am running into a similar problem with a plugin i am working on but if i encode the string with utf8 it builds the xml tree but won't convert the xml tree into a string. and if i convert it using str() i get an error elsewhere. so not exactly sure how to handle that kind of issue because i have to iterate through the whole thing..... unless maybe if i use isinstance to check and see if it's not a basestring and if not then convert it and if it's unicode then to encode it. i dunno i will have to mess with it

I'd have to see the code.

[davidmark]

I now FINALLY understand Unicode encoding...

http://farmdev.com/talks/unicode/

I haven't read that tutorial, but Unicode is a character set. An example encoding is UTF-8.

you decode a string to unicode. and you encode unicode to a string that was the magical tidbit that helped me to understand it better

To be clear, you decode bytes to create a Unicode string. You encode Unicode strings to get bytes.

so now this thing should work.

I'll look at it...

[davidmark]

Apparently that plugin is compiled. Can you post the source so I can review what you did?

Thanks!

[skribb]

hey skribb, you mind if i make a new topic called Broadcaster Unicode Error and merge this one with another one that exists.

Go ahead , you're the admin

[davidmark]

I posted my proposed solution; will work for any Unicode characters, provided the sender uses UTF-8 encoding.

viewtopic.php?f=4&t=8222&p=43750#p43750

[kgschlosser]

it's not compiled.


it's a zip file.. you have to change the extension to zip

[kgschlosser]

I know i am admin. but still... even admins shouldn't go around just doing things. and asking is always the best way

but because there could also be a diverse number of people using different character sets. what i propose as a solution to this jumble. is to make a convince class spinner for choosing character sets and add that to EG. so that plugin devs can easily create it for their configuration dialog. unfortunately i would love to be able to have it pass back an instance of something that could do the work behind the scenes but you cannot store an instance inside of a class. however..... i could have it store the selection and if the plugin developer used the spinner at all it would automatically do the encoding or decoding if they used self.plugin.TriggerEvent this is a means


now i didn't iterate through the bits because i am not sure how the udp plugin functions. i have to go and look at the code and make sure that they it doesn't run eval or ast on any of the payload data. because ya don't want to encode or decode a non basestring

[davidmark]

it's not compiled.


it's a zip file.. you have to change the extension to zip

Oh, okay. I have posted what I consider to be the ideal solution in the other thread (and in pull requests). Expect it might vary a bit from yours. Will look when I have a chance. I suggest you look at my solution as well.

Thanks!

[davidmark]

but because there could also be a diverse number of people using different character sets. what i propose as a solution to this jumble. is to make a convince class spinner for choosing character sets and add that to EG. so that plugin devs can easily create it for their configuration dialog. unfortunately i would love to be able to have it pass back an instance of something that could do the work behind the scenes but you cannot store an instance inside of a class. however..... i could have it store the selection and if the plugin developer used the spinner at all it would automatically do the encoding or decoding if they used self.plugin.TriggerEvent this is a means

You are over-thinking this thing. Have a look at my pull requests, as well as eg.systemEncoding (which is used by the Network Sender and Receiver plugins). I can't imagine any character sets that would not work with UTF-8, but if we use eg.systemEncoding then will be a moot point as users can change it to whatever they prefer.

now i didn't iterate through the bits because i am not sure how the udp plugin functions. i have to go and look at the code and make sure that they it doesn't run eval or ast on any of the payload data. because ya don't want to encode or decode a non basestring

If you mean the Network Receiver and Network Sender, see above (and not sure what they have to do with this Broadcaster plugin). All we are doing is decoding the bytes to create Unicode strings to pass to our own TriggerEvent. We know what happens when we don't decode.

To be clear: none of these changes will affect non-Ascii characters and those were the only characters that worked at all previously. Anything else threw exceptions.

[davidmark]

I looked at the zipped solution. I expect we should go with mine.

For one, there's no way we should be using a try-except wrapped around our own code (e.g. TriggerEvent). We have control of both ends and already know what happens if we pass non-Ascii characters to it. Also understand that Ascii characters are not changed on UTF-8 decoding.

Also don't need any of the isinstance checks, nor do we ever want to encode bytes received from Asynccore. That's done on send. You are also encoding or decoding an array at one point.

And you left out the encode step on send, so the solution is asymmetrical (i.e. a round trip send and receive will mutate non-Ascii characters).

I can save you some time as I've thought all this through very carefully and ran appropriate tests in the console for any bits I was unsure about. Unless I made a typo, my proposed solution is sound.

[kgschlosser]

you have to understand. mine is a band-aide not a fix. and i personally don't care who goes with what. and I really have not dedicated a lot of time to this. so as long as it's fixed that's fine.. all i was trying to do what to make it work. but if a fix needs to be done the whole plugin really needs to be re written with threading and the use of sockets and not asyncore.

as this would most likley not be a problem if not for the use of asyncore.

and you have spent a lot of time working with this issue. and i was just trying to handle the problem. not rewrite a plugin.

[davidmark]

you have to understand. mine is a band-aide not a fix.

I understand that perfectly. I posted some band-aids myself in the other thread to log lots of info and optionally catch exceptions. All designed to be temporary.

and i personally don't care who goes with what.

I only care to the extent that it helps us move forward towards a permanent solution. You admitted that you were new to Unicode and I'm new to Python but...

and I really have not dedicated a lot of time to this. so as long as it's fixed that's fine..

Exactly. I have as I have a stake in the solution. I don't blame you for not devoting a lot of time to a feature you don't use. I assume that you, like most people, ignore the Broadcaster plug-in entirely. It's a good idea unless you get sucked into using that epic PoS called VoxCommando. I did at the beginning and still in the process of scraping it off my shoe.

all i was trying to do what to make it work. but if a fix needs to be done the whole plugin really needs to be re written with threading and the use of sockets and not asyncore.

Me too, except I'm not going anywhere near rewriting it to use something besides the other epic PoS involved with this mess. I'm going to eventually stop using Broadcaster, but want to make it as competent as possible in the meantime. Seems like I'm pretty close at this point and I appreciate your help along the way.

as this would most likley not be a problem if not for the use of asyncore.

There are two distinct problems here, with some potential for overlap: encoding/decoding of Unicode characters and the mysterious asyncore-related screw-ups that seem to produce gibberish on occasion.

and you have spent a lot of time working with this issue. and i was just trying to handle the problem. not rewrite a plugin.

I've spent more than my share at this point, but the end result amounts to 5-6 lines changed in the plug-in. That solved one problem for sure: we can now send and receive Unicode strings with confidence, even when they contain non-ASCII characters. Rewriting it is definitely not on my agenda either, even if the asyncore issues continue. We'll see what happens with the exceptions now and add a try-except where appropriate to keep the Broadcaster from shutting down. That's going to be the end of it for me.

Thanks!