Page 1 of 2

How To keep and recover websocket connections

Posted: Thu Feb 19, 2015 9:59 am
by krambriw
Hello,
This is a long post, but hey, this is the Coding Corner!

I thought this story might be some useful for those deploying WebSocket connections in web pages. I do myself, having some cheap mobile tablets allowing me to control functions in EventGhost. WebSockets allows, as you know, to push information to the tablet, eliminating the page reload to get status changes.

There are certain things that I have discovered and actually spent a number of days to overcome. When you are establishing a WebSocket connection, you would most likely
- keep an eye on it so that it is ready and alive
- send keep alive messages from the client (I read somewhere that this is recommended)
- reconnect if for some reason it is closed
- monitor for incoming messages and, after a time out depending on how frequently you expect messages, close the WebSocket connection and try to open it again

After digging around on internet, I found the key. When a websocket connection is opened you can follow it's status using the readyState. These constants are used by the readyState attribute to describe the state of the WebSocket connection.
Constant Value Description
CONNECTING 0 The connection is not yet open.
OPEN 1 The connection is open and ready to communicate.
CLOSING 2 The connection is in the process of closing.
CLOSED 3 The connection is closed or couldn't be opened.
So with this in mind, I started to experiment a bit.

Opening the WebSocket connection first time is normally not causing any problem, readyState is 1 and you can both send and receive messages. But once it is established and gets interrupted, readyState changes to 3. In your code you have most likely implemented a timer that triggers a reconnect attempt. Here you have to be careful, especially if you are connecting over the network. Each attempt needs to time out before you try again and this is the major mistake I did in the beginning, I simply retried to quickly...it worked in the local machine but not over the network. What happened was the following:

1) EG Webserver was running
2) My tablets had connected and had established WebSocket connections, everything working fine
3) I close down EG
4) Tablets losing connections, starts retrying connections (every 2 seconds)
5) I start EG again
6) Now EG receives a large number of connection requests from each tablet (all depending on how long EG was stopped) since they where queued up and things are then messed up, sad for EG had to work hard

Something was needed here. The thing was, I wanted tablets to reconnect as fast as possible. Using the readyState I was able to handle the connections properly and then the above scenario worked ok without a lot of unnecessary attempts.

Just to give some examples of well working client test web pages, I provide them below. The first one is doing the job, it is connecting and recovering a WebSocket connection. In addition, I have added a ping function (keepAlive) that sends a message to the WebSocket server every 30 second. This version would normally be enough but I found another problem (when using Chrome) where the connection timed out but was kept open. I suspect this can be browser dependent (other people have reported similar problems with Chrome). Therefore I have added also a second example further down below.

First example:

Code: Select all

<html lang="sv">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> 
<META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">
<META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE">

<title>EventGhost connection and recovery test</title>

<script language="javascript">
initial = window.setInterval(refresh, 2000);
keepAlive = window.setInterval(kAlive, 30000);
wSocket = null;
inProgress = false;
var prot = "";
var ip = location.host;

if (location.protocol == "https:") {
	prot = "wss://";
}
else {
	prot = "ws://";
}

//var url = prot + ip + "/ws"; //mandatory for Tornado
var url = prot + ip; //enough for Webserver


function init() {
	// event handlers for WebSocket:
	inProgress = true;
	wSocket = new WebSocket(url);
	writeToScreen('init.begin:readyState :'+wSocket.readyState);
	writeToScreen('init.begin:inProgress :'+inProgress);
	wSocket.onopen = function() {
		document.getElementById("output").style.backgroundColor ="white"; //White
		writeToScreen('onOpen:readyState :'+wSocket.readyState);
		doSend('Ping: '+keepAlive);
	}
	wSocket.onmessage = wsMessage;
	wSocket.onclose = function() {
		document.getElementById("output").style.backgroundColor ="#C0C0C0"; //Grey
		writeToScreen('onClose:readyState :'+wSocket.readyState);
	}
	inProgress = false;
	writeToScreen('init.finished:inProgress :'+inProgress);
}
 					
function kAlive() {
	doSend('Ping: '+keepAlive);
}

function refresh() {
	if (wSocket.readyState == 3){
		if (inProgress == false) {
			writeToScreen('refresh:inProgress :'+inProgress);
			init();
		}
	}
}

function writeToScreen(message) { 
var pre = document.createElement("p"); 
pre.style.wordWrap = "break-word"; 
pre.innerHTML = message; 
output.appendChild(pre); 
} 

function doSend(strng) {
	writeToScreen('doSend:readyState :'+wSocket.readyState);
	if (wSocket.readyState == 1) {
		wSocket.send(JSON.stringify(strng));
	}
}

function wsMessage(event) {
	writeToScreen(event.data);
}

window.addEventListener("load", init, false); 

</script> 

<div id="output"></div> 


In this second example, I have added some more advanced features using an external javascript, worker.js. This is monitoring for expected messages to arrive. If this doesn't happen within a defined time period, the script is forcing down the WebSocket connection. Then it tries to reconnect the normal way as in the first example. Also the second example is working very well during my tests. Only thing to think about is to adjust the time period for expected messages so that it fits your environmnet. In my case, i have 90 seconds, and you adjust it in the worker.js (I'll show below).

Second example:

Code: Select all

<html lang="sv">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> 
<META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">
<META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE">

<title>EventGhost connection and improved recovery test</title>

<script language="javascript">
var myVar = 0;
var worker = new Worker('worker.js');
initial = window.setInterval(refresh, 2000);
keepAlive = window.setInterval(kAlive, 30000);
wSocket = null;
inProgress = false;
var prot = "";
var ip = location.host;

if (location.protocol == "https:") {
	prot = "wss://";
}
else {
	prot = "ws://";
}

//var url = prot + ip + "/ws"; //mandatory for Tornado
var url = prot + ip; //enough for Webserver


function doWork(prm) {
	worker.postMessage(prm);
}

function checkEven(val) {
	return (val%2 == 0);
}

worker.onmessage = function (evt) {
	myVar = evt.data[1];
	writeToScreen('counter :'+evt.data[0]);

	if (evt.data[0] == 999) {
		document.getElementById("output").style.color ="red";
		wSocket.close();
	}
	else {
		if (checkEven(evt.data[0])) {
			document.getElementById("output").style.color ="black";
		}
		else {
			document.getElementById("output").style.color ="blue";
		}
	}
}

function init() {
	// event handlers for WebSocket:
	inProgress = true;
	wSocket = new WebSocket(url);
	writeToScreen('init.begin:readyState :'+wSocket.readyState);
	writeToScreen('init.begin:inProgress :'+inProgress);
	wSocket.onopen = function() {
		document.getElementById("output").style.backgroundColor ="white"; //White
		writeToScreen('onOpen:readyState :'+wSocket.readyState);
		doSend('Ping: '+keepAlive);
	}
	wSocket.onmessage = wsMessage;
	wSocket.onclose = function() {
		document.getElementById("output").style.backgroundColor ="#C0C0C0"; //Grey
		writeToScreen('onClose:readyState :'+wSocket.readyState);
	}
	inProgress = false;
	writeToScreen('init.finished:inProgress :'+inProgress);
}
 					
function kAlive() {
	doSend('Ping: '+keepAlive);
}

function refresh() {
	if (wSocket.readyState == 3){
		if (inProgress == false) {
			writeToScreen('refresh:inProgress :'+inProgress);
			init();
		}
	}
}

function writeToScreen(message) { 
var pre = document.createElement("p"); 
pre.style.wordWrap = "break-word"; 
pre.innerHTML = message; 
output.appendChild(pre); 
} 

function doSend(strng) {
	writeToScreen('doSend:readyState :'+wSocket.readyState);
	if (wSocket.readyState == 1) {
		wSocket.send(JSON.stringify(strng));
	}
}

function wsMessage(event) {
	writeToScreen(event.data);
    doWork(myVar);
}

window.addEventListener("load", init, false); 

</script> 

<div id="output"></div> 

This is the code for the worker.js script. What you eventually need to do is to modify the lines 20 and 41
if (i > 30) {
myVar=setInterval(function(){myTimer()},3000);
The timer with interval (3 seconds) is incrementing i every 3 seconds means that after 90 seconds, we will get a re-connection attempt. You may experiment with what is best for you.

The worker.js

Code: Select all


var myVar = 0;
var i = 1;
var j = 0;
var myArray = new Array();
var myTimers = new Array();

function checkEven(val)
{
    return (val%2 == 0);
}

    
    function myTimer() {
    myArray[0] = i;
    myArray[1] = myVar;
    myArray[2] = myTimers;
    postMessage(myArray);
    i++;
    if (i > 30) {
        myArray[0] = 999;
        myArray[1] = myVar;
        myArray[2] = myTimers;
        postMessage(myArray);
		i = 1;
    }
}


onmessage = function (event) {
    for (j=0; j<myTimers.length; j++) {
        clearInterval(myTimers[j]);
    }
    if (checkEven(i)) {
        i = 2;
    }
    else {
        i = 1;
    }
    myVar = null;
    myVar=setInterval(function(){myTimer()},3000);
    myTimers.splice(0, myTimers.length, myVar);
}

Attached are the files, download and unzip them all in the webroot of EG's webserver and try out.

All my own web pages are now updated with the same logic and, cross your fingers, still working as expected. I hope the samples are helpful when you build or modify your own web pages!
websocket_tests.zip
(2.51 KiB) Downloaded 142 times
Best regards,
Walter

Re: How To keep and recover websocket connections

Posted: Sat Feb 21, 2015 5:39 am
by Pako
Dear Walter!
You've done a really remarkable and useful work!
I appreciate it greatly and I think, that this method of presentation is very valuable.
I have some additions to it.
When I was working on adding support for WebSocket in Webserber plugin, I dealt with the ping-pong functions too.
Finally I decided to integrate WebSocket without it. I had two reasons for this:
1) I saw it as an undue complication
2) I knew another way (we can call it a passive way of example), which is easier. Now I am describing this passive way a bit more detail:

I met this way when I was doing Pushbullet plugin. The principle is very simple.
Server broadcasts every 30 seconds to every client this message: {"type": "nop"}.
The client just passively monitors this (no response sends), and if the message does not arrive within a certain time,
it tries to restart the WebSocket connection.

Best regards,
Luboš

Re: How To keep and recover websocket connections

Posted: Sat Feb 21, 2015 7:04 am
by krambriw
Dear Lubos,
It is my pleasure and thank you for your kind words! My system is now running so fine it is amazing how well it is functioning after I implemented the changes.
The client just passively monitors this (no response sends), and if the message does not arrive within a certain time,
it tries to restart the WebSocket connection.
This is actually how the second example works (and also how I have implemented my private web pages for the tablets). It works but in worst case your client has to wait maybe 30 seconds (or until the timer fires) before it gets connected again. This might be ok for normal operation but I targeted a solution to make it as fast as possible. With the ping/pong you should expect a response within milliseconds and can re-connect immediately.

BTW Do you want me to look into the same for the Webserver plugin? I think I have 'warmed up' on the topic so I could maybe find a proposal for the Webserver plugin as well

My very best regards,
Walter

Re: How To keep and recover websocket connections

Posted: Sat Feb 21, 2015 7:19 am
by Pako
Dear Walter !
krambriw wrote:BTW Do you want me to look into the same for the Webserver plugin? I think I have 'warmed up' on the topic so I could maybe find a proposal for the Webserver plugin as well
Of course - if you fancy, let's go!
But I suppose it will be much more difficult than in the case of Tornado plugin.

Best regards,
Luboš

Re: How To keep and recover websocket connections

Posted: Mon Feb 23, 2015 11:52 pm
by piert
Yes, please, Walter!

I am using http commands to talk from my KNX domotica system to eventghost. This works, but after several days there always comes a point where the communication stops and I have to reboot my KNX system for it to work again.

I don't know how to find out what the real problem is here: it might be the websocket issue.
It would be nice to test it out.

Thanks for all your great contributions! (both you and Pako rule!)
Regards,
Perry

Re: How To keep and recover websocket connections

Posted: Thu Feb 26, 2015 2:51 pm
by krambriw
@Piert
The topic I have investigated is how our websocket servers in EG are behaving (Tornado & Webserver). I have no idea and I doubt this will improve your connection with KNX domotica. Maybe the attached html files can give some ideas.
Best regards

Re: How To keep and recover websocket connections

Posted: Thu Feb 26, 2015 3:14 pm
by krambriw
I have now spent some time debugging further...

1) I have modified the Webserver plugin a bit to enable my debugging
- Added support for client initiated high level ping/pong
- Detecting 'WebSocket Protocol Error' and 'Masked frame from server' errors in handle_one_request function
to improve session keep alive and to allow a quick connection recovery
- Additional printouts when a problem is detected (eg.PrintInfo)
__init__.py
(187.79 KiB) Downloaded 157 times
2) Setup: I have both Tornado and Webserver running in EG on a separate PC (win7 64bit) and connection to them (on ports 8282 and 8383 respectively) with the same test Web page from separate tabs with open javascript console in Chrome. The result is interesting since it is not showing the same and when/if problems arises, it is not happening at the same time.

For the Webserver plugin, the following errors are from time to time reported:

Code: Select all

From EG log: decoded: êMasked frame from server
From javascript console: A server must not mask any frames that it sends to the client.

From EG log: decoded: êWebSocket Protocol Error
From javascript console: Invalid frame header

From EG log: closing: êInvalid UTF-8 in text frame
From javascript console: Could not decode a text frame as UTF-8.
For the Tornado plugin, the following can occur:

Code: Select all

From javascript console: Invalid frame header
From javascript console: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 0
Why this happens is hard to say, could very well be collisions in my network, could maybe be Windows interfering with the NIC's, who knows. Anyway, important at least for me, is that connections are restored reliable and as fast as possible. The attached html file includes logic to handle this. It also implements the high level client initiated Ping and monitoring of the Pong response. My feeling is that this ping/pong makes connections with both websocket servers more stable even if not perfect.
crr_test.zip
(1.68 KiB) Downloaded 132 times

Re: How To keep and recover websocket connections

Posted: Thu Feb 26, 2015 6:09 pm
by krambriw
In addition

Related to this error from the Webserver plugin,

Code: Select all

From EG log: closing: êInvalid UTF-8 in text frame
From javascript console: Could not decode a text frame as UTF-8.
I once also captured this traceback message in the EG log. If I remember, char(129) is an extended ascii character, maybe this is why the utf-8 decoding failed??

Code: Select all

16:26:56   closing: êInvalid UTF-8 in text frame
16:26:56   Traceback (most recent call last) (1694):
16:26:56     File "C:\Program Files (x86)\EventGhost\plugins\Webserver\__init__.py", line 3749, in BroadcastMessage
16:26:56       client.write_message(message)
16:26:56     File "C:\Program Files (x86)\EventGhost\plugins\Webserver\__init__.py", line 1041, in write_message
16:26:56       self.request.send(chr(129))
16:26:56     File "socket.pyc", line 165, in _dummy
16:26:56   error: [Errno 9] Bad file descriptor

Re: How To keep and recover websocket connections

Posted: Sun Mar 01, 2015 7:58 am
by Pako
Dear Walter!
I looked at your edit (Webserver plugin), and I think it is not entirely correct solution.
Maybe I did not understand well, but it seems that I can not send (from client to server) any message that contains the string "Ping".
This is a serious flaw, I think.
So I tried to make another solution.
Because it is a "high level ping / pong", Ping is processed only in on_ws_message WebSocket method.
Because now Ping is "packed" into json string/dictionary, as well as answer (Pong) is done the same way.
Note1:
On the client-side the function kAlive needs to be changed as follows:

Code: Select all

function kAlive() {
	//console.log("Ping: "+keepAlive);
   doSend({'method':'Ping','id':keepAlive});
}
Of course, it also affects the eventual treatment of response (Pong).
Note2:
It is clear, that just as it is impossible to send a message that contains the string "WebSocket Protocol Error". But I think that this is not a serious problem.
Note3:
I previously had not noticed, that we're talking about "high level ping / pong". I thought of "low level" so I pointed to the difficulty of implementation.
But I think that the "high level" is good enough for us.

Best regards,
Luboš

Re: How To keep and recover websocket connections

Posted: Sun Mar 01, 2015 7:14 pm
by krambriw
Dear Lubos,
Thank you very much, I must been overlooking that somehow :oops:

Anyway, regarding ping/pong, currently the low level ping/ping (that is supposed to be implemented by browsers for instance) does not have an api available fromm javascript (what I have found by googling...). So the only remaining we can do is exactly that, the high level. Lets see have other users experience this.

The typical problems I have seen (error messages in javascript console) has been reported by other users and seems to be related stronger to Chrome. I have however also seen similar with both UC Browser and Firefox. Currently I have a stomach feeling that Firefox is working better, I get less disconnects than with Chrome.

My kind regards, Walter

Re: How To keep and recover websocket connections

Posted: Sun Mar 01, 2015 7:37 pm
by Pako
Dear Walter,
I would like to release a new version EventGhost and I still have some questions.
1) Are you satisfied with this solution Ping/Pong protocol?
2) After all, I'd like to also solve the problem with the string "WebSocket Protocol Error". In which case, it appears?
I suppose it is not wrapped in a JSON string. If so, then I can try it in the following way:
1) Test, if message is JSON string.
2a) If so, proceed normally
2b) If it is not, only then test, whether the string "WebSocket Protocol Error" is present.
That's slightly better than the current version.

Best regards,
Luboš

Re: How To keep and recover websocket connections

Posted: Mon Mar 02, 2015 6:57 am
by krambriw
Dear Lubos,
1) Are you satisfied with this solution Ping/Pong protocol?
Yes, I think it works very well
2) After all, I'd like to also solve the problem with the string "WebSocket Protocol Error". In which case, it appears?
I noticed this problem as it was logged in the javascript console of the browser as 'Invalid frame header'. At this moment the connection was terminated (not by my plugin). I then just added the code to the Webserver plugin to print the decoded data just to try to find out what was the reason behind. Next time it happened the decoded data was "WebSocket Protocol Error"

1) I suppose it is not wrapped in a JSON string. If so, then I can try it in the following way:
1) Test, if message is JSON string.
2a) If so, proceed normally
2b) If it is not, only then test, whether the string "WebSocket Protocol Error" is present.
That's slightly better than the current version.
I agree, if this is possible it would be much better

In addition, just a question, to make the same html-page compatible also with the Tornado plugin, should this new Ping/Pong be considered there as well? Would it improve? I mean I have tried the same html page and they do work for both but I wonder if the solutions have different quality levels?

Kind regards, Walter

Re: How To keep and recover websocket connections

Posted: Mon Mar 02, 2015 7:36 am
by Pako
Dear Walter !
krambriw wrote:I noticed this problem as it was logged in the javascript console of the browser as 'Invalid frame header'. At this moment the connection was terminated (not by my plugin). I then just added the code to the Webserver plugin to print the decoded data just to try to find out what was the reason behind. Next time it happened the decoded data was "WebSocket Protocol Error"
Thank you for the explanation, I expected something similar.
krambriw wrote:I agree, if this is possible it would be much better
Well, I'll do it this way.
krambriw wrote:In addition, just a question, to make the same html-page compatible also with the Tornado plugin, should this new Ping/Pong be considered there as well? Would it improve? I mean I have tried the same html page and they do work for both but I wonder if the solutions have different quality levels?
Yes, of course. I also thought Tornado plugin, but first I wanted to have confirmed that this solution satisfies. I'll do it ASAP.

Best regards,
Luboš

Re: How To keep and recover websocket connections

Posted: Mon Mar 02, 2015 8:56 am
by Pako
Dear Walter,
here is a modified version of Webserver plugin.
Can you please try it?

Thanks and best regards,
Luboš

Re: How To keep and recover websocket connections

Posted: Mon Mar 02, 2015 11:04 am
by krambriw
Dear Pako, thank you
It is now put into test, I will run it for a while and report back

Best regards, Walter