SockJS and Tornado for Python real-time web projects
At Djangocon EU last year, I gave a talk about how you could easily get up and running with real-time web applications using Django and Socket.IO. I have now been running an application using this principle for a year and I am everything but a fan of it. I decided i had to switch to something else.
My first impression of SockJS wasn't very good. I thought the documentation was poor and the commit on the main (client) library wasn't very new. However, after reviewing other possibilites I decided that SockJS with sockjs-tornado had to be the best option at the moment.
I found a great gist that showed most of the way. From this gist you get the basic transport and just need to implement some sort of authentication. I also had to set it up for running on Heroku, which unfortunately means waiving one of the essential features of SockJS, namely Websockets (btw, heroku - please add support for this!). In this case I was not too concerned about authorization, just authentication. If your usecase is different, you might want to implement another/your own authentication.
from tornado import web, ioloop
from sockjs.tornado import SockJSRouter, SockJSConnection
import json
import os
import tornadoredis
import urlparse
from django.core import signing
import logging
class Connection(SockJSConnection):
clients = set()
def send_error(self, message, error_type=None):
"""
Standard format for all errors
"""
return self.send(json.dumps({
'data_type': 'error' if not error_type else '%s_error' % error_type,
'data': {
'message': message
}
}))
def send_message(self, message, data_type):
"""
Standard format for all messages
"""
return self.send(json.dumps({
'data_type': data_type,
'data': message,
}))
def on_open(self, request):
"""
Request the client to authenticate and add them to client pool.
"""
self.authenticated = False
self.channel = None
self.send_message({}, 'request_auth')
self.clients.add(self)
def on_message(self, msg):
"""
Handle authentication and notify the client if anything is not ok,
but don't give too many details
"""
try:
message = json.loads(msg)
except ValueError:
self.send_error("Invalid JSON")
return
if message['data_type'] == 'auth' and not self.authenticated:
try:
channel = signing.loads(
message['data']['token'],
key=SECRET_KEY,
salt=message['data']['salt'],
max_age=40 # Long time out for heroku idling processes.
# For other cases, reduce to 10
)
except (signing.BadSignature, KeyError) as e:
self.send_error("Token invalid", 'auth')
return
self.authenticated = True
self.channel = channel
self.send_message({'message': 'success'}, 'auth')
logging.debug("Client authenticated for %s" % channel)
else:
self.send_error("Invalid data type %s" % message['data_type'])
logging.debug("Invalid data type %s" % message['data_type'])
def on_close(self):
"""
Remove client from pool. Unlike Socket.IO connections are not
re-used on e.g. browser refresh.
"""
self.clients.remove(self)
return super(Connection, self).on_close()
By now, you have probably realized that we are storing clients on the class. This means we can't properly load balance this to multiple nodes without session stickyness. This sucks. I'm trying to figure out a way to store clients inside redis, but for now, session stickyness is required (or if you just run it on one server, you should be fine).
If you watched my talk, you already know that I love redis. My original Socket.IO solution was basically just a frontend to redis. Redis has served me very well for a year, so now I'll show you how to automatically forward messages from redis to the clients. First, add this method to the above class:
@classmethod
def pubsub_message(cls, msg):
for client in cls.clients:
if client.authenticated and client.channel == msg.channel:
client.send(msg.body)
Now we just need to run the Tornado server and subscribe to redis:
if __name__ == '__main__':
url = urlparse.urlparse(os.environ.get('REDISTOGO_URL', os.environ.get('OPENREDIS_URL', 'redis://localhost:6379')))
pool = tornadoredis.ConnectionPool(host=url.hostname, port=url.port)
c = tornadoredis.Client(connection_pool=pool, password=url.password)
c.connect()
c.psubscribe("*", lambda msg: c.listen(Connection.pubsub_message))
Router = SockJSRouter(Connection, '/namespace-here', dict(disabled_transports=['websocket'])) # Disable websockets for heroku
app = web.Application(Router.urls)
app.listen(os.environ.get("PORT", 8080))
ioloop.IOLoop.instance().start()
You definitely want to remove the part where I disable websockets, if you are lucky enough to be hosted somewhere where they support websockets. (Please mail me, if you know a host that offers websockets in eu-west). Furthermore you should make a prefix, so you are still able to use pubsub in redis, without it being sent to SockJS. 'namespace-here' is the url your Connection class will be served at. This will enable you to namespace your different services later on.
Now, put it all in a file and run it!
Create a simple TemplateView or equivalent in your web application and insert the following JavaScript in script tags:
function connect(url, auth_json) {
var sock = new SockJS(url);
sock.onopen = function() {
sock.send(auth_json);
};
sock.onmessage = function (event) {
data = jQuery.parseJSON(event.data);
if (data.data_type == 'data') {
// parse your data here
} else if (data.data_type == 'auth_error') {
throw data.data.message;
}
};
}
connect("http://localhost:8080/namespace-here", "");
In our web application (in this case, django) we generate `` like this:
from django.core import signing
import json
auth_json = json.dumps({
'data_type': 'auth',
'data': {
'salt': SOME_SALT,
'token': signing.dumps('channel',SECRET_KEY, salt=SOME_SALT)
}
})
And make sure to include SockJS somewhere before the above:
<script src="http://cdn.sockjs.org/sockjs-0.3.min.js"> </script>
Now you should be ready to start sending events to the browser, from python through redis. Let's try.
>>> import redis
>>> r = redis.Redis()
>>> r.publish("channel", "hello")
0L
Depending whether "channel" matches your pattern and whether you have an open connection, this will return either 0L or 1L, afaik the number of "listeners" to your message. You will get front-end error though, as you are not passing in properly formatted JSON. My script assumes you send messages formatted like this:
{
"data_type": "data",
"data": {
"message": "here",
"or any other": "key/value pairs"
}
}
That was all for now. If you enjoyed this blog post, check out my twitter for updates.
testing