New version of django-hvad

Sat, 18 May 2013 12:05:00 -0500

During the sprints at Djangocon I finally had the opportunity to work a bit on django-hvad. The package now supports Django 1.5. This release will probably be the last before a 1.0 release that will drop the 'nani' compatibility module. The 0.3 release is mainly a stability and performance update.

Thanks to all the contributors.

Get it today from your favorite cheese shop

SockJS and Tornado for Python real-time web projects

Mon, 08 Apr 2013 12:21:00 -0500

At Djangocon EU last year, I gave a talk about how you could easily get up and running with real-time web applications using Django and Socket.IO. I have now been running an application using this principle for a year and I am everything but a fan of it. I decided i had to switch to something else.

My first impression of SockJS wasn't very good. I thought the documentation was poor and the commit on the main (client) library wasn't very new. However, after reviewing other possibilites I decided that SockJS with sockjs-tornado had to be the best option at the moment.

I found a great gist that showed most of the way. From this gist you get the basic transport and just need to implement some sort of authentication. I also had to set it up for running on Heroku, which unfortunately means waiving one of the essential features of SockJS, namely Websockets (btw, heroku - please add support for this!). In this case I was not too concerned about authorization, just authentication. If your usecase is different, you might want to implement another/your own authentication.

from tornado import web, ioloop
from sockjs.tornado import SockJSRouter, SockJSConnection
import json
import os
import tornadoredis
import urlparse
from django.core import signing
import logging

class Connection(SockJSConnection):
    clients = set()

    def send_error(self, message, error_type=None):
        """
        Standard format for all errors
        """
        return self.send(json.dumps({
            'data_type': 'error' if not error_type else '%s_error' % error_type,
            'data': {
                'message': message
            }
        }))

    def send_message(self, message, data_type):
        """
        Standard format for all messages
        """
        return self.send(json.dumps({
            'data_type': data_type,
            'data': message,
        }))

    def on_open(self, request):
        """
        Request the client to authenticate and add them to client pool.
        """
        self.authenticated = False
        self.channel = None
        self.send_message({}, 'request_auth')
        self.clients.add(self)

    def on_message(self, msg):
        """
        Handle authentication and notify the client if anything is not ok,
        but don't give too many details
        """
        try:
            message = json.loads(msg)
        except ValueError:
            self.send_error("Invalid JSON")
            return
        if message['data_type'] == 'auth' and not self.authenticated:
            try:
                channel = signing.loads(
                    message['data']['token'],
                    key=SECRET_KEY,
                    salt=message['data']['salt'],
                    max_age=40  # Long time out for heroku idling processes.
                    # For other cases, reduce to 10
                )
            except (signing.BadSignature, KeyError) as e:
                self.send_error("Token invalid", 'auth')
                return
            self.authenticated = True
            self.channel = channel
            self.send_message({'message': 'success'}, 'auth')
            logging.debug("Client authenticated for %s" % channel)
        else:
            self.send_error("Invalid data type %s" % message['data_type'])
            logging.debug("Invalid data type %s" % message['data_type'])

    def on_close(self):
        """
        Remove client from pool. Unlike Socket.IO connections are not
        re-used on e.g. browser refresh.
        """
        self.clients.remove(self)
        return super(Connection, self).on_close()

By now, you have probably realized that we are storing clients on the class. This means we can't properly load balance this to multiple nodes without session stickyness. This sucks. I'm trying to figure out a way to store clients inside redis, but for now, session stickyness is required (or if you just run it on one server, you should be fine).

If you watched my talk, you already know that I love redis. My original Socket.IO solution was basically just a frontend to redis. Redis has served me very well for a year, so now I'll show you how to automatically forward messages from redis to the clients. First, add this method to the above class:

    @classmethod
    def pubsub_message(cls, msg):
        for client in cls.clients:
            if client.authenticated and client.channel == msg.channel:
                client.send(msg.body)

Now we just need to run the Tornado server and subscribe to redis:

if __name__ == '__main__':
    url = urlparse.urlparse(os.environ.get('REDISTOGO_URL', os.environ.get('OPENREDIS_URL', 'redis://localhost:6379')))
    pool = tornadoredis.ConnectionPool(host=url.hostname, port=url.port)
    c = tornadoredis.Client(connection_pool=pool, password=url.password)
    c.connect()
    c.psubscribe("*", lambda msg: c.listen(Connection.pubsub_message))

    Router = SockJSRouter(Connection, '/namespace-here', dict(disabled_transports=['websocket']))  # Disable websockets for heroku

    app = web.Application(Router.urls)
    app.listen(os.environ.get("PORT", 8080))
    ioloop.IOLoop.instance().start()

You definitely want to remove the part where I disable websockets, if you are lucky enough to be hosted somewhere where they support websockets. (Please mail me, if you know a host that offers websockets in eu-west). Furthermore you should make a prefix, so you are still able to use pubsub in redis, without it being sent to SockJS. 'namespace-here' is the url your Connection class will be served at. This will enable you to namespace your different services later on.

Now, put it all in a file and run it!

Create a simple TemplateView or equivalent in your web application and insert the following JavaScript in script tags:

function connect(url, auth_json) {
    var sock = new SockJS(url);
    sock.onopen = function() {
        sock.send(auth_json);
    };
    sock.onmessage =  function (event) {
        data = jQuery.parseJSON(event.data);
        if (data.data_type == 'data') {
            // parse your data here
        } else if (data.data_type == 'auth_error') {
            throw data.data.message;
        }
    };
}
connect("http://localhost:8080/namespace-here", "");

In our web application (in this case, django) we generate `` like this:

from django.core import signing
import json

auth_json = json.dumps({
    'data_type': 'auth',
    'data': {
        'salt': SOME_SALT,
        'token': signing.dumps('channel',SECRET_KEY, salt=SOME_SALT)
    }
})

And make sure to include SockJS somewhere before the above:

<script src="http://cdn.sockjs.org/sockjs-0.3.min.js"> script>

Now you should be ready to start sending events to the browser, from python through redis. Let's try.

>>> import redis
>>> r = redis.Redis()
>>> r.publish("channel", "hello")
0L

Depending whether "channel" matches your pattern and whether you have an open connection, this will return either 0L or 1L, afaik the number of "listeners" to your message. You will get front-end error though, as you are not passing in properly formatted JSON. My script assumes you send messages formatted like this:

{
    "data_type": "data",
    "data": {
        "message": "here",
        "or any other": "key/value pairs"
    }
}

That was all for now. If you enjoyed this blog post, check out my twitter for updates.

New version of django-health-check

Thu, 28 Mar 2013 17:15:00 -0500

You might never have heard of django-health-check before, but it is one of my open source projects. You can write small plugins (5-10 lines) that does basic checks for e.g. file system, database and cache.

Whats new:

I completely refactored the API for returning the status. Previously you would return a HealthCheckStatusType - now you either return True or raise one of the provided exceptions: ServiceUnavailable or ServiceReturnedUnexpectedResult.

With the newest release I made sure the code works with python2.7+ (yes, that includes python3).

Go ahead and grab it:

Github

PyPi

Kristian Øllegaard

New version of django-hvad

SockJS and Tornado for Python real-time web projects

New version of django-health-check