As Six Apart’s services have grown in popularity and size we have had increasingly more and more people and companies wanting to stay abreast of new content being made available on our network. Traditionally this has been handled by us “pinging,” a process by which we send the content to their server over a very simple protocol. But this model has not proven very scalable for us. We have on occasion made the really tough decision not to ping someone wanting to be notified because of the impacts it might have on performance and because of the resources it would take to build out an infrastructure scalable enough to handle pinging upwards of 40 or so endpoints at a rate of 15-20 posts per second, which is about the volume of posts Six Apart deals with on any given day. On the flip side, we have had partners keel over because our pinging them overwhelmed their systems.

So how do we keep all of our customers and partners up to date in a way that is cost effective, scalable, easy to implement (for everyone) and easy to support?

First, we had to break the model where Six Apart was responsible for sending its content independently to all its subscribers. That model would be tantamount to Federal Express implementing a point-to-point package delivery service with no centralized distribution system - such a system might work, but it can’t scale, and its not cost effective.

So we did what FedEx did, we centralized our content distribution system by creating the “Six Apart Update Stream.” Six Apart Updates allows anyone to connect to steady and endless stream of content generated by our users. The content of the stream is broadcast using the Atom Syndication Format so that subscribers can easily parse out and listen for the content that is of interest to them.

But be warned, we have often referred internally to this “stream” as a “fire hose” because there is still a tremendous amount of data that is broadcast through this service. But despite the torrent you can at least rest assured that the stream is extraordinarily unlikely to overwhelm your system. Why? Because as soon as your client falls behind in reading the stream’s contents, the service will stop transmitting to your client until it can catch up; and as soon as it does the service will indicate to your client how many posts they missed.

What follows is a simple Perl script, that illustrates how easy it is connect and consume the stream. It utilizes Miyagawa’s XML::Atom::Stream perl module to connect to the stream and display all the titles of all the posts created by Six Apart users and customers.

Note: We would love to build a library of clients capable of consuming this stream, so if you have written a client in .NET, PHP, Ruby, Java, or any other language, please share it with us. We would love to link to it and share it with others.

Resources

• Six Apart Updates Homepage

Clients

• Perl
• Python