LinkedIn Engineering | Status Feature

With the introduction of the Status feature, I wanted to take a moment to discuss the engineering behind this launch. Since joining LinkedIn in September 2006, I've been helping evolve the communications system into a highly scalable messaging platform. This release is very exciting for us as it builds upon the foundation we've put in place, making features like Status possible.

One of the key value propositions of LinkedIn is being informed about what your network is doing.  This awareness can come in many forms: what questions are my connections asking?  What news is my network reading?  Propagating this information out to the LinkedIn network in a scalable manner is a difficult problem, but provides tremendous value to the end user.  In order to support  this going forward, we began to create a unified service that would provide network updates to users.

We've converted our back end to use this new service, and late last year we used the new Network Updates Service to create a time-based feed of the events in your network.  We are currently processing over 40 million updates posted daily, which will continue to increase as the network grows.  For someone like myself who has recently graduated from university, this is a daunting number, and yet, it's why I continue to enjoy working at LinkedIn.

To handle this capacity, we use a cluster of ActiveMQ machines for distributed JMS processing, EHCache for fast in-process caching, and an efficient update procedure for when new updates are persisted.  We've designed the system to be scalable right from the start, horizontally partitioning across multiple databases as growth demands.

We will continue to provide new updates into your feed, and for those of you who do not visit the site regularly, you will be receiving these updates in a bi-weekly email so you can be kept up to date with the latest activity in your network. I would like to thank the team that worked on the Status feature, particularly Scott Schlegel (Web Development), James Lau (Engineering), and Qian Su (Engineering).

If you would like to hear more details our messaging platform, or engineering at LinkedIn in general, I will be speaking at some upcoming conferences along with Ruslan Belkin and Nicholas Dellamaggiore:

- SDWest, March 4, 7:00pm - 8:30.
- JavaOne, May 3-6 2008
- Velocity, June 23-24, 2008, Burlingame CA

And, at the events particularly the SD West Birds of a Feather event, "There Will be Pizza"!