Tech Talk: Prof. John Ousterhout on RAMCloud
October 21, 2011
This is a part of our continuing series on engineering and analytics at LinkedIn. This post covers our most recent Tech Talk featuring Prof. John Ousterhout of Stanford. If this isn’t your cup of Java, check back tomorrow for regular LinkedIn programming. Else, check out our Engineering Blog. - Ed.
Performance of disk based systems has not kept pace and handling the scale that large web applications demand has been a challenge. Developers have to rely on aggressive caching which creates its own set of problems. In his talk, Prof Ousterhout talked about an entirely new approach of dealing with this problem i.e. shift the focus on online data storage from disk to DRAM. While DRAM is still more expensive than disk, for lot of online data, DRAM is cheap enough that most of online data can now fit in RAM.
RAMCloud is a new general purpose storage system being developed at Stanford that keeps all data in DRAM and uses disk only for backup. It is built to scale to 1000+ servers holding 100+ TB data. While each DRAM based server is able to deliver much higher throughput than a disk based one, the biggest impact comes from the latency which is many orders of magnitude lower. For web applications, this translates into ability to perform many more lookups per page, which means richer user experience.
During rest of his talk, Prof. Ousterhout explained how crash recovery is done in RAMCloud which has some very innovative ways of speeding up recovery while keeping the memory footprint as low as possible.
Watch the above video for more details!