Scala at LinkedIn

The LinkedIn Web site was launched in 2003 and is now the largest professional networking site in the world with more than 65 million members, representing 200 countries and executives from every Fortune 500 company.

Chris Conrad, Engineering Manager, a part of the Search, Network and Analytics team at LinkedIn presented Norbert at ScalaDays 2010. Chris works on the LinkedIn Social Graph represented by some 65+ million nodes, 680+ million edges and 250+ million request per day.

He explains that their People Search Engine receives around 15 million queries a day, or 250 queries per second with up to 100 tokens per query. Queries are satisfied by a scatter-gather approach across a large server farm which presented a challenge for efficient message routing and resource managemet, reliable 24x7 operation and easy application development.

Norbert is a framework written in Scala that makes it fast and easy to write asynchronous, cluster aware, message based client/server applications. Built on Apache ZooKeeper and JBoss Netty, Norbert provides out of the box support for notifications of cluster topology changes, application specific routing and load balancing, scatter/gather style APIs and partitioned workloads. Written by the SNA team at LinkedIn, Norbert is open sourced under the Apache License.

Chris explains that they use Scala because it makes concurrent support easier with Actors and promotes code reuse with traits and mixins. He found it really nice that the "cake pattern" could be implemented elegantly using self types to declare dependencies and have the compiler do the checking. He says that the "killer feature [of Scala] is the seamless integration of Java and Scala, making it low risk to introduce, and reduces the overhead to experiment. It makes software development more fun and a lot less frustrating."