Swarm - Concurrency with Scala Continuations

With Swarm [1], Ian Clarke demonstrates a novel approach to programming distributed systems using Scala's serializable delimited continuations. Ian, CEO of Uprizer Labs and developer of SenseArray [2], is possibly best known for designing and coordinating Freenet [3], a popular P2P distributed network storage application.

He presented Swarm at the IEEE Ninth International Conference [4] on Peer-to-Peer Computing in Seattle,USA that took place September 9-11. You can watch a video where he describes Swarm [5].

As recently announced [6], continuations will be included in Scala 2.8 as a compiler plug-in. Ian has innovatively applied this technology to create Swarm, a potentially ground breaking Open Source project that significantly eases the writing of programmes for loosely and tightly coupled multiple processors environments.

The delimited continuations feature of Scala allows you to save a function state and resume it later. Ian explains that, unlike other languages, the Scala implementation is unique in that the resulting object is portable; it can be transferred to another computer and resumed there. He then shows how this enables Swarm to move computations, in an efficient and potentially transparent way, between processors to operate on distributed data structures.

Ian demonstrates a proof of concept prototype and outlines data clustering concepts to be used. He finishes his presentation by outlining some of the challenges that lay ahead, the arrangement of the data for optimal efficiency, GC, multi-threading, class loader for a Global namespace and creating a web framework to use Swarm.

If you want to work on state-of-the-art Scala applications, this would be a fascinating project to join. See more about the project and source code here [7].

Other groups see continuations as a powerful programming tool too. Jay A. McCarthy from Brigham Young University, Provo, UT, USA presented a paper at ICFP 2009 about "Automatically RESTful Web Applications" [8] He reported having good results by not keeping session state in web servers but using serializable continuations to keep the state exclusively on the clients.

Jay McCarthy says "Our programs use the same features as non-scalable, continuation-based Web programs, so we do not sacrifice expressive power for performance. In particular, we allow continuation marks in Web programs. Our system uses 10 percent (or less) of the memory required by previous approaches."

Although Jay McCarthy used PLT Scheme, you could use the same technique just as well with the Scala continuations.