Scala in the Browser?

Grzegorz Kossakowski, now studying for a Masters with the Scala team at EPFL, has been working on a research project to enable Scala to be used directly to write browser applications. In this interview he explains how he has made it possible for anyone to write Scala browser screen code in Eclipse and a few seconds later see the results in a browser. A real neat trick using GWT!

He has created a 7 minute YouTube video where he walks through the steps of writing a Scala program, deploying to a browser and debugging it interactively. Click Read More to read the rest of the interview.

As he explains, this is a proof-of-concept research project and not a 'product' announcement. That said, the project is remarkably mature and you can down load the tool set to give it a whirl, instructions here. If you like it, then Grzegorz would welcome feedback and contributors. This fascinating research was sponsored through internships first at Google with Lex Spoon and then recently at Typesafe with Martin Odersky.

How did it all happen?

I was keen to do an internship at Google. Happily they were interested and started looking for a a suitable project. I had liked Scala for some time and I discovered Lex Spoon, a well know Scala enthusiast was striving to make Scala and GWT work together. GWT is a project that allows you to take Java programs and run them in the browser environment using translation into Javascript. Lex wanted to enable people to write Scala and execute it directly in the browser and like Java uses GWT. Luckily for me he was too busy to work his dream project so I had found a cool job making it work and Lex as a great host.

Time ran out with Lex and the project became a personal hobby. I got lucky again. Martin Odersky at Typesafe liked the concept and Typesafe sponsored me to finish the project, taking me on as an intern. It was a tremendous environment, ready help from people like Martin, Adrian and Lukas from EPFL and Iulian from Typesafe.

What's the project status?

The Scala to Javascript project is a research project and not yet what we could call a product. It has got quite mature and you can really play with it to create quite complex browser projects in Scala using the Eclipse IDE. The YouTube demo is 4500 lines of Scala that exercises just about every aspect and widget of GWT. You can have fun, but it's not quite at the bet-your-job on it stage yet. There are bugs. We need quite a bit of work on documentation and smoothing the edges. Still need to work out how to structure projects, how to manage releases, create a library that wraps more of the GWT APIs. There are lots of interesting but manageable things to do that can advance the project quickly, just the sort of things that the community could help with and a great time to join the project

What is GWT?

The simplest way of thinking about GWT is that it is a Java to Javascript compiler. You write Java and GWT compiles it to Javascript that then runs in a browser. The idea is that simple. Making it happen is a little more difficult. GWT has to provide things like emulation for Java standard libraries, basic types and collections and then at run time emulates jvm behaviour for system services like exception handling, primitive type emmulation and so forth. The Scala implementation uses the same infrastructure.

GWT, a project with growing interest, provided a wonderful starting point for my own Scala project. It resulted from the dedicated work of great bunch of developers, people like, Lex Spoon, Rob Heittman, Aaron Novstrup and Stephen Haberman.

The Scala+GWT project has been running for over a year and during most of that time it was not really usable at all for real programs. Compiler projects are different to other projects. You need 90% of the functionality before even simple programs can run. Then the last 10% is a struggle as you discover the edge cases. For a developer, it's really frustrating. You work for months before seeing something work at all. It was just the same for the Scala+GWT project. Quite a bit of contributor interest at the begining but it soon faded away, nothing to try out. Compiler projects are difficult to join part-time. The learning curve is just too long and steep. The Scala compiler is sophisticated and intricate, a huge challenge for any newcomers. On top of that GWT is another compiler with another set of challenges. It's a lot to swallow.

We have got there. Now we have a working prototype that anyone can use and there are lots of areas people can usefully contribute without diving into the guts of the compiler. We finally got to the fun phase, making new releases every couple of weeks adding features that developers care about, like speeding up the compilation or making development in the Eclipse IDE really nice. Now we are seeing a lot of interest.

Why Scala instead of Java?

Using GWT libraries and abstractions to program widgets forces you to produce lots of boiler plate. For example event handling is implemented as an anonymous class which requires many lines of code. Very much a consequence of the Java idiom being carried into GWT.

Typically I found that what would be 20-30 lines of Java code turns into a few lines of Scala. GWT helps you write the boiler plate but it is very verbose and hard to maintain. With Scala you get the style freedom of Javascript but with all the static type safety, and that's a real help. You have all the same familiar Scala tools, the standard libraries and the enormously powerful Scala collections. 

My personal motivation for doing the project was knowing how much nicer it is to work in Scala. I wanted to bring that to browser application development. Now anyone who is familiar with Scala and Swing can easily write Scala applications for deployment in a browser. The API's are slightly different but the style and basic abstractions are the same. That's neat and a joy to use. Moreover, we are not restricted to GWT APIs. People looking for Scala2JS compiler should give Scala+GWT a try, too.

What were the challeges?

Getting some of the Scala libraries to run on GWT required some thought. Several of them use reflection and GWT does not support that feature. Platform specific dependencies like IO or JVM specific APIs are a problem too. Most Scala libraries, those not using low level platform services, should work with little or no modification. No real limitation as you do not general expect to do IO in a browser, it's restricted by the browser security layer.

To do this we took two really complicate projects the GWT and Scala compilers modifying each to integrate with the other. The Scala back-end was modified to produce "jribble", a human readable bridge language that we designed to suit the needs of GWT and GWT was then modified to compile it.

Jribble was designed by starting with Java and removing all restrictions which are problematic for Scala. At the same time keeping it structured so that GWT can compile it successfully. It required a lot of work to get this right. Array handling is difficult. In Java they are special citizen and not erased while in Scala they are a regular generic collection and you can use it like any other generic collection. The Scala compiler uses lots of tricks to make arrays pleasant to use and generate efficient byte code for the JVM. We had to create a completely new implementation of those optimisations for GWT but without the help of reflection. A very intricate and tricky task.

Primitive types have different ranges and there is no counterpart for Java’s Double in Javascript. Here we could go quickly as it was already taken care of by GWT for the Java emmulation. Scala uses Java primitives so one less thing to worry about.

We started with jribble having a similar syntax to Java and used the parser combinators from Scala in GWT to compile the language. Although very flexible and easy to implement it was slow. We eventually abandoned it and encode jribble as google protobuf messages. They are already used in GWT so no increase in dependencies and they offer a very compact, fast to read binary representation that did not need a parser. Changing from Jribble to proto path reduced the 'parsing' phase from 120 seconds to 1 sec for our 4,500 line Scala demo and some 40,000 jribble lines.   http://code.google.com/apis/protocolbuffers/docs/overview.html

This optimisation more than halved the overall compile-to-server-deployed Javascript turnaround time from 5 minutes to 2 minutes. It takes takes a few seconds to do the same thing in Developer mode.

What is Developer Mode?

In the development mode with Eclipse the process is different. The compiler generates JVM byte code for the Scala code just as it normally would. GWT then runs this code in a JVM emmulator. When the code executing in the JVM needs to read or change browser state, this is done by sending an RPC to an attached browser which sends the result back to the JVM. When you change a line and recompile only the byte code classes that have been changed need to be reloaded in the jvm, rather than compiling everything right through to Javascript. This is very quick and takes a few seconds rather than minutes for the full server Javascript application build, even for a big application. 

The developer gets a really responsive and quick turn around to code changes. They can see the effects of each change in the browser as Scala code is developed. All the JVM excellent debugger tools are available too. Break points can be set, exceptions caught and single stepping used. Scala refactoring tools are so much better than with Javascript. See for yourself in the YouTube demo or try yourself by installing the Eclipse plugin.

When you are satisfied with your application then you can invoke deployment mode to create the server Javascript version that will run in the client browser.

How does it perform?

If you write the same application in Javascript and Scala the performance varies from the same as Javascript to around 10 times slower for heavy collections use. The collections cause lots of method calls to be generated that need to be emulated in Javascript with indirections. If you have a list of 100 elements and a Map this may be around 10 times slower than a native Javascript equivalent.

However modern browser performance and Javascript optimisation are continually advancing. For example the new Chrome  V8 runs our Scala Sudoku test application ten times faster than the current version of Firefox.

We can see how to optimise the GWT code generation too to close the performance gap. So still some work to do! As you know for most browser code performance is less important than shorter development time. Scala and GWT come with a very rich libraries for creating and interacting with widgets. You can inject HTML where ever you want or manipulate DOM directly and many services already have Scala wrappers. 

While we are talking about HTML I should mention that in Java you must be careful to avoid escaping strings, especially if getting input from a user. You can easily get script injection problems. With Scala it is so much safer as HTML literals are supported and escaping strings are handled automatically. The Scala compiler catches these problems for you. One less thing to worry about.

What's next?

There is no real support for multi-threading in todays browsers. Parallel Collections are not currently supported, but as multi-threaded browser standards firm up we will be ready to add support for the primitives that underpin Scalas parallel collection, fork-join and work stealing. Then it will be a simple matter to turn on the parallel collections capablity. The same applies to actors. It would be neat to have Akka actors running both in a browser and on a server and let them communicate seamlessly.

For a real product we need to integrate our changes with upstream projects like GWT and Scala. Right from the beginning Lex was really careful to design our required changes in such a way that we do not disrupt the existing design of projects. Changes are orthogonal to existing APIs, they carefully thought out not to alter existing APIs or functionality for current users. So from a technical point of view it should be straight forward to merge the changes. This means that people will be able to just download current versions of GWT and Scala and be ready to go. No more messing about.

There is a technically challenge with pattern matching. In Scala it is very efficient and optimised for the JVM. It has been hard to make this efficient in the generated Javascript code.  For the moment the compiler will flag certain valid Scala pattern match cases as problematic that you must modify to meet the current constraints of the implementation. However, Adriaan Moors is working on an new, at the moment experimental, re-implementation of pattern matching that would allow us to also generate efficient pattern matching Javascript. With this, the last outstanding constraint resolved we get 100% coverage of Scala language features.

Who will use Scala GWT?   

I don't think Scala GWT is a silver bullet for all browser application development, but a great place for people who want to write Scala and not learn Javascript or people who are writing sophisticated financial applications and need Scala’s power. Great for week-end projects to create browser applications.

Java developers already using GWT would find it much more productive with much less boilerplate frustration.

What about you?

Now working on my Master thesis at EPFL but 25% of my time still available to lead this project. I will be re-organising the project resources to make them easy for people use and contribute. I would love to help contributors get started, perhaps on wrappers around GWT libraries that are more idiomatic Scala or documentation. One of my first tasks is to help Adriaan with the pattern matching changes. Also, I’ll be supporting project by promptly fixing bugs reported by our users.

My own Masters research project is into a DSL capability to generate Javascript from Scala and perhaps to look at a Scala backend to generate Javascript directly. Exciting stuff. A busy time ahead.