Scala comes to .Net

Miguel Garcia, part of the Scala group at EPFL, has been striving to make the Scala productivity available to .Net developers too. In a project funded by Microsoft, he has now reached a major milestone in delivering that capability. In this interview Miguel tells you how you can now use Scala on .Net and gives an insight into the way this challenging project has been achieved and the future direction.

Why do people need Scala on .Net?

Miguel:

There are many reasons why this is attractive both to the developer and to companies in general. Scala is a concise and highly productive programming language. By using Scala on .Net, developers can produce applications more quickly and have the possibility of deploying them across the two major industry platforms, JVM and .Net.

For the programmer it makes a great deal more sense too, learning one language to build applications for both environments. From a company point of view scarce resources, good developers, can be moved easily between platforms reducing training costs, increasing flexibility and reducing risks. .Net provides an integration platform for several languages and this implementation of Scala inter-operates nicely. You can use existing .Net libraries or applications without re-writing everything in Scala. A low risk way to explore the potential of Scala. Ultimately it means that many tools and applications created for the .Net and JVM environments can be ported from one to the other. A win-win situation for everyone.

As a Scala developer you will hardly notice that you are running on .Net.

Can I run Scala programs on .Net now?

Miguel:

The simple answer is yes, with a few limitations that will be remove by the fall.

If you have a Scala program working on the JVM then for the most part you only need to recompile it with the Scala.Net compiler and it will run there too.

The key limitation for the moment is that Scala programs cannot use libraries in .Net that are compiled using CLR generics, such as the .Net collections. However, for this particular example that's a minor limitation as the much better Scala collections library would be the natural developer choice anyway. Just to be clear: it's only CLR generics that are not implemented in the Scala.Net compiler, all the Scala features (including Scala generics) are available to the programmer. The .Net generics will be supported in the fall.

How do you develop programs?

Miguel:

You can develop Scala programs directly in MS Visual Studio. In the fall the Visual Studio plug-in for Scala will support the basic functionality expected of the IDE namely, code completion, code browsing, line breaks, expression breaks and exception reporting and so on. However, in the short term the current plug-in does not yet support code completion or code browsing. For that reason some developers may prefer to write code on the JVM using Eclipse or Intellij and then do final debugging under Visual Studio. You can find a "how to use" guide here and a slide presentation overview of Scala.Net project here.

The current implementation is already quite robust. The Scala.Net compiler does compile itself, some 100k lines or so of Scala code.

How did you make this all work?

Miguel:

The main difficulty to overcome is that Scala programs make heavy use of the Java JDK and this functionality is naturally not supported under .Net and the Scala compiler must generate code suitable for .Net

The Java community have already created IKVM which allows Java programs to be converted to MSIL and a .Net library that provides the basic JDK support. This has been a long running open source project driven by Jeroen Frijters. Its motivation is turning Java into a "write once run anywhere". IKVM has been in existence for a long time and is a very reliable open source project with commercial support. The IKVM library has been the critical enabler for bootstrapping the compiler on .Net and has proved to be an excellent foundation for the effort to port the Scala library to .Net.

Scala .Net has a rich history, not seen by people outside the Compiler hacker community. There have been several people like Nikolay Mihaylov and Lukas Rytz who have made progress on creating a cross-compiler. I picked up at the point where we had a cross-compiler but was unable to bootstrap itself in the .Net environment. That means having an executable of Scala.net that compiles its own sources. This is a classic chicken and egg problem of colossal proportions.

The cross-compiler was an excellent piece of work but ran on JDK only. Its sources had a lot of dependencies on JDK APIs. This is pretty typical of any Scala applications. Scala programmers still rely on features provided by the JDK, although many times the equivalent can be found in the Scala SDK. We had the cross-compiler but missed a way to automatically replace the JDK dependencies with calls to libraries in the .Net environment.

The IKVM compiler takes JVM byte code and turns it into .net assembly. This byte code level change, although a big step, does not in itself bring us to true bootstrapping because we are still left with the JDK dependencies. The task therefore became creating a pre-processor that would modify the Scala source code directly replacing JDK dependencies with IKVM .Net equivalents, a major mapping exercise.

How exactly do you do this mapping?

Miguel:

The first step was to create an initial set of transforms that would be needed at the source level. The next stage was to extend the Scala compiler to apply the transforms. Not all compilers are flexible enough or provide these types of services, however. The Scala compiler is very modular and supports plug-ins so it was a lot easier to adapt. A new Scala plugin JDK2IKVM was created that applies the source transform in a batch mode. The resulting source code then passes through the rest of the standard compiler phases producing MSIL byte-code with IKVM library references. Voila, we have Scala.Net, a Scala compiler and it can compile itself. Bootstrapping achieved. In reality we had to iterate around the compile/transform several times to capture all the needed transform cases.

Behind the scenes we use a refactoring technology and its technique that could be used in other migration scenarios. For example convert from Java.IO to Scala.IO automatically. You can learn about these techniques and all the things we are doing for porting Scala code to other platforms at The Scala Compiler Corner. If you are a compiler hacker then there is lots of fascinating stuff to see.

Of course the Scala compiler is just another Scala application so it is easy to understand that this process can be used for any Scala application developed for the JVM.

How about Primitive Type compatibility?

Miguel:

Primitive types offered by the .Net virtual machine CLR are a superset of the JVM types. There's also value types which are like records. From a compiler writer point of view it is easy to convert from one to the other. Strings work well with the transform procedure. It is unlikely that any corner cases have been missed as they have been well tested. The IKVM has had many years of use in the IKVM community.

What comes next?

Miguel:

As mentioned before we will be extending the Visual Studio plug-in to give the full developer IDE experience and adding support of .Net generics. In fact, the debugging experience is already quite usable, for example rather than a whole line at a time one can step one sub-expression at a time.

However there are a few parts of the SDK will require a modification to run on the .Net environment. One example is the parallel collections which will require that the internal scheduling and thread management is made .Net thread model compliant. Dealing with this and other examples will have a nice payoff, programs that benefit from this type of performance boost will get it on .net too.

The transform map will be extended to support some basic libraries that are commonly used by developers, IO for example.

Are there any recommendations for users/developers?

Miguel:

Yes, the how-to for Scala.Net shows a technique to have .Net API's available as stubs that makes them visible for code completion on JVM based Scala IDEs. Once the application has been developed then it can be moved to .net. This is just an interim workaround until the full scala plug-in is available for Visual Studio.

But most of all start to use Scala on .Net. The more feedback I get, the better I can make it work for you.