Binary compatibility: status and outlook

When 2.8 was around the corner there was a big discussion about (the lack of) binary compatibility in Scala.

My answer then was that we would keep binary compatibility for minor revisions and look into ways how to address the problem for major revisions. The first half of the promise was kept. 2.8.1 was binary compatible with 2.8. Now that 2.9 is around the corner, the question is what can we do for binary compatibility now and in the future?

If you are too impatient to read the details: we not there yet, but are making progress. In particular, we have developed the basic technology that lets us maintain binary compatibility for future releases.

Now to the details: There are in fact many sources of binary compatibility. Some of the most common ones are:

1. New methods or fields are added to existing traits. Implementations (but not clients) of these traits need to be recompiled.

For instance, the new collection libraries add methods `par' and `seq' to all collection traits, as well as several other methods.

2. Classes and companion objects are moved to different packages, with type aliases and forwarders in a package object in the old location. Any clients of those classes have to be recompiled.

For instance, the 2.9 standard library has moved several annotation classes from package `scala' to package `scala.annotation'.

3. Lazy vals are added to a class. This shifts the indices
used to access bitmaps in all subclasses.

4. Methods get different signatures. Parameter types might be widened, or result types narrowed or widened. Again, all callers of such a method need to be recompiled.

For instance, collection operations such as `zip' used to take an Iterable as argument, now they take a GenIterable (GenIterable is the common superclass of Iterable and ParIterable). There are many other examples like this.

What can we do to find and fix these? We have been working on a tool which will detect binary compatilities between different versions of a library. The tool is in prototype stage now. We will make it available as soon as it is in a usable state. With the tool, we can have a better idea where binary incompatibilities have been generated, and can roll back if they are accidental.

Some binary incompatibilities are not accidental, but the result of conscious generalizations and enrichments of the libraries. We do not want to sacrifice these possible improvements for binary compatibility. What we do instead is search for technological solutions. For problems of the first three kinds (new trait members, moved classes, new lazy vals), we are working on a tool that can fix these issues by rewriting code on the client side. For problems of the third kind (changed method signatures), we have developed a new solution based on bridge methods. Let's say you have a method

  def combine(other: Seq[String]): Unit

and you want to generalize the method to take Traversables instead:

  def combine(other: Traversable[String]): Unit

That change is source compatible (all code calling the old `combine' will work with the new one), but it is not binary compatible, because the new method handle has type Traversable in its signature where the old one had Seq. Since callers refer to a method by giving its name and full signature, callers compiled against the old method will not work with the new one.

To fix this kind of problem (which can be diagnosed using our tool), we can add a bridge method:

@bridge def combine(other: Seq[String]): Unit =
  combine(other: Traversable[String])

Since there is now a method with the right signature, binary compatibility is maintained. At the same time, the bridge method will be invisible to freshly compiled Scala code, so all recompiled code will pick up only the new method, not the bridge. This also ensures that bridges do not affect overloading or overriding relationships for the other methods; they are purely there for bytecode generation. Using bridge methods, we can compensate the significant changes caused by the arrival of parallel collections in 2.9.

So these are the schemes we are working on. We are reasonably confident that they will solve Scala's binary compatibility once they are ready. For 2.9, bridge methods ensure that quite a lot of code compiled against 2.8 will continue to operate, but my no means all code. So it is very much advised to recompile your projects for 2.9. Recompilation should in most cases be painless because 2.9 is by-and-large source compatible with 2.8. There's one exception: If your application uses features that were already deprecated in 2.8, it might find these features removed in 2.9. So it's a good idea to get rid of deprecation warnings before upgrading.

  -- Martin

----------------------------------------------
Martin Odersky
Prof., EPFL and CEO, Scala Solutions
PSED, 1015 Lausanne, Switzerland