Introducing Scalafix: a code migration tool for Scala

I am excited to announce the first release of scalafix, a new tool to prepare Scala code for future Scala versions, including Scala 2.13 and beyond as well as Dotty, a next generation Scala compiler.

This effort follows the Scala Center Advisory Board proposal: “Clarification of Scala to Dotty migration path”.

What can scalafix do?

Scalafix takes care of easy, repetitive and tedious code transformations so you can focus on the changes that truly deserve your attention. In a nutshell, scalafix reads a source file, transforms usage of unsupported features into newer alternatives, and writes the final result back to the original source file. Scalafix aims to automate the migration of as many unsupported features as possible. There will be cases where scalafix is unable to perform automatic migration. In such situations, scalafix will point to the offending source line and provide instructions on how to refactor the code. The objective is that your existing Scala code base gets up and running faster with new Scala versions.

Scalafix and Scala 2

Scala 2.13 is still being planned, so we don’t know yet exactly what language changes it will include. But with Scalafix available to help migrate existing code, changes could become possible that would previously have been considered too disruptive because they would require too much code to be updated by hand. Possible examples include:

• Rewriting code to avoid the use of the now-disfavored language.postfixOps syntax
• Deprecation of any2StringAdd, in favor of string interpolation
• Deprecation of procedure syntax, as in Dotty
• Changes to the Scala collections API

Scalafix and Dotty

There is a lot to be excited over in Dotty, faster compilation times, faster+smaller binaries and awesome error messages to name a few things. However, some Scala 2.x applications can’t immediately enjoy these benefits due to several breaking changes in Dotty. Scalafix is part of our strategy to smoothen the migration process for those applications.

What can scalafix do?

Scalafix comes with a command line interface and an SBT plugin. Running scalafix is as simple as:

# CLI
scalafix Code.scala
# SBT, after adding plugin to project/plugins.sbt
sbt scalafix

More detailed instructions on how to setup scalafix are on the project website.

This initial release implements two rewrites: ProcedureSyntax and VolatileLazyVal. More rewrite rules are planned for future releases.

The ProcedureSyntax rewrite works like this:

// before
def main(args: Seq[String]) { // note lack of '='
  println("Hello scalafix!")
}
// after
def main(args: Seq[String]): Unit = { // note ': Unit ='
  println("Hello scalafix!")
}

Deprecating or dropping procedure syntax is part of a general effort to simplify the Scala language.

The VolatileLazyVal rewrite adds Dotty’s @volatile annotation to lazy vals, like this:

lazy val x = ... // before
@volatile lazy val x = ... // after

With @volatile, Dotty uses a deadlock free scheme that is comparable-if-not-faster than the scheme used in scalac.

How does scalafix work?

All the heavy-lifting behind scalafix happens in scala.meta, a new metaprogramming toolkit aspiring to succeed scala.reflect. Scala.meta makes it easy to do non-trivial code transformations while preserving syntactic details in the original source file. This key attribute of scala.meta makes it suitable for both developer tools like scalafix as well as compile-time metaprograms like macros.

The grand vision of scala.meta is to provide a single interface to accomplish most common metaprogramming tasks in Scala. The idea is to untie metaprograms from compiler implementations, providing a platform-independent API that works both in the Scala compiler, Dotty and tools like IntelliJ IDEA. Metaprogram authors benefit from a user-friendly interface built on immutable ADTs and type classes. Compiler authors benefit from the fact that they don’t need to expose compiler internals in order to support popular features like macros.

In scalafix, we use the scala.meta API to parse Scala source files, traverse abstract syntax trees, inspect tokens (including comments!) and then perform the minimum required transformations to the source file. Thanks to scala.meta dialects, which abstract away syntactic differences between variants of Scala, we can use the same programs to manipulate regular Scala source files, SBT files as well as code that uses new Dotty syntax such as union types and trait parameters. All this powerful functionality is implemented behind a minimalistic interface. The NonVolatileLazyVal implementation is 15 lines of code. It could comfortably fit in three tweets.

We would not be able to achieve this level of expressiveness with scala.reflect, the current standard metaprogramming framework for Scala. Why? To name a few reasons, scala.reflect does not preserve syntactic details such as comments and scala.reflect has no notions of dialects, so it is bound to only support Scala 2.x.

How does scalafix evolve?

As we have seen above, the functionality of scalafix heavily relies on the infrastructure provided by scala.meta. However, in order to write more sophisticated scalafix rewrite rules, there are two main features missing in scala.meta, namely a scalac converter and semantic API. We are closely collaborating with Eugene Burmako, the lead developer of scala.meta, in the development efforts of these two features.

Scalac converter

The scalac converter is a key subsystem of scala.meta. For every Scala feature, the converter recognizes its representation in the Scala compiler and translates it into the corresponding data structures of the scala.meta API.

Over the last months, we’ve made great improvements to the converter, which is under development in the scalamacros/paradise repository. We have established a comprehensive test suite that consists of a sample of over 26.000 Scala source files collected from popular open-source projects. Thanks to the joint effort of the team of contributors, the converter now supports about 19.000 source files form the test corpus. We are continuously working to increase that number, aiming to bring language coverage to 100%.

Semantic API

The semantic API will enable scala.meta programs to inspect compiler information such as type inference, name resolution and other functionality that requires a compilation step. This opens opportunities for new scalafix rewrite rules that cannot be done on a purely syntactic level, like NonVolatileLazyVal and ProcedureSyntax.

We already have a working prototype of the first scalafix rewrite that uses the semantic API: ExplicitImplicit. ExplicitImplicit inserts type annotations for implicit definitions, like this:

// before
implicit val x = (_: String).length
// after
implicit val x: String => Int = (_: String).length

ExplicitImplicit requires the semantic API in order to get the inferred type annotations.

Towards new-style macros

One could say that by contributing to scala.meta, we get two features for the price of one. First, an improved converter and semantic API enables us to implement more sophisticated scalafix rewrite rules. Secondly, we accelerate the development of a new macro system that will work both with scalac and Dotty.

As announced at Scala Days 2016, the current macros based on scala.reflect will be going away. Such macros have a number of known issues, including over complicated API and lacking IDE support. More importantly, the deep coupling between old-style macros and scalac means macros written in the old system effectively cannot be ported to Dotty. The plan is to discontinue support for scala.reflect macros in favor of a new macro system called inline/meta.

A technology preview of the new-style macros came out this summer. This early release is still somewhat rough around the edges. Nevertheless, scalafmt is already using these new macro annotations in production for more than a month now. If you are interested in learning more, the best place to get started is the Whirlwind Tour of scala.meta.

Much like scalafix, new-style macros crucially rely on the infrastructure provided by scala.meta. Concretely, the converter and the semantic APIs constitute a significant chunk of the implementation effort behind the new macro system.

Interestingly, our contributions to scala.meta were originally motivated by the needs of scalafix. However, they are also immediately helping the development of this new universal macro system for Scala. We are excited that our open-source collaboration with the scala.meta team brings multiple benefits for the good of everyone!

Get involved

Are you interested in metaprogramming, developer tools and running experiments on millions of lines of Scala code? Come chat with us on Gitter, and we’ll discuss how you can make a difference in shaping the developer tools of the future!