Non-Java JVM Language Interop

This is a four-days-overdue report from a meeting that several of us had at OSCON on the subject of interop between non-Java languages on the JVM.  Specifically, those in attendance were as follows: Daniel Spiewak, Ola Bini, Carson Gross, Ben Evans, Stuart Sierra, Aaron Bedra, Martijn Verburg, Charles Nutter.  (my apologies if I forgot anyone, I'm trying to reconstruct this list from memory in clockwise order; I'm pretty sure this is accurate)  In case you're too lazy to use Google, the represented languages were as follows: Scala, Ioke, Seph, Gosu, Java, Clojure, JRuby, Mirah.

I have a full set of meeting notes, but I can very effectively boil it all down to the following three points:

• Static/Static interop is easy.  Dynamic/Dynamic interop is even easier.  The hard problem is Static/Dynamic.  If we can solve that problem, then the other two will be trivial.
• We need a set of BiConverter interfaces to handle runtime value conversion between languages.  The most important ones are as follows:
• String
• Primitives (or rather, what would be primitives in Java)
• Functions (note: Scala overloads arities 0-22, but most languages do far less)
• Collections
• Dynalink is the bee's knees.  It solves a whole host of interop problems that would be either impossible or incredibly class-stub heavy.  Unfortunately, it does this by basing itself entirely on invokedynamic.  The good news is that this allows it to be both ridiculously fast and amazingly flexible.  The general conclusion is that the Java 7 requirement is probably not going to be an issue, since the only people who want Scala/JRuby/Gosu interop are likely to be early adopters.

Note that this will probably become a JSR eventually, but we need to hash out some of the details first.

Another thing we discussed as a very preliminary idea was a second level to this interop layer, one which could run ahead-of-time to produce class stubs from dynamic constructs.  A good existing example of this functionality is JRuby's Java class compiler, which can produce real Java classes (with methods and everything!) from most Ruby classes.  This would be a separate tool that would depend on the runtime layer and would allow much tighter integration with static languages.

There was some discussion of having all of us implement a common set of interfaces for basic concepts.  Functions were the specific example.  This would allow Scala functions to be used by Gosu, JRuby, etc and vice versa.  Unfortunately, it pollutes everyone's class hierarchies, so we came away generally decided that this wasn't such a hot idea.  Unfortunately, this means a lot of wrapping (using BitConverters), which is both slow and quite problematic in terms of object identity.  Not so much a problem for functions, but definitely more concerning for things like collections.  We didn't come up with any bright ideas in this area, though dynalink magic could solve some of these problems for certain limited interop scenarios.

The immediate action items for us are to look at dynalink and think about what sort of conversion interfaces we might need.  It probably makes sense for this effort to center on dynalink anyway since that's an established project which is already addressing this niche.  I'm not sure there is anything immediate for the Scala core team to deal with, but this is probably something with which we want to maintain involvement.  Interop is good for everyone!

Daniel

P.S.  I know a few of the folks at the meeting do monitor this list.  Please feel free to chime in with anything I forgot.