Compiling Scala library in limited (GWT) environment

Hi,
This e-mail is an experience report on my attempts to compile Scala library in a limited Java-like environment. My attempts highlight some problems with current library design and implementation and I hope that my report will be a good starting point for discussing some refactorings.
GWT implements subset of Java's standard library and allows to use it in a browser. Things that are not implemented include reflection, i/o, multi-threading and many other little things.
My approach was to start with src/library directory and start applying source-to-source transformations using Miguel's jdk2ikvm tool. Below you'll find summary of patches I had to apply in order to compile large subset of src/library source code. Every chapter will start with a list of files I had to delete (with some comments explaining why) and optional description of patches I had to apply to remaining files in a given package.
Parallel collectionsSince multi-threading is not supported I had to exclude entire parallel collections. In order to do that I deleted following files/directories: "scala/collection/parallel","scala/collection/Parallelizable.scala", "scala/collection/CustomParallelizable.scala","scala/collection/generic/GenericParCompanion.scala", "scala/collection/generic/GenericParTemplate.scala","scala/collection/generic/GenericParTemplate.scala","scala/collection/generic/ParFactory.scala","scala/collection/generic/ParMapFactory.scala", "scala/collection/generic/ParSetFactory.scala","scala/collection/generic/CanCombineFrom.scala","scala/collection/generic/HasNewCombiner.scala",
One thing one notes is that I had to explicitly list a lot of traits in generic package for exclusion. It would be nice if we had generic.parallel or parallel.generic package where all of those traits would be located. This way I could easily exclude them with just one rule.
Following transformations applied to remaining source code:

• remove parents of collection traits that got excluded above
• get rid of par/parCombiner definitions

Overall, I must say that parallel collections were reasonably easy to deal with.
Rest of collectionsI also excluded following files://depends on threads"scala/collection/generic/Signalling.scala", //depends on scala.io"scala/collection/immutable/PagedSeq.scala",//javascript has nothing corresponding to weak refs"scala/collection/mutable/WeakHashMap.scala",
Moreover, I had to exclude classes/methods implementing serialization support in collections. Fortunately enough it's easy to do it in automated way just by creating a rule that says: exclude members that refer to java.io.Object{Input, Output}Stream.
scala/concurrent/, scala/parallel/, scala/io/I guess it's obvious why those three packages got excluded. I mention them for completeness.
scala/sysI'm not sure if I really understand what's the purpose of this package. Things that got excluded: "scala/sys/BooleanProp.scala","scala/sys/Prop.scala","scala/sys/PropImpl.scala","scala/sys/ShutdownHookThread.scala","scala/sys/SystemProperties.scala", "scala/sys/process/",
I didn't bother to check if properties are abstracted properly if I could easily provide an implementation that does not refer to I/O. This is something I might do later on.
I don't know what's purpose of ShutdownHookThread.scala but it got nuked.
I also excluded almost everything from package.scala file. The only thing that I kept was error method. It's being referenced from many places and is handy to have. Not sure if error should be mixed with other, JVM specific definitions.
scala/refAnything there relies either on reflection or weak/soft references. This package is used in implementation of structural types but I don't plan to support them for the moment so I excluded this package completely from my considerations.
scala/reflect//we cannot exclude the whole reflect package, compiler depends on some classes and we need manifests"scala/reflect/generic/", "scala/reflect/api/","scala/reflect/Print.scala",//refers to reflection"scala/reflect/ScalaBeanInfo.scala","scala/reflect/NameTransformer.scala",
It's rather hard to figure out minimal set of files needed from this package. Manifests are obviously a must. However, if I try to exclude some more of files in scala/reflect I blow up Scala compiler because compiler definitions depend on those files. Would be great to arrange things here so it would be more clear what's an mandatory and what's optional.
scala/runtime//refers to reflection, needed by structural types implementation"scala/runtime/MethodCache.scala",
scala/xmlOh boy, you probably can feel my grumping coming. Let's start with exclusion set of files/packages://depends on org.xml.* stuff, depends on I/O, etc. "scala/xml/package.scala","scala/xml/include/sax/",//everything apart from XhmtlEntities and TokenTests, probably those two should be moved to some other package"scala/xml/parsing/ConstructingHandler.scala", "scala/xml/parsing/ConstructingParser.scala","scala/xml/parsing/DefaultMarkupHandler.scala","scala/xml/parsing/ExternalSources.scala","scala/xml/parsing/FactoryAdapter.scala", "scala/xml/parsing/FatalError.scala","scala/xml/parsing/MarkupHandler.scala","scala/xml/parsing/MarkupParser.scala","scala/xml/parsing/MarkupParserCommon.scala", "scala/xml/parsing/NoBindingFactoryAdapter.scala","scala/xml/parsing/ValidatingMarkupHandler.scala","scala/xml/parsing/XhtmlParser.scala",//depends on I/O "scala/xml/pull/XMLEventReader.scala", "scala/xml/persistent/","scala/xml/factory/",
First of all, XhtmlEntities and TokenTests should probably be moved somewhere else because everything else in xml package depend on them through xml/Utility.scala file. If those files get moved we can easily exclude whole parsing package.
Another problem is that xml/package.scala and xml/XML.scala depend heavily on I/O and org.xml package. Ideally, stuff in top level package should be self-contained so if one wants to use just xml literals in it's program it should be easy to extract necessary parts of xml package to support it.
scala/util//we are removing this because it depends on sys/Prop.scala, so it might be included again once props are being handled "scala/util/control/NoStackTrace.scala","scala/util/parsing/","scala/util/Properties.scala",//depends on I/O (serialization)"scala/util/Marshal.scala", //depends on java.util.regex that GWT doesn't support, bummer"scala/util/matching/",//depends on threads"scala/util/DynamicVariable.scala",//depends on Console, not sure if it makes sense in GWT env "scala/util/logging/"
The biggest confusion here is about util/control package. I'm not sure if I should just go ahead and wipe it completely or not. Given the fact that it has some platform-dependent logic in Exception.scala it's a very tempting route. Specifically, Exception.scala depends on reflection and depends on java.lang.InterruptedException that is not supported by GWT (it simply doesn't exist). Actually, I'd rather see Exception.scala moved to sys or runtime or compat package.
The rest of stuff I had to exclude is less problematic as it's more clear what's is purpose and it's clear to me that I can safely exclude it.
scala/Enumeration.scala, scala/math/BigDecimal.scala Enumeration implementation depends heavily on reflection and I don't see an easy way to fix that so I excluded it. It turns out, that Enumeration is used in exactly one place in library: implementation of BigDecimal. It's used for enumerating all possible rounding modes. Fortunately enough, it was relatively easy to create a patch that removes RoundingMode object and method setScale that takes mode as a parameter. In any case, Enumeration depending heavily on reflection is a problem for me.
scala/Symbol.scalaCurrent implementation of symbols depends on WeakReferences in order to implemented UniqenessCache. Since JavaScript doesn't offer anything corresponding I had to reimplement UniqenessCache using normal HashMap. Not sure if this is going to be problematic, i.e. lead to memory leaks.
Nonetheless, I'd like to see UniqenessCache moved to some platform-specific package so it would be easier to replace it without touching definition of Symbol itself.
scala/Predef.scala, scala/package.scala I had to remove currentThread from package.scala. It's good to see it deprecated already. Can we just remove it in trunk?
The same applies to Predef.exit. Since it's deprecated too, can we remove it in trunk?
All in all, it's good to see scala/Predef and scala/package going into right direction.
SummaryMain problems with compiling library in contrained environment come from dependencies on:

• I/O (including serialization support)
• Threads
• reflection
• platform-specific utilities like Weak/Soft references, specific types of exception, etc.

If could refactor various parts of library to better encapsulate those dependencies in separate packages it would be much easier to use Scala library e.g. in a browser (assuming it's compiled to Javascript). I can see that serialization support added to collections probably must stay as it is. It's not a big problem because it's easy to create automatic rules that capture necessary source2source transformations that would exclude code specific to serialization.
The biggest potential gains I see in scala/reflect, scala/xml and scala/util packages. Also, cleaning up scala/collection/generic package would be nice too.
Thoughts/comments are welcome.
--
Grzegorz Kossakowski