lifting out core compiler structures to make them available via reflection

This is just great.

Thanx for doing this in the first place and for sharing the experience.

On Mon, Feb 1, 2010 at 17:34, martin odersky wrote:
> I did some serious refactoring of the compiler over the weekend. The
> end result is that all core datastructures of the compiler are now
> mirrored in package scala.reflect.generic. Why do this? First, it
> gives us a generic Unpickler that can read Scala symbol informations
> into arbitrary data. Second, it gives us a nice summary what the
> essence of Scala is, unencumbered by concerns of type inference, error
> recovery, or phase structure.
>
> The core of scala.reflect.generic is class Universe, which contains
> via mixins Types, Symbols, Names, Constants, AnnotationInfos, and
> other core types as abstract type members. The compiler's SymbolTable
> class inherits from Universe and instantiates these abstract type
> members to concrete types. Another planned for instantiation is by a
> Scala reflection API. Some core functionality can be shared by both
> instantiations. For instance, the Unpickler can be shared.
>
> Case classes in the compiler are abstracted out as abstract types and
> extractor values. the concrete case class then implements the abstract
> extractor value. For instance:
>
> In scala.reflect.generic.Types
>
>  type ThisType <: Type
>  val ThisType: ThisTypeExtractor
>
>  abstract class ThisTypeExtractor {
>    def apply(sym: Symbol): Type
>    def unapply(tpe: ThisType): Option[Symbol]
>  }
>
> Then in nsc.symtab.Types we implement this by:
>
>  case class ThisType(sym: Symbol) ...
>  object ThisType extends ThisTypeExtractor
>
> The refactoring experience was rather pleasant, and demonstrated,
> that, at least in this case, Scala's type system and abstractions
> provide excellent support.
>
> Overall, I moved/created about 2600 lines of code in package
> reflect.generic and touched about 500 lines of code in the compiler
> proper. I had to fix hundreds of type errors to get things going. Once
> that was done, only 5 run-time errors remained. These were:
>
> 1. One pattern match, where I accidentally matched for the wrong class
> (no type checks for the right pattern matches, unless the base type is
> sealed, so this counts as a gotcha from (a little bit of) dynamic
> typing).
>
> 2. + 3. Two initalization errors where moving things around made some
> things initialize before their dependenices initialize. We know that
> initalization is error-prone, so this is no suprise.
>
> 4. One error involving passing the wrong manifest to an array
> creation. What happened was that we created an array of elements of
> one of these abstract type members. The actual array created was an
> array of Objects, because Object was the erasure of the upper bound of
> the abstract type member. But then in the derived compiler class, we
> expected an array of the actual type that implemented the abstract
> type member. This suggest that we should fix the rules for manifests
> and no longer make abstract type produce the manifest of their upper
> bound. If we had done this, this problem would manifest (no pun
> intended) itself as a type error rather than as a ClassCastException.
>
> 5. One logical error where I thought I could drop some code to handle
> a weird exception because it could never happen. Only of course it did
> happen. Thanks to the regression tests that showed me my error.
>
> Overall that's pretty encouraging for a an effort of this size!
>
> Cheers
>
>  -- Martin
>

On Mon, Feb 01, 2010 at 04:34:31PM +0100, martin odersky wrote:
> I did some serious refactoring of the compiler over the weekend.

This is great news. I am all over it.

A couple questions. Can we make Trees#Traverser abstract? Isn't the
tree traversal logic a property of the shape of the AST?

And: shouldn't TreePrinter just be a Traverser which prints things?

These things were on my mind anyway as you might guess from my recent
CompactTreePrinter commit, but now even more so. I could make these
things happen if you approve of them.

On Mon, Feb 1, 2010 at 5:25 PM, Paul Phillips wrote:
>
> On Mon, Feb 01, 2010 at 04:34:31PM +0100, martin odersky wrote:
>> I did some serious refactoring of the compiler over the weekend.
>
> This is great news.  I am all over it.
>
> A couple questions.  Can we make Trees#Traverser abstract? Isn't the
> tree traversal logic a property of the shape of the AST?
>
> And: shouldn't TreePrinter just be a Traverser which prints things?
>
> These things were on my mind anyway as you might guess from my recent
> CompactTreePrinter commit, but now even more so.  I could make these
> things happen if you approve of them.
>
I think both make sense, yes. Tree#Transformer is more tricky because
of the dependencies added by transformUnit.

(What I did was sort of a first stab at the problem; I tried to be
conservative by lifting only the minimal amount of code.)

Cheers

On Mon, Feb 01, 2010 at 06:18:07PM +0100, martin odersky wrote:
> I think both make sense, yes. Tree#Transformer is more tricky because
> of the dependencies added by transformUnit.

Yeah, I'd have to work myself up to that kind of ambition. But I think
traverser will be straightfowrardish.

> (What I did was sort of a first stab at the problem; I tried to be
> conservative by lifting only the minimal amount of code.)

Still, that is 2800 lines of code I so wanted in the library, so even if
it stops right there I am one happy compiler interfacer.