creating new ClassDefs in compiler plugins?

Hello Sander!

Excuse my ignorance, but would it be possible to achieve your goal using AOP?

Cheers,
Viktor

2009/4/21 Sander Sõnajalg <sander_s@ut.ee>

Hi,

Could anybody amongst the scala compiler developers please shed some light
on the following question:

What i'm trying to do is add a new class in my compiler plugin (and the
plugin is running rather late: for reasons unclear to me i've found it
easiest to run after the phase "cleanup"). So what i'm trying to do is
pattern-match to the parent PackageDef, and use tree-copier to replace it
with my own PackageDef where i've added my new ClassDef into the "stats :
List[Tree]". After some messing around, the actual adding it is not a
problem, but what is really frustrating is that no matter how hard i try,
i cannot find a way to manually create a ClassDef that the latter phases
(GenJVM) would accept: some type infos or prefixes seem to not have been
initialized properly, though i use the "newClass" creator method on
packageDef.symbol and run the typer / positioner on the created classDef
tree. The eventual problem is an assertion error during the "icode" phase,
where the compiler basically stumbles because of some classname prefixes
not being set correctly(?)



  [scalac] java.lang.AssertionError: assertion failed: package class <root>
  [scalac]     at scala.Predef$.assert(Predef.scala:92)
  [scalac]     at
scala.tools.nsc.symtab.Symbols$Symbol.fullNameString(Symbols.scala:1246)
  [scalac]     at
scala.tools.nsc.symtab.Symbols$Symbol.fullNameString(Symbols.scala:1251)
  [scalac]     at
scala.tools.nsc.symtab.Symbols$Symbol.fullNameString(Symbols.scala:1256)
  [scalac]     at
scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:84)


When i compare my plugin-created ClassDef with a ClassDef that's actually
been defined in the compiled scala source-code, i notice the stuff that
*my* ClassDef lacks gets set to the source-code ClassDef during the Namer
phase.. but it doesn't seem to be possible to later run the Namer on my
own ClassDef without heavy hacking.


Finally I tried a different approach: not to try to create my own Symbol
for the new ClassDef, but take some existing ClassDef and clone it's
symbol... and hughh, this even compiled.. but this really feels somewhat
like a hack, and i still strongly feel i'm not really getting what's going
on in there.

When I used the compiler Swing AST-browser to compare the differences
between the cloned and the manually-created class-symbol, the differences
still refer that the issue must lay somewhere in these class prefixes..

[debugging info of the cloned class, which is OK for icode:]

  "Symbol tpe: hello.this.NewClassByCloning
  TypeRef( ThisType(hello),  NewClassByCloning,  [ ])"

[debugging info of the manually-created class, which results in
AssertionError during icode:]

  "Symbol tpe: NewClassNotCloned
  TypeRef( NoPrefix(),  NewClassNotCloned,  [ ])"

But when do these prefixes get set/ how can i work around this thing?


What i mean by "creating a classdef by hand" is something like this:

     ...

     val newNonCloneClassName : Name =
newTypeName("NewClassNotCloned").toTypeName

     val nonCloneClassSymbol = packageDef.symbol.newClass(myPos,
newNonCloneClassName)
     val nonCloneScope = newClassScope(nonCloneClassSymbol)

     val nonCloneClassInfoType = ClassInfoType(List(ObjectClass.tpe,
ScalaObjectClass.tpe),
                       nonCloneScope, nonCloneClassSymbol)
     nonCloneClassSymbol.setInfo(nonCloneClassInfoType)

     val nonCloneTemplate = Template(List(), emptyValDef, List())
     nonCloneTemplate.symbol.owner = nonCloneClassSymbol

     val nonCloneClassTree = ClassDef(nonCloneClassSymbol, nonCloneTemplate)

     val typedNonCloneTree =
       atPos(packageDef.pos) (
         localTyper.typed {
             nonCloneClassTree
          }
       )

     ...


I know it must be possible to add new classes during the later compiler
phases, as you do it as well in the AddInterfaces phase.. though there it
is as well implemented by cloning the symbols of already existing classes.
But what would be the easy/right way to do it..? (i've been messing with
it for ages, now : // )


Thanks a lot if anybody has a moment to think about this issue :)
Sander Sõnajalg

--
Viktor Klang
Senior Systems Analyst

> Hello Sander!
>
> Excuse my ignorance, but would it be possible to achieve your goal using
> AOP?
>
> Cheers,
> Viktor
>

Mm... no, not really, as my goal is to re-implement the way compiler
translates on of the Scala's language features (namely the ApplyDynamic /
calling methods on structural types), and the solution i want to test
involves generating some new traits during the compile-time. The only way
I could think of to work around the need to make the compiler generate the
classes i need would be to insert some special statements to the class
constructors that would invoke an external runtime library that would
generate my required classes with CGLib or ASM.. but this would be a very
hackish solution and would delay some stuff until the runtime that should
by it's nature actually be done during the compile-time.

Sander S.

2009/4/21 Sander Sõnajalg <sander_s@ut.ee>

> Hello Sander!
>
> Excuse my ignorance, but would it be possible to achieve your goal using
> AOP?
>
> Cheers,
> Viktor
>

Mm... no, not really, as my goal is to re-implement the way compiler
translates on of the Scala's language features (namely the ApplyDynamic /
calling methods on structural types), and the solution i want to test
involves generating some new traits during the compile-time. The only way
I could think of to work around the need to make the compiler generate the
classes i need would be to insert some special statements to the class
constructors that would invoke an external runtime library that would
generate my required classes with CGLib or ASM.. but this would be a very
hackish solution and would delay some stuff until the runtime that should
by it's nature actually be done during the compile-time.

Sander,

Ah, thank's for clearing things up for me ;)

What's your prognosed increased classfile size / count for generating traits OTF?
 

Sander S.

--
Viktor Klang
Senior Systems Analyst

>
> Sander,
>
> Ah, thank's for clearing things up for me ;)
>
> What's your prognosed increased classfile size / count for generating
> traits
> OTF?
>
>

Hey,
At the compile-time, I'm hoping to get away with one tiny compile-time
extra trait per one globally unique dynamically called method signature.
At the run-time, ASM will be used to generate some extra stuff as
required, but this overhead should be okay. Anyway.. wouldn't want to do
it all with ASM, as i refuse to believe that there isn't a proper way to
get it to work with just an NSC plugin without too much hacking. :) If i
won't happen to get any helpful tips from the mailing-list, i'll just try
to reverse-engineer this know-how from the AddInterfaces phase's
implementation.

Cheers,
Sander

2009/4/21 Sander Sõnajalg <sander_s@ut.ee>

>
> Sander,
>
> Ah, thank's for clearing things up for me ;)
>
> What's your prognosed increased classfile size / count for generating
> traits
> OTF?
>
>

Hey,
At the compile-time, I'm hoping to get away with one tiny compile-time
extra trait per one globally unique dynamically called method signature.
At the run-time, ASM will be used to generate some extra stuff as
required, but this overhead should be okay. Anyway.. wouldn't want to do
it all with ASM, as i refuse to believe that there isn't a proper way to
get it to work with just an NSC plugin without too much hacking. :) If i
won't happen to get any helpful tips from the mailing-list, i'll just try
to reverse-engineer this know-how from the AddInterfaces phase's
implementation.

Well, I'll be following your progress :)
 

Cheers,
Sander

--
Viktor Klang
Senior Systems Analyst

>

ok.. got it myself after about 6 hours about debugging ://, one just has
to write:
myNewClassSymbol.owner =
definitions.getModule(newTermName("myPackage")).moduleClass
, and then the typer will do the rest

PS.
.. jesus guys, how do you manage to write so intense code with *so* sparse
code commenting/documentation -- don't things really get messy for
yourselves? one can easily find source code files with >1000 LOC and 0
comments inside the compiler code-base :D for an outsider like myself who
doesn't know all the contracts and tricks used in the compiler, it
sometimes just takes ages to debug/understand, even though the API itself
could be well-designed. :)

Cheers,
Sander

On Tuesday April 21 2009, Sander Sõnajalg wrote:
> ...
>
> PS.
> .. jesus guys, how do you manage to write so intense code with *so*
> sparse code commenting/documentation -- don't things really get messy
> for yourselves? one can easily find source code files with >1000 LOC
> and 0 comments inside the compiler code-base :D for an outsider like
> myself who doesn't know all the contracts and tricks used in the
> compiler, it sometimes just takes ages to debug/understand, even
> though the API itself could be well-designed. :)

If it was hard to write, it should be hard to read.

> Cheers,
> Sander

RRS

On Wed, Apr 22, 2009 at 2:57 AM, Randall R Schulz wrote:
> On Tuesday April 21 2009, Sander Sõnajalg wrote:
>> ...
>>
>> PS.
>> .. jesus guys, how do you manage to write so intense code with *so*
>> sparse code commenting/documentation -- don't things really get messy
>> for yourselves? one can easily find source code files with >1000 LOC
>> and 0 comments inside the compiler code-base :D for an outsider like
>> myself who doesn't know all the contracts and tricks used in the
>> compiler, it sometimes just takes ages to debug/understand, even
>> though the API itself could be well-designed. :)
>
> If it was hard to write, it should be hard to read.
>

Care to expand on this? Whilst I understand that some code can be
complicated I fail to see how this stops you documenting it or writing
it so it can be maintained. The lack of help provided with compiler
plugins indicates that very few people understand them at all,
providing comments would also help EPFLers to pick it up.

It's a matter of time, really. I am probably the only one who can
comment on most of the compiler internals, and I try to do that when I
touch some functionality. But I am so overloaded that I simply do not
find the time. There are always so many more urgent things to do. To
alleviate this problem we have started to do some code walkthroughs on
video, which are available on our site.

Cheers

On Tuesday April 21 2009, Chris Twiner wrote:
> On Wed, Apr 22, 2009 at 2:57 AM, Randall R Schulz
wrote:
> > On Tuesday April 21 2009, Sander Sõnajalg wrote:
> >> ...
> >>
> >> PS.
> >> .. jesus guys, how do you manage to write so intense code with
> >> *so* sparse code commenting/documentation ...
> >
> > If it was hard to write, it should be hard to read.
>
> Care to expand on this? ...

It's a joke. (And I don't do smilies.)

I was told the FP community had a better-than-average sense of humor...

Randall Schulz

On Wed, Apr 22, 2009 at 3:23 PM, Randall R Schulz wrote:
> On Tuesday April 21 2009, Chris Twiner wrote:
>> On Wed, Apr 22, 2009 at 2:57 AM, Randall R Schulz
> wrote:
>> > On Tuesday April 21 2009, Sander Sõnajalg wrote:
>> >> ...
>> >>
>> >> PS.
>> >> .. jesus guys, how do you manage to write so intense code with
>> >> *so* sparse code commenting/documentation ...
>> >
>> > If it was hard to write, it should be hard to read.
>>
>> Care to expand on this? ...
>
> It's a joke. (And I don't do smilies.)
>
> I was told the FP community had a better-than-average sense of humor...
>

At least Viktor has the common decency to make funny jokes.

On Wed, Apr 22, 2009 at 03:42:59PM +0200, Chris Twiner wrote:
> At least Viktor has the common decency to make funny jokes.

I have possibly more reason than anyone not to have a sense of humor
about it (having spent at least several hundred hours over the last year
reading scalac sources) and I thought it was funny.

> To
> alleviate this problem we have started to do some code walkthroughs on
> video, which are available on our site.
>

Yes! these video lectures are indeed of great help, please keep publishing
them on scala-lang.org when you give any :) If there was a similar
lecture on Trees as there already are for Symbols and Types, i think the
most crucial parts for a new-comer to understand what's happening inside
the transformations would be covered, at least to some extent:)

With best wishes,
Sander S.

You can find examples of classes that are created without cloning in
the specialized branch. The specialization phase creates plenty of new
classes :)

Normally one shouldn't set owners, the factory methods on symbols
should take care of that. The pseudo code you showed earlier is
suspicious, since you set the owner of the template's symbol, without
first setting any symbol on that Template. You could try creating
untyped trees (without any symbols) and let the 'typed' method create
them for you (it should work fine for top level classes).

iulian
PS. There's a scala-internals mailing list that's a better fit for
this kind of questions.

2009/4/22 Sander Sõnajalg :
>>
>
> ok.. got it myself after about 6 hours about debugging ://, one just has
> to write:
> myNewClassSymbol.owner =
> definitions.getModule(newTermName("myPackage")).moduleClass
> , and then the typer will do the rest
>
>
> PS.
> .. jesus guys, how do you manage to write so intense code with *so* sparse
> code commenting/documentation -- don't things really get messy for
> yourselves? one can easily find source code files with >1000 LOC and 0
> comments inside the compiler code-base :D for an outsider like myself who
> doesn't know all the contracts and tricks used in the compiler, it
> sometimes just takes ages to debug/understand, even though the API itself
> could be well-designed. :)
>
> Cheers,
> Sander
>
>
>

> You can find examples of classes that are created without cloning in
> the specialized branch. The specialization phase creates plenty of new
> classes :)
>
> Normally one shouldn't set owners, the factory methods on symbols
> should take care of that. The pseudo code you showed earlier is
> suspicious, since you set the owner of the template's symbol, without
> first setting any symbol on that Template. You could try creating
> untyped trees (without any symbols) and let the 'typed' method create
> them for you (it should work fine for top level classes).
>
> iulian
> PS. There's a scala-internals mailing list that's a better fit for
> this kind of questions.
>

Okay, thanks a lot, that's actually the first "real" answer to my question
i have received :)
About this "suspicious" pseudo-code.. That's actually the main problem:
though i eventually usually manage to hack my things to work somehow, i'm
permanently having that "suspicious" feeling that my code might not make a
very conventional use of the compiler inner APIs and how they should
"normally" be used. But with the current situation with the docs, i don't
have much other chance to check on this except for comparing with the code
in the existing transformations.

Thanks!
Sander

Sander Sõnajalg wrote:
>> You can find examples of classes that are created without cloning in
>> the specialized branch. The specialization phase creates plenty of new
>> classes :)
>>
>> Normally one shouldn't set owners, the factory methods on symbols
>> should take care of that. The pseudo code you showed earlier is
>> suspicious, since you set the owner of the template's symbol, without
>> first setting any symbol on that Template. You could try creating
>> untyped trees (without any symbols) and let the 'typed' method create
>> them for you (it should work fine for top level classes).
>>
>> iulian
>> PS. There's a scala-internals mailing list that's a better fit for
>> this kind of questions.
>>
>>
>
> Okay, thanks a lot, that's actually the first "real" answer to my question
> i have received :)
> About this "suspicious" pseudo-code.. That's actually the main problem:
>
The 'suspicious' part was the missing NullPointerException I'd expect,
had you set the owner of a symbol of a tree that was just created. New
trees have null symbols (and types) before you run 'typed' on them.

> though i eventually usually manage to hack my things to work somehow, i'm
> permanently having that "suspicious" feeling that my code might not make a
> very conventional use of the compiler inner APIs and how they should
> "normally" be used. But with the current situation with the docs, i don't
> have much other chance to check on this except for comparing with the code
> in the existing transformations.
>
Yes.. and the code walk-throughs:

http://www.scala-lang.org/node/598

iulian

On Wed, Apr 22, 2009 at 03:47:29AM +0300, Sander Sõnajalg wrote:
> .. jesus guys, how do you manage to write so intense code with *so*
> sparse code commenting/documentation -- don't things really get messy
> for yourselves? one can easily find source code files with >1000 LOC
> and 0 comments inside the compiler code-base :D

After you have spent a certain amount of time in the compiler, you will
come to feel angry resentment at every comment you encounter, because it
might be trying to mislead you! I have modified my editor not to show me
any comments in scala sources so they cannot tempt me with their siren
songs of reasons and explanations. AH THE LIMITLESS SERENITY

I find your technique generally useful, not only in the compiler.

2009/4/23 Paul Phillips <paulp@improving.org>

On Wed, Apr 22, 2009 at 03:47:29AM +0300, Sander Sõnajalg wrote:
> .. jesus guys, how do you manage to write so intense code with *so*
> sparse code commenting/documentation -- don't things really get messy
> for yourselves? one can easily find source code files with >1000 LOC
> and 0 comments inside the compiler code-base :D

After you have spent a certain amount of time in the compiler, you will
come to feel angry resentment at every comment you encounter, because it
might be trying to mislead you! I have modified my editor not to show me
any comments in scala sources so they cannot tempt me with their siren
songs of reasons and explanations.  AH THE LIMITLESS SERENITY

--
Paul Phillips      | Beware of bugs in the above code; I have only
Apatheist          | proved it correct, not tried it.
Empiricist         |     -- Knuth
up hill, pi pals!  |----------* http://www.improving.org/paulp/ *----------