Re: Parsing and semantically analyzing scala itself

On Wed, Dec 17, 2008 at 11:55 PM, Iulian Dragos <jaguarul@gmail.com> wrote:
> On Tue, Dec 16, 2008 at 9:18 AM, Tharaka Abeywickrama
>
<tharakawick@gmail.com> wrote:
>> Hi,
>> I have two questions:
>> 1) How do you parse scala itself, including semantic analysis that does type
>> resolution etc. My goal is to build a static analysis tool for scala that
>> does thorough inter-procedural analysis. Do you have anything comparable for
>> scala that matches the Java AST/DOM provided by the eclipse platform?
>
> The scala compiler uses an abstract syntax tree internally. You can
> write a compiler plugin in order to get access to it. There is a guide
> about that on the link sent by Alex, but as David has noticed, it is
> no easy feat.

seconded on the no easy feat.

> Depending on the kind of analysis you want to do, you may have a look
> at the package scala.tools.nsc.backend.analysis. It works on a control
> flow graph intermediate representation.
>
>> 2) How do you do annotation processing in scala? is there a tool that
>> mirrors apt and gives you a DOM that represents the code?
>
> You can write compiler plugins for that.
>

I would warn that whilst you can write compiler plugins for it, there
is very little you can do with the information. You cannot provide
any form of transformation based on annotations.

I unfortunately wasted a week on trying to alter the code tree from
annotation information. It has also been mentioned that the 2.8.0
code tree will probably place even further restrictions upon the
developer.

If its really a build time concern it may be an option use code gen
tools instead, such as jostraca or even xslt / tapestry like a pre
compiler.

Your best bet for runtime annotation processing (if you don't want
code transformations) is to use a static annotation and process from
java reflection as you would normally do.

cheers,
Chris

You can add a phase to the compiler, thus adding extra checks or extra tree rewrites that apply after type checking has finished.

On Thu, Dec 18, 2008 at 2:33 AM, Chris Twiner wrote:
>>> 2) How do you do annotation processing in scala? is there a tool that
>>> mirrors apt and gives you a DOM that represents the code?
>>
>> You can write compiler plugins for that.
>>
>
> I would warn that whilst you can write compiler plugins for it, there
> is very little you can do with the information. You cannot provide
> any form of transformation based on annotations.
>
> I unfortunately wasted a week on trying to alter the code tree from
> annotation information. It has also been mentioned that the 2.8.0
> code tree will probably place even further restrictions upon the
> developer.

I'm not sure what you mean by 'from annotation information', but I am
positive you can change ASTs from a compiler plugin. A plugin packages
one phase that has equal rights to phases in the compiler itself. That
means it may be a transformer, and return new trees in the overridden
'transform' method. I'd be happy to help if you give us more specific
problems you encountered. I know documentation is lacking, so now's
the right time to advertise the new scala-internals list, where you
might find people who can help.

Cheers,
iulian

scala-internals? cool! I subscribed just now :-)

On Thu, Dec 18, 2008 at 7:33 AM, Daniel Kröni wrote:
> i found the following statement on http://www.scala-lang.org/node/140
>
>> You can add a phase to the compiler, thus adding extra checks or extra
>> tree rewrites that apply after type checking has finished.
>
> aren't tree rewrites transformations on the ast? this is exactly what i
> want to give a try: modifying the ast based on annotations. this should be
> possible.
>

Changing the tree was doable after the typers stage but changes based
on inserting into companion objects would never be visible. The big
problem is that to add stuff into the companion object you need info
before the typers stage. My aim was to basically add functions into
the companion objects based on annotations.

http://www.nabble.com/-scala-tools--compiler-plugin---phase-issues-with-...

shows some of the back story for my attempts.

On Thu, Dec 18, 2008 at 9:51 AM, Iulian Dragos wrote:
> On Thu, Dec 18, 2008 at 2:33 AM, Chris Twiner wrote:
>>>> 2) How do you do annotation processing in scala? is there a tool that
>>>> mirrors apt and gives you a DOM that represents the code?
>>>
>>> You can write compiler plugins for that.
>>>
>>
>> I would warn that whilst you can write compiler plugins for it, there
>> is very little you can do with the information. You cannot provide
>> any form of transformation based on annotations.
>>
>> I unfortunately wasted a week on trying to alter the code tree from
>> annotation information. It has also been mentioned that the 2.8.0
>> code tree will probably place even further restrictions upon the
>> developer.
>
> I'm not sure what you mean by 'from annotation information', but I am
> positive you can change ASTs from a compiler plugin. A plugin packages
> one phase that has equal rights to phases in the compiler itself. That
> means it may be a transformer, and return new trees in the overridden
> 'transform' method. I'd be happy to help if you give us more specific
> problems you encountered. I know documentation is lacking, so now's
> the right time to advertise the new scala-internals list, where you
> might find people who can help.
>
Hi Iulian,

the http://www.nabble.com/-scala-tools--compiler-plugin---phase-issues-with-...
link should provide enough information as to what is going on with
this problem set.

A number of people from epfl tried to help out with this, but I got no
firm solutions, only shock at the differences between tree
transformation and the symbolic tables between namers and typers
phase.

In short you can only transform a tree that exists, annotations exist
in this tree and are readily available *after* the typers phase (full
of good information and symbol entries for example) but you can't
insert methods to be generated before hand into the synthetic methods
of a module (as the case class handling does) as this takes place in
the namers phase. The 2.8.0 will merge the namers and typers phase
anyway rendering that option mute.

The idea is at http://code.google.com/p/scala-scales/wiki/VirtualConstructorPreSIP,
while it was pointed out that its not really SIP material, it should
highlight what was being attempted. Given the typers information (and
largely just namers is enough) I could generate the functions in the
companion object (tree print would work etc). I'm happy to send a zip
file of the projects used.

Don't get me wrong if there is some simple trick that was being missed
I'm happy to try it out again, but I've quite hapily moved my codebase
on with adding the adts by hand. I'd rather not play with the plugins
again until 2.8.0 shows up with the phase SIP in place, and who knows
the virtual classes may solve the underlying problem anyway (although
nobody seems to say what exactly they are).

Either way its an absolute truth to say the compiler internals are not
for the feint of heart, and that people should search for other
simpler methods to solve their problems first.

cheers,
Chris