Where to start with understanding of writing compiler plugins?

Start here: http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

On 25 May 2011 19:00, Ken McDonald <ykkenmcd@gmail.com> wrote:

It's become obvious to me that adding the features I'd like to see in Scaladoc is going to require more than just hacking the basic Scaladoc source. So I next have to figure out if I can do anything worthwhile using compiler plugins. I know people have asked before as to the best place to start with understanding writing compiler plugins, but there is so little formal documentation out there on this, and so much is being done, I thought it was worth asking again.
For what it's worth, I don't need to know about the real hard-core stuff; I need to access to when the scaladoc comments are being processed, and to whatever internal compiler structures let me know what inherits from what, defines what, is an inner class of what, and so forth. And, oh yeah, I'd _love_ it if there were an easy way of getting the code (not the AST or anything, just the raw code) that makes of the body of a method.
I'm not promising anything will come of this--these Scaladoc projects are fairly low priority for me at the moment. But who knows...?

Thanks,Ken

--
Kevin Wright

gtalk / msn : kev.lee.wright@gmail.comkev.lee.wright@gmail.commail: kevin.wright@scalatechnology.com
vibe / skype: kev.lee.wrightquora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

2011/5/25 Ken McDonald :
> It's become obvious to me that adding the features I'd like to see in
> Scaladoc is going to require more than just hacking the basic Scaladoc
> source. So I next have to figure out if I can do anything worthwhile using
> compiler plugins. I know people have asked before as to the best place to
> start with understanding writing compiler plugins, but there is so little
> formal documentation out there on this, and so much is being done, I thought
> it was worth asking again.

Ken,

It'd be helpful if you told us how much you know about compilers
construction in general. If not that much I suggest you to read a few
introductory chapters of some book so you get familiar with basic
terminology. This will help you with place's like excellent Miguel's
compiler corner mentioned by Kevin.

Ken,

It'd be helpful if you told us how much you know about compilers
construction in general. If not that much I suggest you to read a few
introductory chapters of some book so you get familiar with basic
terminology. This will help you with place's like excellent Miguel's
compiler corner mentioned by Kevin.

--
Grzegorz Kossakowski

Not nearly as much as I should. I know a moderate amount about parsing, a few bits and pieces like peephole optimization, am pretty familiar with garbage collection, and know effectively nothing about the very sophisticated techniques that I imagine are being used right now. 

Since I don't actually need to do anything with code (externally, internally, or JVM), I'm hoping this won't be too much of a hindrance. What I really need is access to data structures that describe the syntactic structure of the program and some, but not a lot, of the semantic structure. For example, I want to draw inheritance diagrams, and for that I need the syntactic info that "class A extends B with C", but not the more semantically meaningful linear supertypes chain--it doesn't make for a very pretty or useful inheritance diagram :-). (Well, actually, I do need the linear supertypes also because that should be part of the inheritance diagram, but it's arguably the less important part.)
However, do feel free to recommend reading. I love learning about this kind of stuff, it's just a matter of time constraints.
Thanks,Ken

2011/5/25 Ken McDonald :
> Not nearly as much as I should. I know a moderate amount about parsing, a
> few bits and pieces like peephole optimization, am pretty familiar with
> garbage collection, and know effectively nothing about the very
> sophisticated techniques that I imagine are being used right now.
>
> Since I don't actually need to do anything with code (externally,
> internally, or JVM), I'm hoping this won't be too much of a hindrance. What
> I really need is access to data structures that describe the syntactic
> structure of the program and some, but not a lot, of the semantic structure.
> For example, I want to draw inheritance diagrams, and for that I need the
> syntactic info that "class A extends B with C", but not the more
> semantically meaningful linear supertypes chain--it doesn't make for a very
> pretty or useful inheritance diagram :-). (Well, actually, I do need the
> linear supertypes also because that should be part of the inheritance
> diagram, but it's arguably the less important part.)
> However, do feel free to recommend reading. I love learning about this kind
> of stuff, it's just a matter of time constraints.

Ken,

I learned a bit about internals of nsc from Lex last summer while
hacking scala+gwt. I started with virtually no compiler knowledge and
found that it's crucial that at the beginning you understand
difference between trees, types and symbols. Confusing them won't lead
you anywhere.

Most pieces of information you need will be extracted from Symbols.
There's a lot of functionality in nsc that will help to answer your
queries.

When it comes to reading, I skimmed through Dragon book[1] and it was
enough to get me going. It's quite old book but basics do not change
that much over time.

[1] http://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_and_Tools

Another crucial nuance is `info` versus `tpe` (type).
The info is a symbol corresponds roughly to the signature that you write down in its definition or that we load from a classfile.The type of a symbol is the info of the symbol, except for types, where the type refers to that symbol rather than its signature.
type Foo = String
`Foo`.info (the info of the symbol that represents the type alias Foo) is TypeRef(scalaPackageTpe, `String`, List())whereas `Foo`.tpe is a type reference to the type alias: TypeRef(`Foo`.owner.thisType, `Foo`, List())  (`Foo`.owner.thisType is the type you use to refer to `this` in Foo's owner)

It's important to remember that types depend on context, and this is where asSeenFrom, and it's friendlier incarnations, such as memberType and memberInfo come in.
suppose we introduce type parameters in the above example
class Bar[T] {  type Foo = T}
class BarString extends Bar[String]
now, when you're in BarString, if you just ask for the info for Foo, you'll still get Tto compute the expected `String`, call ThisType(BarString).memberInfo(`Foo`)

The morale of this story: think twice whether you call tpe or info on a symbol, and then think again whether you should transform it using the asSeenFrom family.
cheersadriaan
ps: the story complicates further with type constructors, where you need to consider whether to use tpe or typeConstructor

Better start here: http://www.scala-lang.org/node/598 Watch the
videos, that should clarify the tree/type/symbol triplet.

good luck.
iulian

On Wed, May 25, 2011 at 8:00 PM, Ken McDonald wrote:
> It's become obvious to me that adding the features I'd like to see in
> Scaladoc is going to require more than just hacking the basic Scaladoc
> source. So I next have to figure out if I can do anything worthwhile using
> compiler plugins. I know people have asked before as to the best place to
> start with understanding writing compiler plugins, but there is so little
> formal documentation out there on this, and so much is being done, I thought
> it was worth asking again.
> For what it's worth, I don't need to know about the real hard-core stuff; I
> need to access to when the scaladoc comments are being processed, and to
> whatever internal compiler structures let me know what inherits from what,
> defines what, is an inner class of what, and so forth. And, oh yeah, I'd
> _love_ it if there were an easy way of getting the code (not the AST or
> anything, just the raw code) that makes of the body of a method.
> I'm not promising anything will come of this--these Scaladoc projects are
> fairly low priority for me at the moment. But who knows...?
>
> Thanks,
> Ken

Those vids don't seem to want to play for me, unfortunately.
Ken