Profiling the compiler

On Tue, Dec 27, 2011 at 11:51 AM, Grzegorz Kossakowski <grzegorz.kossakowski@gmail.com> wrote:

I'd like to do some profiling of a compiler. I'm mostly interested in inspecting object allocations.

What, I might get a profiling buddy!? That would be great! Wait'll you see some of the stuff I see!  

Ideally, I'd like to take heap snapshots at several times during single compiler run and inspect them by querying single snapshot and probably comparing results of the same query ran against different snapshots.

If you have yourkit installed, and "ant yourkit.build" which sticks a little extra library in lib/extra, then you can use -Yprofile:<phases> and/or -Yprofile-memory to get some yourkit integration.  It's not very sophisticated or polished.  I like -Yprofile-memory because it takes the heap snapshot before leaving the compiler, which means it's about the same heap snapshot you would get by taking one on shutdown, except everything is strongly reachable instead of nothing being strongly reachable.

Example of query I'd like to execute is to produce distribution of sizes of lists allocated by the compiler. I'd love to define subset of heap I'm interested in by e.g. looking into objects allocated only in typers. Is there any profiler that allows me to do that kind of analysis fairly easily?

You can do that with yourkit no problem.  If you use -Yprofile:typer it will turn the profiling on at typer and turn it off when it's over.  If you use -Yprofile:all you can profile all phases and it advances the object generation at each phase, so you can see exactly which objects are allocated when.  You can also use the yourkit gui to turn on and off profiling interactively.  There are also endless triggers and filters and such, and at some point I started writing custom yourkit probes so I could profile in a very targetted way, because indiscriminate recording takes forever.

Paul, I remember you did some memory profiling and you shared results on Strings that are being allocated. How did you obtain those results?

Most things I do with yourkit, but that particular thing I did by taking a heap dump and analyzing it with jhat, I think because yourkit was too slow and jhat is more specialized. I can also recommend the eclipse memory analyzer for heap analysis.  But for most things, yourkit is the most appealing.
Attached are the largest memory consumers from a run where I compiled src/compiler/scala/reflect, and the largest direct creators of garbage from another run compiling the same thing.