intercepting compiler classpath lookups

[+iulian]
Hey Iulian, do you know if there is an easy way to get the dependencies of some piece of scala source code from the compiler?
Thanks,Burak

On Mon, Nov 28, 2011 at 5:49 PM, John Belmonte <john@neggie.net> wrote:

I'm looking for a way to intercept classpath lookups during compile.
This is for the purpose of letting IDE's work easily with a
proprietary build system and a manageable subset of a large codebase.
The approach is to use the compiler to yield all jar dependencies of a
set of source files and then prepare a skeleton jar having its
classpath set to the discovered dependencies.  This last jar is in
turn used by the IDE project, providing an indirection into the large
set of dependent jars not of interest to the developer.

For the Java/Eclipse case the JDT API supports intercepting classpath
lookups by way of the FileSystem.Classpath interface [1].  I haven't
found something similar in the Scala compiler.  I have seen the
analysis compile functionality, but this only allows collecting
dependency information and not intercepting lookups.  (In my case I
need class lookups to fall back to the output of the proprietary
build.)

I'm wondering if there's a way to do this intercepting which I've
overlooked.  Assuming there isn't, would it be considered a reasonable
compiler enhancement?


[1] http://javasourcecode.org/html/open-source/eclipse/eclipse-3.5.2/org/eclipse/jdt/internal/compiler/batch/FileSystem.Classpath.html

--
Burak Emir
--
http://burak.emir.googlepages.com

I'll try to put this in shorter terms:
When we compile some source with dependencies A,B,C, there might be two jars F that contains A,B, and G, which contains C.
If I understand correctly, John wants an easy way to figure out the jars F and G on which the source code depends.
-- Burak

On Wed, Nov 30, 2011 at 7:48 PM, Burak Emir <burak.emir@gmail.com> wrote:

[+iulian]
Hey Iulian, do you know if there is an easy way to get the dependencies of some piece of scala source code from the compiler?
Thanks,Burak

On Mon, Nov 28, 2011 at 5:49 PM, John Belmonte <john@neggie.net> wrote:

I'm looking for a way to intercept classpath lookups during compile.
This is for the purpose of letting IDE's work easily with a
proprietary build system and a manageable subset of a large codebase.
The approach is to use the compiler to yield all jar dependencies of a
set of source files and then prepare a skeleton jar having its
classpath set to the discovered dependencies.  This last jar is in
turn used by the IDE project, providing an indirection into the large
set of dependent jars not of interest to the developer.

For the Java/Eclipse case the JDT API supports intercepting classpath
lookups by way of the FileSystem.Classpath interface [1].  I haven't
found something similar in the Scala compiler.  I have seen the
analysis compile functionality, but this only allows collecting
dependency information and not intercepting lookups.  (In my case I
need class lookups to fall back to the output of the proprietary
build.)

I'm wondering if there's a way to do this intercepting which I've
overlooked.  Assuming there isn't, would it be considered a reasonable
compiler enhancement?


[1] http://javasourcecode.org/html/open-source/eclipse/eclipse-3.5.2/org/eclipse/jdt/internal/compiler/batch/FileSystem.Classpath.html

--
Burak Emir
--
http://burak.emir.googlepages.com

--
Burak Emir
--
http://burak.emir.googlepages.com

On Wed, Nov 30, 2011 at 7:58 PM, Burak Emir <burak.emir@gmail.com> wrote:

I'll try to put this in shorter terms:
When we compile some source with dependencies A,B,C, there might be two jars F that contains A,B, and G, which contains C.
If I understand correctly, John wants an easy way to figure out the jars F and G on which the source code depends.

I think having something like this makes sense.  Not sure it exists already, but Paul would know.
Cheers
 -- Martin

SBT is doing this, IIRC in it's dependency analysis.  WE really should pull that stuff into the compiler so it's available for use from other build tools.  I know I could use it inside Maven.
- Josh

On Wed, Nov 30, 2011 at 2:04 PM, martin odersky <martin.odersky@epfl.ch> wrote:

On Wed, Nov 30, 2011 at 7:58 PM, Burak Emir <burak.emir@gmail.com> wrote:

I'll try to put this in shorter terms:
When we compile some source with dependencies A,B,C, there might be two jars F that contains A,B, and G, which contains C.
If I understand correctly, John wants an easy way to figure out the jars F and G on which the source code depends.

I think having something like this makes sense.  Not sure it exists already, but Paul would know.
Cheers
 -- Martin

On Wed, Nov 30, 2011 at 8:22 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

SBT is doing this, IIRC in it's dependency analysis.  WE really should pull that stuff into the compiler so it's available for use from other build tools.  I know I could use it inside Maven.

AFAIK it's not actually intercepting, but inserting a special phase that looks at all references that come from an AST.
Cheers
 -- Martin 

- Josh

On Wed, Nov 30, 2011 at 2:04 PM, martin odersky <martin.odersky@epfl.ch> wrote:

On Wed, Nov 30, 2011 at 7:58 PM, Burak Emir <burak.emir@gmail.com> wrote:

I'll try to put this in shorter terms:
When we compile some source with dependencies A,B,C, there might be two jars F that contains A,B, and G, which contains C.
If I understand correctly, John wants an easy way to figure out the jars F and G on which the source code depends.

I think having something like this makes sense.  Not sure it exists already, but Paul would know.
Cheers
 -- Martin

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

My question is does he want to *intercept* the class load, or is it ok if we just dump out the references from the AST after the fact.  If this is blaze-related, the difference is important.
- Josh

On Wed, Nov 30, 2011 at 3:10 PM, martin odersky <martin.odersky@epfl.ch> wrote:

On Wed, Nov 30, 2011 at 8:22 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

SBT is doing this, IIRC in it's dependency analysis.  WE really should pull that stuff into the compiler so it's available for use from other build tools.  I know I could use it inside Maven.

AFAIK it's not actually intercepting, but inserting a special phase that looks at all references that come from an AST.
Cheers
 -- Martin  

- Josh

On Wed, Nov 30, 2011 at 2:04 PM, martin odersky <martin.odersky@epfl.ch> wrote:

On Wed, Nov 30, 2011 at 7:58 PM, Burak Emir <burak.emir@gmail.com> wrote:

I'll try to put this in shorter terms:
When we compile some source with dependencies A,B,C, there might be two jars F that contains A,B, and G, which contains C.
If I understand correctly, John wants an easy way to figure out the jars F and G on which the source code depends.

I think having something like this makes sense.  Not sure it exists already, but Paul would know.
Cheers
 -- Martin

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

Right. I totally get what you're doing, was just curious if you had the network filesystem issue.  I'll let Paul comment based on his recent classloader work.  
If I understand thing well enough, I think you may be able to create a subclass of the compiler (similar to how SBT works) and yoink the classloader used in your subclass.   Paul recently spent a lot of time unifying classpaths vs. classloaders, and I'm hoping he'll have a good answer for you :)
- Josh

On Wed, Nov 30, 2011 at 10:10 PM, John Belmonte <john@neggie.net> wrote:

> Hi-- as I tried to describe in original post I need to intercept the
> class lookups and have them fall back to the output the proprietary
> build.

I was screwing around with something like this for scala-gwt (deferring
to the GWT environment to load classes); I didn't come up with an
on-demand way, but instead was creating a custom InMemoryPath up-front:

https://github.com/stephenh/scalagwt-scala/blob/embed/src/compiler/scala...

The line:

override lazy val classPath = {
new JavaClassPath(List(input).toIndexedSeq,ClassPath.DefaultJavaContext)
}

Is where it tells the scala compiler to use the "input" InMemoryPath
instance to load all classes from. (Forgive the code being hackish; it
was a spike and may be doing many things that aren't needed.)

Obviously in your from-the-network case, you don't want to have
everything in-memory, but AFAIU you could write a custom implementation
of ClassPath that loaded bytes only as needed, and then put it last in
the classPath list (vs. having it just be the sole entry like I did).

...it's been awhile since I was in that code, so it may/may not work,
but it makes sense in my head.

More generally, I'm pretty sure I understand what you're asking for,
and similarly would have liked a more convenient way of just
implementing something like:

findSomeClass(binaryName: String): Array[Byte] = {
// environment specific lookup logic goes here
}

And having the compiler call that for class names it cannot otherwise
resolve, without worrying about implementing a ClassPath and maintaining
the hierarchy of packages/files. (Which isn't terribly hard; it should
be easy to write a ClassPath that did that generically, and had an
abstract findSomeClass method like above that you could then just
implement that.)

Anyway, that's my experience from hacking around a few months ago. Good
luck!

- Stephen

On Wed, Nov 30, 2011 at 7:51 PM, Stephen Haberman
wrote:
>  findSomeClass(binaryName: String): Array[Byte] = {
>     // environment specific lookup logic goes here
>   }
>
> And having the compiler call that for class names it cannot otherwise
> resolve, without worrying about implementing a ClassPath and maintaining
> the hierarchy of packages/files.

That's precisely where we're going.

This approach should work for me-- thank you for detailing.

I'm looking forward to the new API mentioned by Paul when that's available too.

On Wed, Nov 30, 2011 at 10:51 PM, Stephen Haberman
wrote:
> I was screwing around with something like this for scala-gwt (deferring
> to the GWT environment to load classes); I didn't come up with an
> on-demand way, but instead was creating a custom InMemoryPath up-front:
>
> https://github.com/stephenh/scalagwt-scala/blob/embed/src/compiler/scala...