Classloader for macros

On Wed, Dec 21, 2011 at 4:07 PM, Eugene Burmako <xeno.by@gmail.com> wrote:

However, classpath matters have proven to be very subtle, so I'd like
to ask for your feedback. Do you also prefer option #1?

Yes, please. It's great that Scala is just a normal library when it comes to deployment and we should do our best to keep it that way, in my opinion.
Best,Ismael

On Wed, Dec 21, 2011 at 8:07 AM, Eugene Burmako <xeno.by@gmail.com> wrote:

However, classpath matters have proven to be very subtle, so I'd like
to ask for your feedback. Do you also prefer option #1? What would you
like to add to the discussion?

You can do #1, and if sufficient need is demonstrated for a separate classpath, #3 can be made available without interfering with #1.  If a -macrocp was given, put it in front of the regular classpath when compiling macros, otherwise compile as usual.

On Wed, Dec 21, 2011 at 5:35 PM, Paul Phillips wrote:
>
>
> On Wed, Dec 21, 2011 at 8:07 AM, Eugene Burmako wrote:
>>
>> However, classpath matters have proven to be very subtle, so I'd like
>> to ask for your feedback. Do you also prefer option #1? What would you
>> like to add to the discussion?
>
>
> You can do #1, and if sufficient need is demonstrated for a separate
> classpath, #3 can be made available without interfering with #1.  If a
> -macrocp was given, put it in front of the regular classpath when compiling
> macros, otherwise compile as usual.
>
I also think we should do #1. This brings up the question, what's the
best way to do this? Is there an easy way to obtain a Java ClassLoader
from a Scala ClassPath?

Cheers

On Wed, Dec 21, 2011 at 9:54 AM, martin odersky <martin.odersky@epfl.ch> wrote:

I also think we should do #1. This brings up the question, what's the
best way to do this? Is there an easy way to obtain a Java ClassLoader
from a Scala ClassPath?

Well, I'm glad you asked.
So what I'd like to do, and all I really need is a little rubber stamping, is re-model everything which presently operates on classpaths to operate on a different abstraction, something with a new name to evade conceptual conflation, like "ClassfileProvider." It'd at its most basic look something like
  trait ClassfileProvider {    def classBytes(className: String): Array[Byte]  }
This would be augmented by more capable variations such as:
  trait ClassfileLoader extends ClassfileProvider {    def classLoader: ScalaClassLoader    def classFile(className: String): Class[_]  }
  trait EnumerableClassfileProvider extends ClassfileProvider {     def allClassNames(): Iterable[String]    def allClassFiles(): Iterable[Class[_]]  }
etc.  And then standard implementations like
  class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider { ... }   class MavenClassfileLoader(group: String, id: String, version: String) extends EnumerableClassfileProvider { ... }  // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
And mechanisms for composition etc.  I want to be able to invoke scala like this (although it might have to be something other than '-cp')

  scala -cp asm%asm%3.2:org.scalaz%scalaz-core%7.0 my.fancy.Program
...and have the standard way of translating "classprovider expressions" into "classproviders" resolve those through the ivy cache.  The existing classpath is just one specific case of that (where all "expressions" are filesystem paths, and the provider is always a "filesystem provider".)
Lots of good stuff emerges from this approach.  I can provide a stub implementation in fairly short order, and I also have a large classpath rewrite which is I think an undeniable improvement but which I've had lying around for months because it mucks with the interface and I assume that will break the IDE.  Let me know.

On Wed, Dec 21, 2011 at 7:23 PM, Paul Phillips wrote:
>
>
> On Wed, Dec 21, 2011 at 9:54 AM, martin odersky
> wrote:
>>
>> I also think we should do #1. This brings up the question, what's the
>> best way to do this? Is there an easy way to obtain a Java ClassLoader
>> from a Scala ClassPath?
>
>
> Well, I'm glad you asked.
>
> So what I'd like to do, and all I really need is a little rubber stamping,
> is re-model everything which presently operates on classpaths to operate on
> a different abstraction, something with a new name to evade conceptual
> conflation, like "ClassfileProvider." It'd at its most basic look something
> like
>
>   trait ClassfileProvider {
>     def classBytes(className: String): Array[Byte]
>   }
>
> This would be augmented by more capable variations such as:
>
>   trait ClassfileLoader extends ClassfileProvider {
>     def classLoader: ScalaClassLoader
>     def classFile(className: String): Class[_]
>   }
>
>   trait EnumerableClassfileProvider extends ClassfileProvider {
>     def allClassNames(): Iterable[String]
>     def allClassFiles(): Iterable[Class[_]]
>   }
>
> etc.  And then standard implementations like
>
>   class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider {
> ... }
>   class MavenClassfileLoader(group: String, id: String, version:
> String) extends EnumerableClassfileProvider { ... }
>   // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
>
Looks like an excellent abstraction to me. I think now that the IDE is
out is the best moment to change over. Ccing the IDE team to see if
they have issues.

Eugene: In light of this I think we can defer the classloader issue
for the moment (i.e. simply include output directory in tools
classpath, so that the normal classloader works). It's a hack, but it
will go away once ClassFileProviders are in.

Cheers

ClassProviderFactorySingletonMetaBuilder

On Wed, Dec 21, 2011 at 7:23 PM, Paul Phillips <paulp@improving.org> wrote:

On Wed, Dec 21, 2011 at 9:54 AM, martin odersky <martin.odersky@epfl.ch> wrote:

I also think we should do #1. This brings up the question, what's the
best way to do this? Is there an easy way to obtain a Java ClassLoader
from a Scala ClassPath?

Well, I'm glad you asked.
So what I'd like to do, and all I really need is a little rubber stamping, is re-model everything which presently operates on classpaths to operate on a different abstraction, something with a new name to evade conceptual conflation, like "ClassfileProvider." It'd at its most basic look something like
  trait ClassfileProvider {    def classBytes(className: String): Array[Byte]  }
This would be augmented by more capable variations such as:
  trait ClassfileLoader extends ClassfileProvider {    def classLoader: ScalaClassLoader    def classFile(className: String): Class[_]  }
  trait EnumerableClassfileProvider extends ClassfileProvider {     def allClassNames(): Iterable[String]    def allClassFiles(): Iterable[Class[_]]  }
etc.  And then standard implementations like
  class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider { ... }   class MavenClassfileLoader(group: String, id: String, version: String) extends EnumerableClassfileProvider { ... }  // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
And mechanisms for composition etc.  I want to be able to invoke scala like this (although it might have to be something other than '-cp')

  scala -cp asm%asm%3.2:org.scalaz%scalaz-core%7.0 my.fancy.Program
...and have the standard way of translating "classprovider expressions" into "classproviders" resolve those through the ivy cache.  The existing classpath is just one specific case of that (where all "expressions" are filesystem paths, and the provider is always a "filesystem provider".)
Lots of good stuff emerges from this approach.  I can provide a stub implementation in fairly short order, and I also have a large classpath rewrite which is I think an undeniable improvement but which I've had lying around for months because it mucks with the interface and I assume that will break the IDE.  Let me know.

--
Viktor Klang

Akka Tech LeadTypesafe - Enterprise-Grade Scala from the Experts

Twitter: @viktorklang

Well, I played with this a bit, and there is an easy way to obtain a
classloader that corresponds to -cp. You just use
ScalaClassLoader.fromURLs(global.classPath.asURLs). So I've been able
to experiment with the idea of a special classloader for macros.

It was relatively easy to instantiate a mirror with a custom
classloader, but then I stumbled upon an "object is not an instance of
a class" problem. Possibly, I have overlooked something, so this is
not a blocker, but it raises a question. Is it okay that macro
expansion is executed in the context of a classloader that is
different from compiler's one?

On Dec 21, 7:40 pm, martin odersky wrote:
> On Wed, Dec 21, 2011 at 7:23 PM, Paul Phillips wrote:
>
> > On Wed, Dec 21, 2011 at 9:54 AM, martin odersky
> > wrote:
>
> >> I also think we should do #1. This brings up the question, what's the
> >> best way to do this? Is there an easy way to obtain a Java ClassLoader
> >> from a Scala ClassPath?
>
> > Well, I'm glad you asked.
>
> > So what I'd like to do, and all I really need is a little rubber stamping,
> > is re-model everything which presently operates on classpaths to operate on
> > a different abstraction, something with a new name to evade conceptual
> > conflation, like "ClassfileProvider." It'd at its most basic look something
> > like
>
> >   trait ClassfileProvider {
> >     def classBytes(className: String): Array[Byte]
> >   }
>
> > This would be augmented by more capable variations such as:
>
> >   trait ClassfileLoader extends ClassfileProvider {
> >     def classLoader: ScalaClassLoader
> >     def classFile(className: String): Class[_]
> >   }
>
> >   trait EnumerableClassfileProvider extends ClassfileProvider {
> >     def allClassNames(): Iterable[String]
> >     def allClassFiles(): Iterable[Class[_]]
> >   }
>
> > etc.  And then standard implementations like
>
> >   class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider {
> > ... }
> >   class MavenClassfileLoader(group: String, id: String, version:
> > String) extends EnumerableClassfileProvider { ... }
> >   // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
>
> Looks like an excellent abstraction to me. I think now that the IDE is
> out is the best moment to change over. Ccing the IDE team to see if
> they have issues.
>
> Eugene: In light of this I think we can defer the classloader issue
> for the moment (i.e. simply include output directory in tools
> classpath, so that the normal classloader works). It's a hack, but it
> will go away once ClassFileProviders are in.
>
> Cheers
>
>  -- Martin

Hi Paul,

On Dec 21, 2011, at 8:23 PM, Paul Phillips wrote:

>
>
> On Wed, Dec 21, 2011 at 9:54 AM, martin odersky wrote:
> I also think we should do #1. This brings up the question, what's the
> best way to do this? Is there an easy way to obtain a Java ClassLoader
> from a Scala ClassPath?
>
> Well, I'm glad you asked.
>
> So what I'd like to do, and all I really need is a little rubber stamping, is re-model everything which presently operates on classpaths to operate on a different abstraction, something with a new name to evade conceptual conflation, like "ClassfileProvider." It'd at its most basic look something like
>
> trait ClassfileProvider {
> def classBytes(className: String): Array[Byte]
> }
>
> This would be augmented by more capable variations such as:
>
> trait ClassfileLoader extends ClassfileProvider {
> def classLoader: ScalaClassLoader
> def classFile(className: String): Class[_]
> }
>
> trait EnumerableClassfileProvider extends ClassfileProvider {
> def allClassNames(): Iterable[String]
> def allClassFiles(): Iterable[Class[_]]
> }
>
> etc. And then standard implementations like
>
> class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider { ... }
> class MavenClassfileLoader(group: String, id: String, version: String) extends EnumerableClassfileProvider { ... }
> // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
>
> And mechanisms for composition etc. I want to be able to invoke scala like this (although it might have to be something other than '-cp')
>
> scala -cp asm%asm%3.2:org.scalaz%scalaz-core%7.0 my.fancy.Program
>
> ...and have the standard way of translating "classprovider expressions" into "classproviders" resolve those through the ivy cache. The existing classpath is just one specific case of that (where all "expressions" are filesystem paths, and the provider is always a "filesystem provider".)
>
> Lots of good stuff emerges from this approach. I can provide a stub implementation in fairly short order, and I also have a large classpath rewrite which is I think an undeniable improvement but which I've had lying around for months because it mucks with the interface and I assume that will break the IDE. Let me know.

Let's keep in mind, from the operations perspective, that an application deployed to a server will most certainly not be given the flexibility to download needed jars but they will have to be at some local CLASSPATH (normally near the application).

That said, I, too, like the idea of the providers stuff. Maybe you would like to have a look at my context-based resource provisioning library with similar usage semantics:

https://github.com/loverdos/streamresource

It basically generalizes the getResource() pattern and delegates resource resolving/provisioning to resource contexts (= the resolvers/providers) that can be composed in hierarchies.

https://github.com/loverdos/streamresource/blob/master/src/main/scala/co...
https://github.com/loverdos/streamresource/blob/master/src/main/scala/co...
https://github.com/loverdos/streamresource/blob/master/src/main/scala/co...

The last one, CompositeStreamResourceContext, is the entity that can give a prioritization to a collection of resource contexts (which one is asked first to provide a resource and so on). My typical use-case is: where do I find an application's properties file? So, I have a default classpath-based context that resolves from the classloader and then I override this with a filesystem-based context. So, if the application is started with a manually provided properties file in the filesystem, then that one is chosen, otherwise the properties from the classpath is chosen.

Best,
Christos

On Thu, Dec 22, 2011 at 4:20 AM, Christos KK Loverdos
wrote:
> Let's keep in mind, from the operations perspective, that an application deployed to a server will most certainly not be given the flexibility to download needed jars but they will have to be at some local CLASSPATH (normally near the application).

That some environments face constraints which others do not is pretty
much the defining principle of the work. I'm not doing anything nutty
like creating new network dependencies. You'll still be able to use
classpaths exactly as you do now.

> https://github.com/loverdos/streamresource

Thanks, I will take a look.