Compiler plugin unit-testing: how to build/get instances of Type?

On Fri, Aug 27, 2010 at 06:27:48PM +0100, Olivier Pernet wrote:
> Unfortunately, if I try to get them by calling eg.
> definitions.StringClass.tpe, I get an AssertionError in the
> constructor of TypeHistory, here

Yes, if you look at the source to Global you will see that the
definitions object is initialized on the first run.

scala> new scala.tools.nsc.Global(new scala.tools.nsc.Settings)
res0: scala.tools.nsc.Global = scala.tools.nsc.Global@63f5e4b6

// like you say
scala> res0.definitions.StringClass.tpe
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:77)

// create a compiler run
scala> new res0.Run
res2: res0.Run = scala.tools.nsc.Global$Run@d00078c

// now init has been run
scala> res0.definitions.StringClass.tpe
res3: res0.Type = java.lang.String

Ok, this seems to work fine in the REPL, but when I try to create the
Run in my unit test, I get the following error:

scala.tools.nsc.MissingRequirementError: object scala not found.
at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
at scala.tools.nsc.symtab.Definitions$definitions$.ScalaPackage(Definitions.scala:37)
at scala.tools.nsc.symtab.Definitions$definitions$.ScalaPackageClass(Definitions.scala:38)
at scala.tools.nsc.symtab.Definitions$definitions$.UnitClass(Definitions.scala:83)
at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:785)
at scala.tools.nsc.Global$Run.(Global.scala:597)

What do you think could cause this?

Olivier

On Fri, Aug 27, 2010 at 21:03, Paul Phillips wrote:
> new res0.Run

I just got it to work:

val settings = new Settings
val scalaVersion = "2.8.0"
settings.classpath.tryToSet(List(
"project/boot/scala-"+scalaVersion+"/lib/scala-compiler.jar" +
":project/boot/scala-"+scalaVersion+"/lib/scala-library.jar"))
val global = new Global(settings)
new global.Run

This is because on the REPL, the Scala library jar is already on the
classpath, but wasn't in my unit test environment.

Olivier

On Sat, Aug 28, 2010 at 00:39, Olivier Pernet wrote:
> Ok, this seems to work fine in the REPL, but when I try to create the
> Run in my unit test, I get the following error:
>
> scala.tools.nsc.MissingRequirementError: object scala not found.
>        at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
>        at scala.tools.nsc.symtab.Definitions$definitions$.ScalaPackage(Definitions.scala:37)
>        at scala.tools.nsc.symtab.Definitions$definitions$.ScalaPackageClass(Definitions.scala:38)
>        at scala.tools.nsc.symtab.Definitions$definitions$.UnitClass(Definitions.scala:83)
>        at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:785)
>        at scala.tools.nsc.Global$Run.(Global.scala:597)
>
> What do you think could cause this?
>
> Olivier
>
>
> On Fri, Aug 27, 2010 at 21:03, Paul Phillips wrote:
>> new res0.Run
>

Would there be a more lightweight way to initialize just the
definitions? Creating a Run slows down my test quite a bit.

Only calling definitions.init crashes on the same assertion as described before.

If I copy the code from the Run constructor:
val phase1 = syntaxAnalyzer.newPhase(NoPhase)
phase = phase1
definitions.init

then it works in the REPL, but in the test I get:
java.lang.NullPointerException
at scala.tools.nsc.symtab.Types$class.scala$tools$nsc$symtab$Types$$unique(Types.scala:2722)
at scala.tools.nsc.symtab.Types$ThisType$.apply(Types.scala:1068)
at scala.tools.nsc.symtab.Symbols$ClassSymbol.thisType(Symbols.scala:1978)
at scala.tools.nsc.symtab.Symbols$TypeSymbol.tpe(Symbols.scala:1821)
at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:781)

Could I make this work? Would it make any speed difference compared to
just new Run?
I just need to be able to get some simple types.

Olivier

On Sat, Aug 28, 2010 at 00:47, Olivier Pernet wrote:
> I just got it to work:
>
> val settings = new Settings
> val scalaVersion = "2.8.0"
> settings.classpath.tryToSet(List(
>            "project/boot/scala-"+scalaVersion+"/lib/scala-compiler.jar" +
>            ":project/boot/scala-"+scalaVersion+"/lib/scala-library.jar"))
> val global = new Global(settings)
> new global.Run
>
> This is because on the REPL, the Scala library jar is already on the
> classpath, but wasn't in my unit test environment.
>
> Olivier
>
>
> On Sat, Aug 28, 2010 at 00:39, Olivier Pernet wrote:
>> Ok, this seems to work fine in the REPL, but when I try to create the
>> Run in my unit test, I get the following error:
>>
>> scala.tools.nsc.MissingRequirementError: object scala not found.
>>        at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:513)
>>        at scala.tools.nsc.symtab.Definitions$definitions$.ScalaPackage(Definitions.scala:37)
>>        at scala.tools.nsc.symtab.Definitions$definitions$.ScalaPackageClass(Definitions.scala:38)
>>        at scala.tools.nsc.symtab.Definitions$definitions$.UnitClass(Definitions.scala:83)
>>        at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:785)
>>        at scala.tools.nsc.Global$Run.(Global.scala:597)
>>
>> What do you think could cause this?
>>
>> Olivier
>>
>>
>> On Fri, Aug 27, 2010 at 21:03, Paul Phillips wrote:
>>> new res0.Run
>>
>

On Sat, Aug 28, 2010 at 12:47:55AM +0100, Olivier Pernet wrote:
> val settings = new Settings
> val scalaVersion = "2.8.0"
> settings.classpath.tryToSet(List(
> "project/boot/scala-"+scalaVersion+"/lib/scala-compiler.jar" +
> ":project/boot/scala-"+scalaVersion+"/lib/scala-library.jar"))

Normally settings.usejavacp.value = true will do what you want.

On Sat, Aug 28, 2010 at 02:42:56AM +0100, Olivier Pernet wrote:
> Would there be a more lightweight way to initialize just the
> definitions? Creating a Run slows down my test quite a bit.

I believe this feature awaits your implementation. Based on your emails
thus far you know everything you need to know: "okay, let's see how it
crashes if I do this..."

On Sat, Aug 28, 2010 at 02:42:56AM +0100, Olivier Pernet wrote:
> If I copy the code from the Run constructor:
> val phase1 = syntaxAnalyzer.newPhase(NoPhase)
> phase = phase1
> definitions.init
>
> then it works in the REPL, but in the test I get:
> java.lang.NullPointerException
> at scala.tools.nsc.symtab.Types$class.scala$tools$nsc$symtab$Types$$unique(Types.scala:2722)

Sometimes if I start pulling a thread like this it goes and goes and
goes, but not this time. That was the whole thread.

- private var uniques: HashSet[AnyRef] = _
+ private var uniques: HashSet[AnyRef] = new HashSet("uniques", initialUniquesCapacity)

// init.scala
import scala.tools.nsc._

val global = new Global(new Settings)
import global._
global.phase = syntaxAnalyzer.newPhase(NoPhase)
definitions.init
println(definitions.StringClass.tpe)

% build/pack/bin/scala -i work/init.scala
Loading work/init.scala...
import scala.tools.nsc._
global: scala.tools.nsc.Global = scala.tools.nsc.Global@bcaeccf
import global._
java.lang.String

On Sat, Aug 28, 2010 at 12:05:25AM -0700, Paul Phillips wrote:
> val global = new Global(new Settings)
> import global._
> global.phase = syntaxAnalyzer.newPhase(NoPhase)
> definitions.init

Oh, and I was briefly hopeful this would be good news for repl startup
time, but I ran it 100 times each way and there was all of four
milliseconds difference between new current.Run() and calling init. So
unless you want to keep dredging I don't think this will be the solution
you've been dreaming of.

On Friday, August 27, 2010, Paul Phillips wrote:
> On Sat, Aug 28, 2010 at 12:47:55AM +0100, Olivier Pernet wrote:
> > val settings = new Settings
> > val scalaVersion = "2.8.0"
> > settings.classpath.tryToSet(List(
> > "project/boot/scala-"+scalaVersion+"/lib/scala-compiler.jar" +
> > ":project/boot/scala-"+scalaVersion+"/lib/scala-library.jar"))
>
> Normally settings.usejavacp.value = true will do what you want.

Because of the project/boot/scala, I'd guess the tests are running through sbt. This kind of thing is the motivation for http://gist.github.com/404272.

-Mark

> On Sat, Aug 28, 2010 at 02:42:56AM +0100, Olivier Pernet wrote:
> > Would there be a more lightweight way to initialize just the
> > definitions? Creating a Run slows down my test quite a bit.
>
> I believe this feature awaits your implementation. Based on your emails
> thus far you know everything you need to know: "okay, let's see how it
> crashes if I do this..."
>
>

On Sat, Aug 28, 2010 at 11:51:44AM -0400, Mark Harrah wrote:
> > Normally settings.usejavacp.value = true will do what you want.
>
> Because of the project/boot/scala, I'd guess the tests are running
> through sbt. This kind of thing is the motivation for
> http://gist.github.com/404272.

Funny, the very same situation in sbt brought me near tears yesterday,
until I finally realized I can get at the classpath from a test via the
context classloader.

val context = Thread.currentThread.getContextClassLoader.asInstanceOf[java.net.URLClassLoader]
val cpurls = context.getURLs.toList
val cpurlString = ClassPath.join(cpurls map (_.getPath) : _*)

[...]
scala.tools.nsc.Main.process(Array("-d", outdir.path, "-cp", cpurlString) ++ srcpaths)

Ha sbt, you cannot defeat me. BTW that url does not discuss the context
classloader. I do not completely understand its wily ways, but does
knowing that this works with current sbt alter your view on solutions?

On Saturday, August 28, 2010, Paul Phillips wrote:
> On Sat, Aug 28, 2010 at 11:51:44AM -0400, Mark Harrah wrote:
> > > Normally settings.usejavacp.value = true will do what you want.
> >
> > Because of the project/boot/scala, I'd guess the tests are running
> > through sbt. This kind of thing is the motivation for
> > http://gist.github.com/404272.
>
> Funny, the very same situation in sbt brought me near tears yesterday,
> until I finally realized I can get at the classpath from a test via the
> context classloader.
>
> val context = Thread.currentThread.getContextClassLoader.asInstanceOf[java.net.URLClassLoader]
> val cpurls = context.getURLs.toList
> val cpurlString = ClassPath.join(cpurls map (_.getPath) : _*)
>
> [...]
> scala.tools.nsc.Main.process(Array("-d", outdir.path, "-cp", cpurlString) ++ srcpaths)

I'd be surprised if this worked if you are compiling classes that reference the compiler. I'm pretty sure that the context class loader won't contain the Scala jars, since those are in a parent. Now, if you have disabled filterScalaJars, those jars might be there. This wouldn't be the usual case, though.

> Ha sbt, you cannot defeat me. BTW that url does not discuss the context
> classloader. I do not completely understand its wily ways, but does
> knowing that this works with current sbt alter your view on solutions?

Not really. It makes assumptions about the context class loader that might change.

Anyway, the patch to scalac part of the proposal ensures you don't even have to do the above. It should just work with sbt. Right now, you can do it manually, though. See the getClasspath("app" ... and getClasspath("scala" ... statements.

-Mark

On Sat, Aug 28, 2010 at 01:10:16PM -0400, Mark Harrah wrote:
> Now, if you have disabled filterScalaJars, those jars might be there.
> This wouldn't be the usual case, though.

You have correctly identified mr. no-filter. Oh well.