To those hacking scalac in IntelliJ

It's a bit ricky to set this up, but it works well and several people at EPFL use IntelliJ.
The compiler sources have the following dependencies: - a scala library (I usually run an "ant" build first and use "build/quick/classes/library")  - the following .jars from "lib": fjbg.jar, msil.jar, forkjoin.jar, jline.jar - ant.jar (in the "lib" folder your local ant installation)
Then, you need to specify the compiler you actually use for compiling. That is done in the Scala facet ("use compiler libraries from specified jars"). I use the "locker" compiler(the one that compiled the quick library), making sure to add "msil.jar" and "fjbg.jar":
Scala compiler libraries: /Users/luc/scala/git/build/locker/classes/compiler:/Users/luc/scala/git/lib/fjbg.jar:/Users/luc/scala/git/lib/msil.jarScala SDK libraries: /Users/luc/scala/git/build/locker/classes/library
If you want to use fsc, you need to run the compile server manually on some commandline (intellij somehow doesn't manage to do it). Go to Project settings, Scala Compiler, tick "use fsc". Then, on a console, the command to run the compile server should look similarto this (on a mac, at least):
/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java -Xmx1024M -Denv.classpath="%CLASSPATH%" -Denv.emacs="%EMACS%" -Dfile.encoding=UTF-8 -classpath /Users/luc/scala/trunk/build/locker/classes/library:/Users/luc/scala/trunk/build/locker/classes/compiler:/Users/luc/scala/trunk/lib/fjbg.jar scala.tools.nsc.CompileServer

Attached my IntelliJ project files. Maybe I should commit them to "trunk/src/intellij" as ".SAMPLE",the ones that are currently there are a bit outdated.
Cheers: Lukas


On Wed, Jul 21, 2010 at 23:06, Grzegorz Kossakowski <grek@google.com> wrote:

Hi,

I'm trying to setup environment to be able to work on scalac source
code from IntelliJ and have an ability to run it from IntelliJ in
order to quikcly test my changes and possibly debug them.

My questions are:
1. How far are projects files one kind find in src/intellij from
something that can be imported to intellij and used for compiling
scalac source code?
   I tried to import complier and library project but it complains
about missing stuff like ant.jar etc. I'm basically wondering how far
I'm.
2. Is it possible to have incremental compiling configured for scalac
source code? Possibly fsc will be helpful? Anyone lucky with getting
it to work?

As you can see I'm asking for general advice when it comes to hacking
scalac from Intellij.

--
Best regards,
Grzegorz Kossakowski

These instructions are a little outdated -- the latest nightly builds
of the Scala plugin no longer have a per-module Scala Facet. See the
new instructions:

http://devnet.jetbrains.net/thread/289722

-jason

On Thu, Jul 22, 2010 at 1:56 PM, Lukas Rytz wrote:
> It's a bit ricky to set this up, but it works well and several people at
> EPFL use IntelliJ.
> The compiler sources have the following dependencies:
>  - a scala library (I usually run an "ant" build first and use
> "build/quick/classes/library")
>  - the following .jars from "lib": fjbg.jar, msil.jar, forkjoin.jar,
> jline.jar
>  - ant.jar (in the "lib" folder your local ant installation)
> Then, you need to specify the compiler you actually use for compiling. That
> is done in
> the Scala facet ("use compiler libraries from specified jars"). I use the
> "locker" compiler
> (the one that compiled the quick library), making sure to add "msil.jar" and
> "fjbg.jar":
> Scala compiler
> libraries: /Users/luc/scala/git/build/locker/classes/compiler:/Users/luc/scala/git/lib/fjbg.jar:/Users/luc/scala/git/lib/msil.jar
> Scala SDK libraries: /Users/luc/scala/git/build/locker/classes/library
> If you want to use fsc, you need to run the compile server manually on some
> command
> line (intellij somehow doesn't manage to do it). Go to Project settings,
> Scala Compiler, tick
> "use fsc". Then, on a console, the command to run the compile server should
> look similar
> to this (on a mac, at least):
> /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java
> -Xmx1024M -Denv.classpath="%CLASSPATH%" -Denv.emacs="%EMACS%"
> -Dfile.encoding=UTF-8 -classpath
> /Users/luc/scala/trunk/build/locker/classes/library:/Users/luc/scala/trunk/build/locker/classes/compiler:/Users/luc/scala/trunk/lib/fjbg.jar
> scala.tools.nsc.CompileServer
>
> Attached my IntelliJ project files. Maybe I should commit them to
> "trunk/src/intellij" as ".SAMPLE",
> the ones that are currently there are a bit outdated.
> Cheers: Lukas
>
>
> On Wed, Jul 21, 2010 at 23:06, Grzegorz Kossakowski wrote:
>>
>> Hi,
>>
>> I'm trying to setup environment to be able to work on scalac source
>> code from IntelliJ and have an ability to run it from IntelliJ in
>> order to quikcly test my changes and possibly debug them.
>>
>> My questions are:
>> 1. How far are projects files one kind find in src/intellij from
>> something that can be imported to intellij and used for compiling
>> scalac source code?
>>    I tried to import complier and library project but it complains
>> about missing stuff like ant.jar etc. I'm basically wondering how far
>> I'm.
>> 2. Is it possible to have incremental compiling configured for scalac
>> source code? Possibly fsc will be helpful? Anyone lucky with getting
>> it to work?
>>
>> As you can see I'm asking for general advice when it comes to hacking
>> scalac from Intellij.
>>
>> --
>> Best regards,
>> Grzegorz Kossakowski
>
>

We simplified compiling ordinary projects in recent IntelliJ scala plugin builds (only nightly builds), and this instruction doesn’t work. Actually it’s impossible now to compile scala compiler sources using IntelliJ.

We are going to build scala compiler sources with IntelliJ (after some our code patching), so after that I’ll send newer instructions about how to build (and without newer instruction I can offer you just to use build 0.3.1814).

 

Best regards,

Alexander Podkhalyuzin.

 

From: rytz.epfl@gmail.com [mailto:rytz.epfl@gmail.com] On Behalf Of Lukas Rytz
Sent: Thursday, July 22, 2010 3:56 PM
To: Grzegorz Kossakowski
Cc: scala-internals@listes.epfl.ch
Subject: Re: [scala-internals] To those hacking scalac in IntelliJ

 

It's a bit ricky to set this up, but it works well and several people at EPFL use IntelliJ.

 

The compiler sources have the following dependencies:

 - a scala library (I usually run an "ant" build first and use "build/quick/classes/library")

 - the following .jars from "lib": fjbg.jar, msil.jar, forkjoin.jar, jline.jar

 - ant.jar (in the "lib" folder your local ant installation)

 

Then, you need to specify the compiler you actually use for compiling. That is done in

the Scala facet ("use compiler libraries from specified jars"). I use the "locker" compiler

(the one that compiled the quick library), making sure to add "msil.jar" and "fjbg.jar":

 

Scala compiler libraries: /Users/luc/scala/git/build/locker/classes/compiler:/Users/luc/scala/git/lib/fjbg.jar:/Users/luc/scala/git/lib/msil.jar

Scala SDK libraries: /Users/luc/scala/git/build/locker/classes/library

 

If you want to use fsc, you need to run the compile server manually on some command

line (intellij somehow doesn't manage to do it). Go to Project settings, Scala Compiler, tick

"use fsc". Then, on a console, the command to run the compile server should look similar

to this (on a mac, at least):

 

/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java -Xmx1024M -Denv.classpath="%CLASSPATH%" -Denv.emacs="%EMACS%" -Dfile.encoding=UTF-8 -classpath /Users/luc/scala/trunk/build/locker/classes/library:/Users/luc/scala/trunk/build/locker/classes/compiler:/Users/luc/scala/trunk/lib/fjbg.jar scala.tools.nsc.CompileServer

 

 

Attached my IntelliJ project files. Maybe I should commit them to "trunk/src/intellij" as ".SAMPLE",

the ones that are currently there are a bit outdated.

 

Cheers: Lukas

 

 

 

On Wed, Jul 21, 2010 at 23:06, Grzegorz Kossakowski <grek@google.com> wrote:

Hi,

I'm trying to setup environment to be able to work on scalac source
code from IntelliJ and have an ability to run it from IntelliJ in
order to quikcly test my changes and possibly debug them.

My questions are:
1. How far are projects files one kind find in src/intellij from
something that can be imported to intellij and used for compiling
scalac source code?
   I tried to import complier and library project but it complains
about missing stuff like ant.jar etc. I'm basically wondering how far
I'm.
2. Is it possible to have incremental compiling configured for scalac
source code? Possibly fsc will be helpful? Anyone lucky with getting
it to work?

As you can see I'm asking for general advice when it comes to hacking
scalac from Intellij.

--
Best regards,
Grzegorz Kossakowski

 

Hello Alexander,

> We simplified compiling ordinary projects in recent IntelliJ scala plugin builds (only nightly builds), and this instruction doesn’t work. Actually it’s impossible now to compile scala compiler sources using IntelliJ.
> We are going to build scala compiler sources with IntelliJ (after some our code patching), so after that I’ll send newer instructions about how to build (and without newer instruction I can offer you just to use build 0.3.1814).

I personally regret that you decided to move away from the "facets" approach to integrating Scala in IntelliJ. I believe that facets were introduced — at least in part — following my repeated requests to Ilya Sergey to allow a clean separation between the compilation classpath and the classpath used to instantiated the Scala compiler. This is absolutely necessary for any type of cross-compilation. Building the Scala compiler and library is one such example of cross-compilation, but not the only one. Any project that has to deal with multiple version of Scala will require control of these two classpaths separately.

To clarify the situation, here are the correct contents for the two classpaths to build the "quick" Scala compiler and library ("quick" is the main build of Scala that corresponds to what is released):

- Build the library first:
- Compiler instantiation classpath is the "locker" compiler and library (in "build/locker/classes/").
- Compilation classpath contains the Java standard library and NOTHING ELSE.

- Build the compiler afterward:
- Compiler instantiation classpath is the same as above.
- Compilation classpath contains the Java standard library, the Scala library compiled above (in IntelliJ, a module dependency), and required libraries: Ant, FJBG, MSIL and JLine.

Any other setup for the classpath will not yield correct builds. The facet approach allowed configuring the classpaths like above. But if IntelliJ now doesn't support that, it cannot safely be used for developing Scala (or for serious cross-compilation).

I understand that new users find the Scala facet confusing. In all fairness, it took me a while to understand what it was about too (despite having spoken with Ilya about it). However, I believe that the confusion may have more to do with the design of the facet configuration page, which was spartan, to say the least, rather than the underlying complexity of the mechanism.

If there is still a chance of keeping a mechanism that allows separately specifying the classpaths for compiler instantiation and compilation (with a better UI than the original facets) that would be extremely useful for us Scala compiler and library developers.

I'd be happy to comment on such a design if it can be of any use.

Cheers,
Gilles.

+1
I really hope that it will be possible again to specify the build compiler and referencesseparately, this is crucial for developing the compiler. It's a main reason why I use IntelliJ and not eclipse.
Lukas

On Thu, Jul 22, 2010 at 16:12, Gilles Dubochet <gilles.dubochet@epfl.ch> wrote:

Hello Alexander,

> We simplified compiling ordinary projects in recent IntelliJ scala plugin builds (only nightly builds), and this instruction doesn’t work. Actually it’s impossible now to compile scala compiler sources using IntelliJ.
> We are going to build scala compiler sources with IntelliJ (after some our code patching), so after that I’ll send newer instructions about how to build (and without newer instruction I can offer you just to use build 0.3.1814).

I personally regret that you decided to move away from the "facets" approach to integrating Scala in IntelliJ. I believe that facets were introduced — at least in part — following my repeated requests to Ilya Sergey to allow a clean separation between the compilation classpath and the classpath used to instantiated the Scala compiler. This is absolutely necessary for any type of cross-compilation. Building the Scala compiler and library is one such example of cross-compilation, but not the only one. Any project that has to deal with multiple version of Scala will require control of these two classpaths separately.

To clarify the situation, here are the correct contents for the two classpaths to build the "quick" Scala compiler and library ("quick" is the main build of Scala that corresponds to what is released):

 - Build the library first:
  - Compiler instantiation classpath is the "locker" compiler and library (in "build/locker/classes/").
  - Compilation classpath contains the Java standard library and NOTHING ELSE.

 - Build the compiler afterward:
  - Compiler instantiation classpath is the same as above.
  - Compilation classpath contains the Java standard library, the Scala library compiled above (in IntelliJ, a module dependency), and required libraries: Ant, FJBG, MSIL and JLine.

Any other setup for the classpath will not yield correct builds. The facet approach allowed configuring the classpaths like above. But if IntelliJ now doesn't support that, it cannot safely be used for developing Scala (or for serious cross-compilation).

I understand that new users find the Scala facet confusing. In all fairness, it took me a while to understand what it was about too (despite having spoken with Ilya about it). However, I believe that the confusion may have more to do with the design of the facet configuration page, which was spartan, to say the least, rather than the underlying complexity of the mechanism.

If there is still a chance of keeping a mechanism that allows separately specifying the classpaths for compiler instantiation and compilation (with a better UI than the original facets) that would be extremely useful for us Scala compiler and library developers.

I'd be happy to comment on such a design if it can be of any use.

Cheers,
Gilles.

On Thu, Jul 22, 2010 at 7:56 AM, Lukas Rytz wrote:
> It's a bit ricky to set this up, but it works well and several people at
> EPFL use IntelliJ.
> The compiler sources have the following dependencies:
>  - a scala library (I usually run an "ant" build first and use
> "build/quick/classes/library")
>  - the following .jars from "lib": fjbg.jar, msil.jar, forkjoin.jar,
> jline.jar
>  - ant.jar (in the "lib" folder your local ant installation)
> Then, you need to specify the compiler you actually use for compiling. That
> is done in
> the Scala facet ("use compiler libraries from specified jars"). I use the
> "locker" compiler
> (the one that compiled the quick library), making sure to add "msil.jar" and
> "fjbg.jar":
> Scala compiler
> libraries: /Users/luc/scala/git/build/locker/classes/compiler:/Users/luc/scala/git/lib/fjbg.jar:/Users/luc/scala/git/lib/msil.jar
> Scala SDK libraries: /Users/luc/scala/git/build/locker/classes/library

Lukas, with your project files I was able to compile scalac from
Intellij not using fsc which is some progres but...

> If you want to use fsc, you need to run the compile server manually on some
> command
> line (intellij somehow doesn't manage to do it). Go to Project settings,
> Scala Compiler, tick
> "use fsc". Then, on a console, the command to run the compile server should
> look similar
> to this (on a mac, at least):
> /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java
> -Xmx1024M -Denv.classpath="%CLASSPATH%" -Denv.emacs="%EMACS%"
> -Dfile.encoding=UTF-8 -classpath
> /Users/luc/scala/trunk/build/locker/classes/library:/Users/luc/scala/trunk/build/locker/classes/compiler:/Users/luc/scala/trunk/lib/fjbg.jar
> scala.tools.nsc.CompileServer

This doesn't work for me. If I just tick use fsc and provide it a port
number found in
/tmp/scala-devel/grek/scalac-compile-server-port

then I get following error:
atal error: object scala not found.

and idea.log shows:
2010-07-22 10:49:55,623 [ 227265] INFO -
ij.compiler.impl.CompileDriver - COMPILATION STARTED
2010-07-22 10:49:55,657 [ 227299] INFO -
lij.compiler.impl.CompilerUtil - Clearing output dirs took 3 ms: 0 min
0sec
2010-07-22 10:49:55,657 [ 227299] INFO -
lij.compiler.impl.CompilerUtil - Refreshing output directories took 27
ms: 0 min 0sec
2010-07-22 10:49:56,279 [ 227921] INFO -
lij.compiler.impl.CompilerUtil - Initial VFS refresh took 620 ms: 0
min 0sec
2010-07-22 10:49:57,882 [ 229524] INFO -
lij.compiler.impl.CompilerUtil - COMPILATION FINISHED; Errors: 2;
warnings: 0 took 2259 ms: 0 min 2sec

if I additionally tick 'reset fsc server' then I get:
Scalac internal error: class java.lang.ClassNotFoundException
[java.net.URLClassLoader$1.run(URLClassLoader.java:217),
java.security.AccessController.doPrivileged(Native Method),
java.net.URLClassLoader.findClass(URLClassLoader.java:205),
java.lang.ClassLoader.loadClass(ClassLoader.java:321),
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294),
java.lang.ClassLoader.loadClass(ClassLoader.java:266),
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:334),
java.lang.Class.forName0(Native Method),
java.lang.Class.forName(Class.java:186),
org.jetbrains.plugins.scala.compiler.rt.FastScalacRunner.main(FastScalacRunner.java:59)]

and idea.log shows the same output as above. Any ideas?

For me, it often only works on the second time, i.e. when it doesn't work, ^C the CompileServerand start it again, then try to compile again in IntelliJ.
How I got the command: Create a new IntelliJ-Scala Hello World project, compile & run. Then create a "Scala Compilation Server" run configuration and run it. You'll see the commandin the output window. Take the command and replace the compiler classpath with the "locker" one (add fjbg / msil). IRC it looked different on Linux, it used some intellij launcher.
Lukas


On Thu, Jul 22, 2010 at 16:56, Grzegorz Kossakowski <grek@google.com> wrote:

On Thu, Jul 22, 2010 at 7:56 AM, Lukas Rytz <lukas.rytz@epfl.ch> wrote:
> It's a bit ricky to set this up, but it works well and several people at
> EPFL use IntelliJ.
> The compiler sources have the following dependencies:
>  - a scala library (I usually run an "ant" build first and use
> "build/quick/classes/library")
>  - the following .jars from "lib": fjbg.jar, msil.jar, forkjoin.jar,
> jline.jar
>  - ant.jar (in the "lib" folder your local ant installation)
> Then, you need to specify the compiler you actually use for compiling. That
> is done in
> the Scala facet ("use compiler libraries from specified jars"). I use the
> "locker" compiler
> (the one that compiled the quick library), making sure to add "msil.jar" and
> "fjbg.jar":
> Scala compiler
> libraries: /Users/luc/scala/git/build/locker/classes/compiler:/Users/luc/scala/git/lib/fjbg.jar:/Users/luc/scala/git/lib/msil.jar
> Scala SDK libraries: /Users/luc/scala/git/build/locker/classes/library

Lukas, with your project files I was able to compile scalac from
Intellij not using fsc which is some progres but...

> If you want to use fsc, you need to run the compile server manually on some
> command
> line (intellij somehow doesn't manage to do it). Go to Project settings,
> Scala Compiler, tick
> "use fsc". Then, on a console, the command to run the compile server should
> look similar
> to this (on a mac, at least):
> /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java
> -Xmx1024M -Denv.classpath="%CLASSPATH%" -Denv.emacs="%EMACS%"
> -Dfile.encoding=UTF-8 -classpath
> /Users/luc/scala/trunk/build/locker/classes/library:/Users/luc/scala/trunk/build/locker/classes/compiler:/Users/luc/scala/trunk/lib/fjbg.jar
> scala.tools.nsc.CompileServer

This doesn't work for me. If I just tick use fsc and provide it a port
number found in
/tmp/scala-devel/grek/scalac-compile-server-port

then I get following error:
atal error: object scala not found.

and idea.log shows:
2010-07-22 10:49:55,623 [ 227265]   INFO -
ij.compiler.impl.CompileDriver - COMPILATION STARTED
2010-07-22 10:49:55,657 [ 227299]   INFO -
lij.compiler.impl.CompilerUtil - Clearing output dirs took 3 ms: 0 min
0sec
2010-07-22 10:49:55,657 [ 227299]   INFO -
lij.compiler.impl.CompilerUtil - Refreshing output directories took 27
ms: 0 min 0sec
2010-07-22 10:49:56,279 [ 227921]   INFO -
lij.compiler.impl.CompilerUtil - Initial VFS refresh took 620 ms: 0
min 0sec
2010-07-22 10:49:57,882 [ 229524]   INFO -
lij.compiler.impl.CompilerUtil -        COMPILATION FINISHED; Errors: 2;
warnings: 0 took 2259 ms: 0 min 2sec

if I additionally tick 'reset fsc server' then I get:
Scalac internal error: class java.lang.ClassNotFoundException
[java.net.URLClassLoader$1.run(URLClassLoader.java:217),
java.security.AccessController.doPrivileged(Native Method),
java.net.URLClassLoader.findClass(URLClassLoader.java:205),
java.lang.ClassLoader.loadClass(ClassLoader.java:321),
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294),
java.lang.ClassLoader.loadClass(ClassLoader.java:266),
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:334),
java.lang.Class.forName0(Native Method),
java.lang.Class.forName(Class.java:186),
org.jetbrains.plugins.scala.compiler.rt.FastScalacRunner.main(FastScalacRunner.java:59)]

and idea.log shows the same output as above. Any ideas?

--
Grzegorz

On Thu, Jul 22, 2010 at 11:05 AM, Lukas Rytz <lukas.rytz@epfl.ch> wrote:

For me, it often only works on the second time, i.e. when it doesn't work, ^C the CompileServerand start it again, then try to compile again in IntelliJ.

It should make things easier on IntelliJ users if the plugin were to run CompileServer itself rather than using fsc's machinery for spawning off a JVM. For that matter, it should benefit further from peeling an additional layer away and, instead of using CompileServer, directly making and caching an instance of Global.
Fsc is designed to help out people at the command line keep a resident compiler instance around. An IDE is already resident, however, so it doesn't have that problem. Thus, fsc is solving a problem an IDE doesn't have, but is adding new ones.
-Lex 

IntelliJ maintains a process boundary between itself and Scalac to
support compilation with different versions of Scalac from one
instance of IntelliJ and one version of the Scala plugin.

It could still spawn a manage a child process transparently. Usability
in this area does need some improvement, but there is some more
fundamental work in progress with type inference.

-jason

On Wed, Jul 28, 2010 at 3:49 PM, Lex Spoon wrote:
> On Thu, Jul 22, 2010 at 11:05 AM, Lukas Rytz wrote:
>>
>> For me, it often only works on the second time, i.e. when it doesn't
>> work, ^C the CompileServer
>> and start it again, then try to compile again in IntelliJ.
>
> It should make things easier on IntelliJ users if the plugin were to run
> CompileServer itself rather than using fsc's machinery for spawning off a
> JVM. For that matter, it should benefit further from peeling an additional
> layer away and, instead of using CompileServer, directly making and caching
> an instance of Global.
> Fsc is designed to help out people at the command line keep a resident
> compiler instance around. An IDE is already resident, however, so it doesn't
> have that problem. Thus, fsc is solving a problem an IDE doesn't have, but
> is adding new ones.