How to make Scala compile faster?

Are you using fsc (http://www.scala-lang.org/docu/files/tools/fsc.html)? I don't know if Eclipse's plugin uses it, but it's worth checking out.
Peter Robinett

On Tue, Oct 27, 2009 at 10:59 PM, John Ky <newhoggy@gmail.com> wrote:

Hi Scala list,

I have a small project, which I build in Eclipse using the Scala plugin.  It's gotten to a size where compilation would take a long time - about a minute or so.  At this point it is becoming counterproductive because on many occasions, I need to do a clean and force a full build after some minor code changes because something in the compilation process became corrupted.

Is there any way, I can recode it to compile faster?  For instance - if I added return types declarations to my methods, would Scala compile my code faster?

Cheers,

-John

On Wed, Oct 28, 2009 at 5:59 AM, John Ky wrote:
> I have a small project, which I build in Eclipse using the Scala plugin.

Version?

Cheers,

Miles

Hi Miles,

Eclipse SDK    3.5.1.M20090917-0800    org.eclipse.sdk.ide
Scala Eclipse Plugin    2.7.6.final    ch.epfl.lamp.sdt.feature.group

Cheers,

-John

On Wed, Oct 28, 2009 at 9:41 PM, Miles Sabin <miles@milessabin.com> wrote:

On Wed, Oct 28, 2009 at 5:59 AM, John Ky <newhoggy@gmail.com> wrote:
> I have a small project, which I build in Eclipse using the Scala plugin.

Version?

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

On Wed, Oct 28, 2009 at 10:45 AM, John Ky wrote:
> Eclipse SDK    3.5.1.M20090917-0800    org.eclipse.sdk.ide
> Scala Eclipse Plugin    2.7.6.final    ch.epfl.lamp.sdt.feature.group

Thanks.

You're seeing bugs in Eclipse's dependency management which are common
to all 2.7.x and earlier releases. This has been completely replaced
for 2.8 and should be a great deal more accurate in the future.

Cheers,

Miles

Hi Miles,

2.8 works very well.

Thankyou.

-John

On Wed, Oct 28, 2009 at 9:55 PM, Miles Sabin <miles@milessabin.com> wrote:

On Wed, Oct 28, 2009 at 10:45 AM, John Ky <newhoggy@gmail.com> wrote:
> Eclipse SDK    3.5.1.M20090917-0800    org.eclipse.sdk.ide
> Scala Eclipse Plugin    2.7.6.final    ch.epfl.lamp.sdt.feature.group

Thanks.

You're seeing bugs in Eclipse's dependency management which are common
to all 2.7.x and earlier releases. This has been completely replaced
for 2.8 and should be a great deal more accurate in the future.

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

On Wed, Oct 28, 2009 at 1:02 PM, John Ky wrote:
> 2.8 works very well.
>
> Thankyou.

Umm ... well, I wasn't actually recommending you to switch to 2.8 just
yet ... it's unreleased, beta quality, all bets are off, etc., etc.

Nevertheless, if it works well for you, then that's great :-)

Cheers,

Miles

Miles Sabin wrote:
> You're seeing bugs in Eclipse's dependency management which are common
> to all 2.7.x and earlier releases. This has been completely replaced
> for 2.8 and should be a great deal more accurate in the future.

Is this new dependency management in 2.8 supposed to work already (or
rather two or three weeks ago when I last updated to the latest nightly

I took a good look at the source for RefinedBuildManager recently. I tried
some things out with BuildManagerTest and came up with several examples that
failed to be recompiled correctly.

I'd like to provide some tests that demonstrate the issues, but I couldn't find
any examples to go off of in the associated commits to the build manager
classes. Is it possible to write such tests for partest?

As an example of the issues I found, adding a subclass to a sealed hierarchy
will not cause pattern matches on the hierarchy to be recompiled. This means
that warnings that a match is not exhaustive will not be printed, for example.

Thanks,
Mark

On Saturday 31 October 2009, Stefan Zeiger wrote:
> Miles Sabin wrote:
> > You're seeing bugs in Eclipse's dependency management which are common
> > to all 2.7.x and earlier releases. This has been completely replaced
> > for 2.8 and should be a great deal more accurate in the future.
>
> Is this new dependency management in 2.8 supposed to work already (or
> rather two or three weeks ago when I last updated to the latest nightly

Hi Mark,

On Sun, 2009-11-01 at 22:30 -0500, Mark Harrah wrote:
> I took a good look at the source for RefinedBuildManager recently. I tried
> some things out with BuildManagerTest and came up with several examples that
> failed to be recompiled correctly.

I'm happy to see this.

I was just saying on IRC the other day that it would be nice if you
would take a look at scalac's dependency manager given your experience
with sbt's one (and the fact that the latter works well according to
many people's experience). :)

Best,
Ismael

P.S. Someone else will have to answer your questions, as I don't know
the answer. ;)

On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah <harrah@bu.edu> wrote:

I'd like to provide some tests that demonstrate the issues, but I couldn't find
any examples to go off of in the associated commits to the build manager
classes.  Is it possible to write such tests for partest?

Not at the moment, as I assume it involves several compilation rounds, modifying files in between. Partest was designed with a more static view of testing. How do you test sbt?
 

As an example of the issues I found, adding a subclass to a sealed hierarchy
will not cause pattern matches on the hierarchy to be recompiled. This means
that warnings that a match is not exhaustive will not be printed, for example.

It would be good to have tickets for these issues. Of course, a way to automatically test the build manager (or even the Eclipse plugin) is necessary, just that for the moment we don't have it.

iulian
 

Thanks,
Mark

On Saturday 31 October 2009, Stefan Zeiger wrote:
> Miles Sabin wrote:
> > You're seeing bugs in Eclipse's dependency management which are common
> > to all 2.7.x and earlier releases. This has been completely replaced
> > for 2.8 and should be a great deal more accurate in the future.
>
> Is this new dependency management in 2.8 supposed to work already (or
> rather two or three weeks ago when I last updated to the latest nightly
> -- I haven't gotten around to working on ScalaQuery since then) or is it
> still work in progress?
>
> To me, it doesn't look any better than in 2.7 and I have to do a full
> rebuild all the time. For example:
> - Rename a class and save -> Sources which use the old name are not
> recompiled and they don't show up as errors until I rebuild
> - Change some fields and save -> Running the app throws an
> IncompatibleClassChange error because dependent sources were not
>  recompiled.

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Monday 02 November 2009, Iulian Dragos wrote:
> On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah wrote:
> > I'd like to provide some tests that demonstrate the issues, but I
> > couldn't find
> > any examples to go off of in the associated commits to the build manager
> > classes. Is it possible to write such tests for partest?
>
> Not at the moment, as I assume it involves several compilation rounds,
> modifying files in between. Partest was designed with a more static view of
> testing. How do you test sbt?

Right, multiple compilation rounds and file changes. I wrote a test framework
that reads the commands and actions to run from a script. Commands are things
like copy, delete, or test for file existence. Actions are any sbt action,
like compile, run, or any custom action defined for the project being tested.
A test script might look like:

$ copy-file changes/First.scala src/A.scala
> compile [success]
$ copy-file changes/Second.scala src/A.scala
> compile [failure]

> > As an example of the issues I found, adding a subclass to a sealed
> > hierarchy
> > will not cause pattern matches on the hierarchy to be recompiled. This
> > means
> > that warnings that a match is not exhaustive will not be printed, for
> > example.
>
> It would be good to have tickets for these issues. Of course, a way to
> automatically test the build manager (or even the Eclipse plugin) is
> necessary, just that for the moment we don't have it.

Certainly, it was just that there were several issues and they are much easier
to explain with test cases. Also, I saw that Miles made some changes that
looked like he might have fixed an issue, but I don't want to have to manually
retry these examples every time there is a change.

-Mark

On Monday 02 November 2009, Ismael Juma wrote:
> Hi Mark,
>
> On Sun, 2009-11-01 at 22:30 -0500, Mark Harrah wrote:
> > I took a good look at the source for RefinedBuildManager recently. I
> > tried some things out with BuildManagerTest and came up with several
> > examples that failed to be recompiled correctly.
>
> I'm happy to see this.
>
> I was just saying on IRC the other day that it would be nice if you
> would take a look at scalac's dependency manager given your experience
> with sbt's one (and the fact that the latter works well according to
> many people's experience). :)

sbt's partial recompilation is the most conservative of all solutions I've
seen, so is the most correct short of recompiling everything, but doesn't try
anything too fancy to be faster.

The latest refined build manager is very aggressive in order to be faster but
still correct. There are two things it does towards this:
1) recompile dependencies only when public API has changed
2) recompile only affected dependencies and not necessarily transitive
dependencies

I think the first one is mostly achieved by the refined build manager (I will
file the issue I found) and is something I think should go in sbt.

I get the impression that the second one needs to be spec'd (not necessarily
in The Spec) because it is tricky and needs review to ensure all cases are
caught, even if the build manager has to be documented as not being able to
handle certain cases. Most issues come from #2.

-Mark

On Mon, 2009-11-02 at 08:18 -0500, Mark Harrah wrote:
> I think the first one is mostly achieved by the refined build manager (I will
> file the issue I found) and is something I think should go in sbt.

Thanks for the explanation. Do you think it makes sense for sbt to use
the refined build manager at some point? If not, why not?

Best,
Ismael

On Monday 02 November 2009, Ismael Juma wrote:
> On Mon, 2009-11-02 at 08:18 -0500, Mark Harrah wrote:
> > I think the first one is mostly achieved by the refined build manager (I
> > will file the issue I found) and is something I think should go in sbt.
>
> Thanks for the explanation. Do you think it makes sense for sbt to use
> the refined build manager at some point? If not, why not?

The refined build manager can only be used as a resident compiler, which means
no, it can't be used in sbt right now.

I like the first part of the refined build manager (detecting public API
changes), though. I'd like to use something similar in sbt, but again, it has
to be modified to work in non-resident mode.

Ignoring the resident compiler issue, I'd want to see a spec or something
(SID?) on the second part before using it in sbt. I don't care if I have to
write it, but it is something I'd want to see reviewed.

My current plan is to keep transitive recompilation but be smarter about when
it is triggered (i.e., use the first part of the refined build manager).

-Mark

On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah wrote:
> On Monday 02 November 2009, Ismael Juma wrote:
>> Thanks for the explanation. Do you think it makes sense for sbt to use
>> the refined build manager at some point? If not, why not?
>
> The refined build manager can only be used as a resident compiler, which means
> no, it can't be used in sbt right now.

Sure, but I don't think it'd take much to get it to provide most of
what sbt needs ... do you see any serious roadblocks?

Cheers,

Miles

On Monday 02 November 2009 17:15:58 Miles Sabin wrote:
> On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah wrote:
> > On Monday 02 November 2009, Ismael Juma wrote:
> >> Thanks for the explanation. Do you think it makes sense for sbt to use
> >> the refined build manager at some point? If not, why not?
> >
> > The refined build manager can only be used as a resident compiler, which
> > means no, it can't be used in sbt right now.
>
> Sure, but I don't think it'd take much to get it to provide most of
> what sbt needs ... do you see any serious roadblocks?
>
> Cheers,
>
>
> Miles
>

Miles,

Can all these new refined build manager's goodnesses be used to replace
current ant fsc task (or be wrapped as a new one)?

On Mon, Nov 2, 2009 at 3:11 PM, Andrew Gaydenko wrote:
> Can all these new refined build manager's goodnesses be used to replace
> current ant fsc task (or be wrapped as a new one)?

That would be a question for Iulian ... in principle I would have thought so.

Cheers,

Miles

On Mon, Nov 2, 2009 at 4:11 PM, Andrew Gaydenko <a@gaydenko.com> wrote:

On Monday 02 November 2009 17:15:58 Miles Sabin wrote:
> On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah <harrah@bu.edu> wrote:
> > On Monday 02 November 2009, Ismael Juma wrote:
> >> Thanks for the explanation. Do you think it makes sense for sbt to use
> >> the refined build manager at some point? If not, why not?
> >
> > The refined build manager can only be used as a resident compiler, which
> > means no, it can't be used in sbt right now.
>
> Sure, but I don't think it'd take much to get it to provide most of
> what sbt needs ... do you see any serious roadblocks?
>
> Cheers,
>
>
> Miles
>

Miles,

Can all these new refined build manager's goodnesses be used to replace
current ant fsc task (or be wrapped as a new one)?

The current build manager cuts some corners in order to keep things simple. The biggest barrier to using it in fsc (or in the ant task) is that it requires running in resident mode. That means it keeps state around (external references of each managed file, dependency graph between compilation units, current types and members of classes). References and the dependency graph can be easily saved/restored, the problem is with 'members'. It uses compiler symbols and types, and therefore needs a compiler instance. They are used for building a change set between an old and a new version of the same file. In principle it could be rewritten to not rely on them, but a lot of functionality already in scalac would need to be duplicateed (for instance, subtype checks). There's a tradeoff between accuracy and time to code these features. Unfortunately I had not much time (getting very close to thesis writing) so I did what I thought would work fine for Eclipse. It can certainly be improved, but I had to (temporarily) hand over even maitnenance.

Mark, spec of the build manager is a very good idea, I'm sure it will reveal missing cases and be good documentation. Simply I have not enough time now...

iulian



--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Mon, Nov 2, 2009 at 2:07 PM, Mark Harrah <harrah@bu.edu> wrote:

On Monday 02 November 2009, Iulian Dragos wrote:
> On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah <harrah@bu.edu> wrote:
> > I'd like to provide some tests that demonstrate the issues, but I
> > couldn't find
> > any examples to go off of in the associated commits to the build manager
> > classes.  Is it possible to write such tests for partest?
>
> Not at the moment, as I assume it involves several compilation rounds,
> modifying files in between. Partest was designed with a more static view of
> testing. How do you test sbt?

Right, multiple compilation rounds and file changes.  I wrote a test framework
that reads the commands and actions to run from a script.  Commands are things
like copy, delete, or test for file existence.  Actions are any sbt action,
like compile, run, or any custom action defined for the project being tested.
A test script might look like:

 $ copy-file changes/First.scala src/A.scala
 > compile [success]
 $ copy-file changes/Second.scala src/A.scala
 > compile [failure]

Wonderful, is this a standalone project that we could use, or part of sbt?

thanks,
iulian

> > As an example of the issues I found, adding a subclass to a sealed
> > hierarchy
> > will not cause pattern matches on the hierarchy to be recompiled. This
> > means
> > that warnings that a match is not exhaustive will not be printed, for
> > example.
>
> It would be good to have tickets for these issues. Of course, a way to
> automatically test the build manager (or even the Eclipse plugin) is
> necessary, just that for the moment we don't have it.

Certainly, it was just that there were several issues and they are much easier
to explain with test cases.  Also, I saw that Miles made some changes that
looked like he might have fixed an issue, but I don't want to have to manually
retry these examples every time there is a change.

-Mark

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Monday 02 November 2009 11:49, Iulian Dragos wrote:
> On Mon, Nov 2, 2009 at 4:11 PM, Andrew Gaydenko wrote:
>
> > On Monday 02 November 2009 17:15:58 Miles Sabin wrote:
> > > On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah wrote:
> > > > On Monday 02 November 2009, Ismael Juma wrote:
> > > >> Thanks for the explanation. Do you think it makes sense for sbt to
use
> > > >> the refined build manager at some point? If not, why not?
> > > >
> > > > The refined build manager can only be used as a resident compiler,
> > which
> > > > means no, it can't be used in sbt right now.
> > >
> > > Sure, but I don't think it'd take much to get it to provide most of
> > > what sbt needs ... do you see any serious roadblocks?
>
> The current build manager cuts some corners in order to keep things simple.
> The biggest barrier to using it in fsc (or in the ant task) is that it
> requires running in resident mode. That means it keeps state around
> (external references of each managed file, dependency graph between
> compilation units, current types and members of classes). References and the
> dependency graph can be easily saved/restored, the problem is with
> 'members'. It uses compiler symbols and types, and therefore needs a
> compiler instance. They are used for building a change set between an old
> and a new version of the same file. In principle it could be rewritten to
> not rely on them, but a lot of functionality already in scalac would need to
> be duplicateed (for instance, subtype checks). There's a tradeoff between
> accuracy and time to code these features. Unfortunately I had not much time
> (getting very close to thesis writing) so I did what I thought would work
> fine for Eclipse. It can certainly be improved, but I had to (temporarily)
> hand over even maitnenance.

I agree that the change set is probably undesirable to try to do non-resident
without a dedicated maintainer. I think it would be a substantial
improvement just to know that a source file's public API has changed. If it
changes, transitive recompilation would then be triggered (including the
original triggering files). More conservative than the refined manager, but
it should be just as correct as sbt's approach and much faster in the common
case of only modifying a method's implementation.

Perhaps this could be done by adding a publicHash lazy val to CompilationUnit
that computes a SHA1 hash of the public API (linearization, self type, member
signatures)? You could then compare hashes to see if it changed.

I know that there is still some work needed to make even this approach help
fsc. It might be a reasonable start though and it would help sbt (with some
work on my part).

> Mark, spec of the build manager is a very good idea, I'm sure it will reveal
> missing cases and be good documentation. Simply I have not enough time
> now...

I'll add it to my list. Not enough time for me either right now. If/When I
get back to working on dependency analysis in sbt or if I try to adapt my
test framework for scalac, I'll try to come up with something.

-Mark

On Monday 02 November 2009 11:53, Iulian Dragos wrote:
> On Mon, Nov 2, 2009 at 2:07 PM, Mark Harrah wrote:
> > On Monday 02 November 2009, Iulian Dragos wrote:
> > > On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah wrote:
> > > > I'd like to provide some tests that demonstrate the issues, but I
> > > > couldn't find
> > > > any examples to go off of in the associated commits to the build
> > manager
> > > > classes. Is it possible to write such tests for partest?
> > >
> > > Not at the moment, as I assume it involves several compilation rounds,
> > > modifying files in between. Partest was designed with a more static view
> > of
> > > testing. How do you test sbt?
> >
> > Right, multiple compilation rounds and file changes. I wrote a test
> > framework
> > that reads the commands and actions to run from a script. Commands are
> > things
> > like copy, delete, or test for file existence. Actions are any sbt
action,
> > like compile, run, or any custom action defined for the project being
> > tested.
> > A test script might look like:
> >
> > $ copy-file changes/First.scala src/A.scala
> > > compile [success]
> > $ copy-file changes/Second.scala src/A.scala
> > > compile [failure]
> >
>
> Wonderful, is this a standalone project that we could use, or part of sbt?

Sort of both. It needs sbt because it is an sbt plugin (the '> action' means
run the sbt action 'action'). So, it is not a part of sbt, but works on sbt
projects.

If the current committers don't have the time/inclination to implement this in
partest, I'll see about modifying mine for scalac. It might be some time
before I can work on this, though.

-Mark

I have a rough test framework for this now as a result of some other work I'm
doing. The only problem (so far) is that I don't think it is possible to tell
if compilation succeeded with the build manager. I think the following lines
from RefinedBuildManager (line 96 or so) are why:

run.compileFiles(files.toList)
if (compiler.reporter.hasErrors) {
compiler.reporter.reset
return
}

One option is to make addSourceFiles, removeFiles, and update return a Boolean
indicating whether they succeeded.

Below is what a test script for issue 2558 looks like. # for comments, $ for
filesystem commands, > for build manager commands. Lines start with - if they
should fail.

# setup scalac to compile to classes directory
$ mkdir classes
> init -d $PWD/classes

# initial compilation of all files, which should succeed
> add A.scala B.scala C.scala

# modify A.scala so that it does not conform to the structural type in B.scala
$ copy-file changes/A.scala A.scala
# The build manager should recompile C.scala and report this error, failing
the compile
-> update A.scala

Thanks,
Mark

On Monday 02 November 2009, Mark Harrah wrote:
> On Monday 02 November 2009 11:53, Iulian Dragos wrote:
> > On Mon, Nov 2, 2009 at 2:07 PM, Mark Harrah wrote:
> > > On Monday 02 November 2009, Iulian Dragos wrote:
> > > > On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah wrote:
> > > > > I'd like to provide some tests that demonstrate the issues, but I
> > > > > couldn't find
> > > > > any examples to go off of in the associated commits to the build
> > >
> > > manager
> > >
> > > > > classes. Is it possible to write such tests for partest?
> > > >
> > > > Not at the moment, as I assume it involves several compilation
> > > > rounds, modifying files in between. Partest was designed with a more
> > > > static view
> > >
> > > of
> > >
> > > > testing. How do you test sbt?
> > >
> > > Right, multiple compilation rounds and file changes. I wrote a test
> > > framework
> > > that reads the commands and actions to run from a script. Commands are
> > > things
> > > like copy, delete, or test for file existence. Actions are any sbt
>
> action,
>
> > > like compile, run, or any custom action defined for the project being
> > > tested.
> > > A test script might look like:
> > >
> > > $ copy-file changes/First.scala src/A.scala
> > >
> > > > compile [success]
> > >
> > > $ copy-file changes/Second.scala src/A.scala
> > >
> > > > compile [failure]
> >
> > Wonderful, is this a standalone project that we could use, or part of
> > sbt?
>
> Sort of both. It needs sbt because it is an sbt plugin (the '> action'
> means run the sbt action 'action'). So, it is not a part of sbt, but
> works on sbt projects.
>
> If the current committers don't have the time/inclination to implement this
> in partest, I'll see about modifying mine for scalac. It might be some
> time before I can work on this, though.
>
> -Mark
>