Worried about the compilation speed of a mid-sized scala project

On 12/12/2010 13:07, Luis Pureza wrote:
> Hello my fellow scala users,

Hello,

> [...]
> Currently, it takes about 4 minutes just to compile those 23000 lines
> of scala (including tests) on a modern machine with 4 CPUs and 4 GB
> [...]
> Some people over the internet seem to agree that the scala compiler is
> slow, but I wonder if our numbers are normal.

That is quite along the time we are experiencing (even a little faster
than what we have), so I think your numbers are normal.

> If they are, I'd like to
> know if there is anything we can do to find out where it is spending
> its time, so that maybe we can do something about it.

I'm interested in that too. I mean, for a fresh, full build, incremental
building is already used when possible ;)

> Also, I'd like
> to know whether scala developers have acknowledged this issue and are
> working on it, or it is not really considered a priority.

The issue is very well known, but AFAIK, really little can be done, and
even a two fold speed-up is not to be expected in the near future. Scala
compiler is just doing a bunch of things, and these things take time.

Anyhow, improvements in that domain will be very welcome.

 I wonder how much help SBT could provide in allowing the project definition to
have some kinf of  "these modules are parallelizable" hints... so that multi core
CPUs could be put to use... or does scalac already does all the parallelism already ?


On Sun, Dec 12, 2010 at 7:30 AM, Francois <fanf42@gmail.com> wrote:

On 12/12/2010 13:07, Luis Pureza wrote:

Hello my fellow scala users,

Hello,

[...]
Currently, it takes about 4 minutes just to compile those 23000 lines
of scala (including tests) on a modern machine with 4 CPUs and 4 GB
[...]
Some people over the internet seem to agree that the scala compiler is
slow, but I wonder if our numbers are normal.

That is quite along the time we are experiencing (even a little faster than what we have), so I think your numbers are normal.

If they are, I'd like to
know if there is anything we can do to find out where it is spending
its time, so that maybe we can do something about it.

I'm interested in that too. I mean, for a fresh, full build, incremental building is already used when possible ;)

Also, I'd like
to know whether scala developers have acknowledged this issue and are
working on it, or it is not really considered a priority.

The issue is very well known, but AFAIK, really little can be done, and even a two fold speed-up is not to be expected in the near future. Scala compiler is just doing a bunch of things, and these things take time.

Anyhow, improvements in that domain will be very welcome.

Hmm interesting - if the modules in an SBT project have
inter-depedencies then i doubt you'd be able to do any parallelization
at that level. If however they were independent (unlikely) then maybe
you could spin up another scalac process or something. Otherwise, it
seems any parallelization in compilation would need to be at the
scalac level because all the compilation processes would need access
to the type information etc. Not that I know a great deal about that
though...

2010/12/12 Maxime Lévesque :
>
>  I wonder how much help SBT could provide in allowing the project definition
> to
> have some kinf of  "these modules are parallelizable" hints... so that multi
> core
> CPUs could be put to use... or does scalac already does all the parallelism
> already ?
>
>
> On Sun, Dec 12, 2010 at 7:30 AM, Francois wrote:
>>
>> On 12/12/2010 13:07, Luis Pureza wrote:
>>>
>>> Hello my fellow scala users,
>>
>> Hello,
>>
>>> [...]
>>> Currently, it takes about 4 minutes just to compile those 23000 lines
>>> of scala (including tests) on a modern machine with 4 CPUs and 4 GB
>>> [...]
>>> Some people over the internet seem to agree that the scala compiler is
>>> slow, but I wonder if our numbers are normal.
>>
>> That is quite along the time we are experiencing (even a little faster
>> than what we have), so I think your numbers are normal.
>>
>>> If they are, I'd like to
>>> know if there is anything we can do to find out where it is spending
>>> its time, so that maybe we can do something about it.
>>
>> I'm interested in that too. I mean, for a fresh, full build, incremental
>> building is already used when possible ;)
>>
>>> Also, I'd like
>>> to know whether scala developers have acknowledged this issue and are
>>> working on it, or it is not really considered a priority.
>>
>> The issue is very well known, but AFAIK, really little can be done, and
>> even a two fold speed-up is not to be expected in the near future. Scala
>> compiler is just doing a bunch of things, and these things take time.
>>
>> Anyhow, improvements in that domain will be very welcome.
>>
>>
>> --
>> Francois ARMAND
>> http://fanf42.blogspot.com
>> http://www.normation.com
>
>

Maven 3 supports parallel builds. It could be worthwhile to see try it
or see how they implemented it.
https://cwiki.apache.org/MAVEN/parallel-builds-in-maven-3.html

@tommychheng
http://tommy.chheng.com

On 12/12/10 5:14 AM, Tim Perrett wrote:
> Hmm interesting - if the modules in an SBT project have
> inter-depedencies then i doubt you'd be able to do any parallelization
> at that level. If however they were independent (unlikely) then maybe
> you could spin up another scalac process or something. Otherwise, it
> seems any parallelization in compilation would need to be at the
> scalac level because all the compilation processes would need access
> to the type information etc. Not that I know a great deal about that
> though...
>
> 2010/12/12 Maxime Lévesque:
>> I wonder how much help SBT could provide in allowing the project definition
>> to
>> have some kinf of "these modules are parallelizable" hints... so that multi
>> core
>> CPUs could be put to use... or does scalac already does all the parallelism
>> already ?
>>
>>
>> On Sun, Dec 12, 2010 at 7:30 AM, Francois wrote:
>>> On 12/12/2010 13:07, Luis Pureza wrote:
>>>> Hello my fellow scala users,
>>> Hello,
>>>
>>>> [...]
>>>> Currently, it takes about 4 minutes just to compile those 23000 lines
>>>> of scala (including tests) on a modern machine with 4 CPUs and 4 GB
>>>> [...]
>>>> Some people over the internet seem to agree that the scala compiler is
>>>> slow, but I wonder if our numbers are normal.
>>> That is quite along the time we are experiencing (even a little faster
>>> than what we have), so I think your numbers are normal.
>>>
>>>> If they are, I'd like to
>>>> know if there is anything we can do to find out where it is spending
>>>> its time, so that maybe we can do something about it.
>>> I'm interested in that too. I mean, for a fresh, full build, incremental
>>> building is already used when possible ;)
>>>
>>>> Also, I'd like
>>>> to know whether scala developers have acknowledged this issue and are
>>>> working on it, or it is not really considered a priority.
>>> The issue is very well known, but AFAIK, really little can be done, and
>>> even a two fold speed-up is not to be expected in the near future. Scala
>>> compiler is just doing a bunch of things, and these things take time.
>>>
>>> Anyhow, improvements in that domain will be very welcome.
>>>
>>>
>>> --
>>> Francois ARMAND
>>> http://fanf42.blogspot.com
>>> http://www.normation.com
>>

On another thread someone mentioned that they had built their work
machine with SSD drives. Anybody have experience using these to
build?

On 12/12/2010 22:27, JamesJ wrote:
> On another thread someone mentioned that they had built their work
> machine with SSD drives. Anybody have experience using these to
> build?

I just bought a SSD last week. The compilation time of the projects is
significantly reduced (i did no calculation, but I'm not afraid anymore
to do a mvn clean install). I could try to compare with Francois, we
work on the same projects and have same hardware except the SSD

However, Eclipse is still behaving as Eclipse. The only advantage is you
know faster when it's crashing

--

Nicolas CHARLES
Normation SAS - http://www.normation.com
44 rue Cauchy – 94110 ARCUEIL
+33 (0)1 83 62 26 96 - +33 (0)6 14 63 25 18

As a comparison, my 2008 Mac Book Pro with 4GB of RAM takes around 40
mins to do an entire build of Lift whilst a friends brand new i7 MBP
with an SSD was taking something stupid like 8 or 10 mins.
Unscientific yes, but perhaps it'll give you some food for thought.

Cheers, Tim

On 12 December 2010 22:58, Nicolas Charles
wrote:
> On 12/12/2010 22:27, JamesJ wrote:
>>
>> On another thread someone mentioned that they had built their work
>> machine with SSD drives.  Anybody have experience using these to
>> build?
>
>
> I just bought a SSD last week. The compilation time of the projects is
> significantly reduced (i did no calculation, but I'm not afraid anymore to
> do a mvn clean install). I could try to compare with Francois, we work on
> the same projects and have same hardware except the SSD
>
> However, Eclipse is still behaving as Eclipse. The only advantage is you
> know faster when it's crashing
>
>
>
> --
>
> Nicolas CHARLES
> Normation SAS - http://www.normation.com
> 44 rue Cauchy – 94110 ARCUEIL
> +33 (0)1 83 62 26 96  - +33 (0)6 14 63 25 18
>
>
>
>

2010/12/12 Maxime Lévesque :
>  I wonder how much help SBT could provide in allowing the project definition
> to
> have some kinf of  "these modules are parallelizable" hints... so that multi
> core
> CPUs could be put to use... or does scalac already does all the parallelism
> already ?

SBT already supports parallel execution. It's listed as one of the
main features in the home page:

"Parallel task execution, including parallel test execution"

Also see the parallelExecution method that can be overridden in your
project definition.

Best,
Ismael

On Sun, Dec 12, 2010 at 9:27 PM, JamesJ wrote:
> On another thread someone mentioned that they had built their work
> machine with SSD drives.  Anybody have experience using these to
> build?

The scala compiler is CPU-bound and SSDs don't help much. I tested
compilation of the scala distribution with and without SSD and the
difference was negligible. Seth Tisue also had similar results.

Best,
Ismael

On 13/12/2010 11:56, Ismael Juma wrote:
> On Sun, Dec 12, 2010 at 9:27 PM, JamesJ wrote:
>> On another thread someone mentioned that they had built their work
>> machine with SSD drives. Anybody have experience using these to
>> build?
> The scala compiler is CPU-bound and SSDs don't help much. I tested
> compilation of the scala distribution with and without SSD and the
> difference was negligible. Seth Tisue also had similar results.
>
> Best,
> Ismael
Indeed, we just did the test
mvn clean install
- With a SSD : 2.09 minutes
- Without a SSD : 2.02 minutes

A bit disappointing ...

Interestingly, it only uses one core, except on small part of the build

Nicolas CHARLES
Normation SAS - http://www.normation.com
44 rue Cauchy – 94110 ARCUEIL
+33 (0)1 83 62 26 96 - +33 (0)6 14 63 25 18

On Sun, Dec 12, 2010 at 12:07 PM, Luis Pureza wrote:
> Also, I'd like
> to know whether scala developers have acknowledged this issue and are
> working on it, or it is not really considered a priority.

The best short-term hope is SBT 0.9.x which will include a more
sophisticated incremental compilation mode that should improve build
times significantly in some cases:

"In sbt 0.7.4, this would cause both B.scala and C.scala to be
recompiled. With the new incremental compilation in 0.9.x, C.scala is
recompiled, the result is analyzed, and it is seen that only the
implementation of C.x changed, not its signature. Therefore, the
transitive dependencies of C.scala do not need to be recompiled. This
will mainly help larger projects and multi-project builds. It does
nothing to speed up initial compilation and can even be slower when
signatures do change because it does multiple compilation runs in this
case."
http://code.google.com/p/simple-build-tool/wiki/090p1tour

Best,
Ismael

Well, for a functional programming language, holding the promise of unlocking all those cores, it's a bit unnerving that the compiler itself has no use for all those cores...?

Having said that, I do understand the complexity of making the thing multithreaded, but if the compiler folks hookup with the parallel data-flow functional programming mavens, why can't we make the compiler use more cores?

I mean we know the single core speed won't improve as much in the future as the number of cores, so shouldn't this be an important part of a 3.0?

Also... having said that, organizing your project into well thought out dependent multiple components (or sub-projects) will allow you to use ant or sbt parallel builds...

Also, after having said that, I do notice significant speed boost, in eclipse, after getting an SSD and, if you don't have the money for an SSD but do have the RAM, then just move the workspace into RAM for the ultimate in I/O performance boost: http://blog.normation.com/2010/07/27/set-up-eclipse-workspace-in-ram/

Cheers,
Razvan

-----Original Message-----
From: Nicolas Charles [mailto:nicolas.charles@normation.com]
Sent: Monday, December 13, 2010 6:05 AM
To: scala-user@listes.epfl.ch
Subject: Re: [scala-user] Worried about the compilation speed of a mid-sized scala project

On 13/12/2010 11:56, Ismael Juma wrote:
> On Sun, Dec 12, 2010 at 9:27 PM, JamesJ wrote:
>> On another thread someone mentioned that they had built their work
>> machine with SSD drives. Anybody have experience using these to
>> build?
> The scala compiler is CPU-bound and SSDs don't help much. I tested
> compilation of the scala distribution with and without SSD and the
> difference was negligible. Seth Tisue also had similar results.
>
> Best,
> Ismael
Indeed, we just did the test
mvn clean install
- With a SSD : 2.09 minutes
- Without a SSD : 2.02 minutes

A bit disappointing ...

Interestingly, it only uses one core, except on small part of the build

Nicolas CHARLES
Normation SAS - http://www.normation.com
44 rue Cauchy – 94110 ARCUEIL
+33 (0)1 83 62 26 96 - +33 (0)6 14 63 25 18

Our project has 13k lines and it takes less than 30 seconds to compile
on a quadcore with 2GB RAM.
We're simply using NetBeans or IntelliJ without any additional build tools.
I wonder if we get better compile times because we minimize package
dependencies.

On 13/12/2010 18:36, Razvan Cojocaru wrote:
> Also, after having said that, I do notice significant speed boost, in eclipse, after getting an SSD and, if you don't have the money for an SSD but do have the RAM, then just move the workspace into RAM for the ultimate in I/O performance boost: http://blog.normation.com/2010/07/27/set-up-eclipse-workspace-in-ram/
>
Ha :) I was using this method prior to use an SSD. And moving from RAM
to SSD doesn't slow down eclipse (doesn't speed it up either)

Nicolas

Well, there's also the fact that continuous cleaning and rebuilding of many files reduce the life of said SSD, from what I understand - this way I have no doubts about doing a clean and rebuild :) granted, more psychology than science... :)

But yeah, no big difference between SSD and memory. However, big difference HDD to memory, eh?

-----Original Message-----
From: Nicolas Charles [mailto:nicolas.charles@normation.com]
Sent: December-13-10 1:33 PM
To: scala-user@listes.epfl.ch
Subject: Re: [scala-user] Worried about the compilation speed of a mid-sized scala project

On 13/12/2010 18:36, Razvan Cojocaru wrote:
> Also, after having said that, I do notice significant speed boost, in
> eclipse, after getting an SSD and, if you don't have the money for an
> SSD but do have the RAM, then just move the workspace into RAM for the
> ultimate in I/O performance boost:
> http://blog.normation.com/2010/07/27/set-up-eclipse-workspace-in-ram/
>
Ha :) I was using this method prior to use an SSD. And moving from RAM to SSD doesn't slow down eclipse (doesn't speed it up either)

Nicolas

Check that you have the modifications to the pom described below:

http://blogtrader.net/blog/how_to_setup_dependencies_aware2

On Sun, Dec 12, 2010 at 10:07, Luis Pureza <pureza@gmail.com> wrote:

Hello my fellow scala users,

I am part of a team which is developing a mid-sized application in
Scala and Java. The application is comprised of four separate modules
and contains about 23000 lines of Scala code. So far, I am very happy
with Scala because it allows me to think in terms of higher
abstractions and to remain productive. I'm sure half the functionality
would not be there if we were just using Java.

However, as the application grows, something that started as a small
nuisance is becoming, line by line, a major pain: compilation time.
Currently, it takes about 4 minutes just to compile those 23000 lines
of scala (including tests) on a modern machine with 4 CPUs and 4 GB
RAM. We use SBT to minor the pain, but sometimes we really have to use
maven which seems to be quite dumb and recompiles everything all the
time. Anyway, even with SBT the problem does not go away: one of those
scalatest files with 1000 lines easily takes 20 seconds to compile.

Some people over the internet seem to agree that the scala compiler is
slow, but I wonder if our numbers are normal. If they are, I'd like to
know if there is anything we can do to find out where it is spending
its time, so that maybe we can do something about it. Also, I'd like
to know whether scala developers have acknowledged this issue and are
working on it, or it is not really considered a priority.

Thank you,

Luís Pureza

--
Daniel C. Sobral

I travel to the future all the time.