Scala compile time --- is 3 minutes for 2000 LOC unusual?

That seems extremely slow. On my machine it takes 20s to compile over
2k LOC (bad measure of LOC: cat `find -name *scala` | wc -l).
Although my machine is a lot beefier than a Macbook Pro. I don't
write scala code on my laptop because compilation is too slow
(Thinkpad X200s).

Do you use continuous compilation? sbt ~compile? It's a major help!

Peace. Michael

On Tue, Jan 10, 2012 at 4:50 PM, Scott Morrison
wrote:
> Dear all,
>
> I've been working on a project recently (related to category theory
> and databases), where we've been observing the compile time
> dramatically increasing, to the point it's a serious obstruction to
> development. Our codebase is approximately 2000 LOC (rather small),
> but it now takes about 3 minutes for "sbt clean compile" to run on a
> 2011 Macbook Pro. We do use lots of abstract type members, and method
> dependent types via -Xexperimental, if this is relevant.
>
> Does this sound unusual? Can anyone suggest ways to reduce compile
> time, or at least to work out what it is about our code that causes
> compile time to be so large?
>
> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
> feeling extraordinarily helpful; "./sbt test" in the main directory
> should compile everything and run the test suite.)
>
> best regards,
> Scott Morrison

On Wed, Jan 11, 2012 at 1:50 AM, Scott Morrison <scott.morrison@gmail.com> wrote:

I've been working on a project recently (related to category theory
and databases), where we've been observing the compile time
dramatically increasing, to the point it's a serious obstruction to
development. Our codebase is approximately 2000 LOC (rather small),
but it now takes about 3 minutes for "sbt clean compile" to run on a
2011 Macbook Pro. We do use lots of abstract type members, and method
dependent types via -Xexperimental, if this is relevant.

Does this sound unusual? Can anyone suggest ways to reduce compile
time, or at least to work out what it is about our code that causes
compile time to be so large?

(The code is available at https://tqft.net/hg/metaphor/ if anyone is
feeling extraordinarily helpful; "./sbt test" in the main directory
should compile everything and run the test suite.)

scalac -Ystatistics might give some insight into what's taking the longest.
-jason 

On 2012, Jan 11, at 1:50 AM, Scott Morrison wrote:
> development. Our codebase is approximately 2000 LOC (rather small),
> but it now takes about 3 minutes for "sbt clean compile" to run on a
> 2011 Macbook Pro. We do use lots of abstract type members, and method
> dependent types via -Xexperimental, if this is relevant.
>
> Does this sound unusual? Can anyone suggest ways to reduce compile
> time, or at least to work out what it is about our code that causes
> compile time to be so large?

Yup, that is slow. sbt clean compile takes 69 seconds on my 2011 macbook pro from 2011. The project is 10kloc.

I have 8GB of ram and an SSD. We found that reduced compile time by about 20% around here, but nothing that can explain your numbers.

yours
Geir

On Tuesday 10 January 2012 16:50:22 Scott Morrison wrote:
> Dear all,
>
> I've been working on a project recently (related to category theory
> and databases), where we've been observing the compile time
> dramatically increasing, to the point it's a serious obstruction to
> development. Our codebase is approximately 2000 LOC (rather small),
> but it now takes about 3 minutes for "sbt clean compile" to run on a
> 2011 Macbook Pro. We do use lots of abstract type members, and method
> dependent types via -Xexperimental, if this is relevant.

I think it really depends on code's degree of abstraction :-) At my case your
project (2432 LOC) takes about 70 seconds to compile, while on the same
machine it takes about 50 seconds to build a project about 16K LOC (also
without comments and blanks).

Andrew

>
> Does this sound unusual? Can anyone suggest ways to reduce compile
> time, or at least to work out what it is about our code that causes
> compile time to be so large?
>
> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
> feeling extraordinarily helpful; "./sbt test" in the main directory
> should compile everything and run the test suite.)
>
> best regards,
> Scott Morrison

It sounds like a long time; what heap are you running the compilation with? Obviously if you are running with 64Mb, this might explain things. I tend to run with 512Mb as a minimum and others on this list use >2g for larger projects
Chris

> Date: Tue, 10 Jan 2012 16:50:22 -0800
> Subject: [scala-user] Scala compile time --- is 3 minutes for 2000 LOC unusual?
> From: scott.morrison@gmail.com
> To: scala-user@googlegroups.com
>
> Dear all,
>
> I've been working on a project recently (related to category theory
> and databases), where we've been observing the compile time
> dramatically increasing, to the point it's a serious obstruction to
> development. Our codebase is approximately 2000 LOC (rather small),
> but it now takes about 3 minutes for "sbt clean compile" to run on a
> 2011 Macbook Pro. We do use lots of abstract type members, and method
> dependent types via -Xexperimental, if this is relevant.
>
> Does this sound unusual? Can anyone suggest ways to reduce compile
> time, or at least to work out what it is about our code that causes
> compile time to be so large?
>
> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
> feeling extraordinarily helpful; "./sbt test" in the main directory
> should compile everything and run the test suite.)
>
> best regards,
> Scott Morrison

On Tue, Jan 10, 2012 at 22:50, Scott Morrison wrote:
> Dear all,
>
> I've been working on a project recently (related to category theory
> and databases), where we've been observing the compile time
> dramatically increasing, to the point it's a serious obstruction to
> development. Our codebase is approximately 2000 LOC (rather small),
> but it now takes about 3 minutes for "sbt clean compile" to run on a
> 2011 Macbook Pro. We do use lots of abstract type members, and method
> dependent types via -Xexperimental, if this is relevant.

Experimental is experimental... :-)

> Does this sound unusual? Can anyone suggest ways to reduce compile
> time, or at least to work out what it is about our code that causes
> compile time to be so large?

It is my impression -- though I could be wrong! -- that some things
about typer are worse than O(n). In fact, some prints I saw looked
downright quadratic. So, if you have lot of types in scope at the same
time, things could get really nasty -- assuming, again, that this is
true.

>
> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
> feeling extraordinarily helpful; "./sbt test" in the main directory
> should compile everything and run the test suite.)

How does one download from that? I saw no obvious link to do so. Or
can I pass the URL to hg?

On 2012-01-11 1:50, Scott Morrison wrote:
> I've been working on a project recently (related to category theory
> and databases), where we've been observing the compile time
> dramatically increasing, to the point it's a serious obstruction to
> development. Our codebase is approximately 2000 LOC (rather small),
> but it now takes about 3 minutes for "sbt clean compile" to run on a
> 2011 Macbook Pro. We do use lots of abstract type members, and method
> dependent types via -Xexperimental, if this is relevant.

LOCs in Scala are not created equal. Take this file (27 sloc)
https://gist.github.com/1594475 and compile it:

$ time
/cygdrive/c/StandaloneApps/scala-2.10.0.r26037-b20111121023229/bin/scalac 1.scala

real 0m4.806s
user 0m0.274s
sys 0m0.305s

Now change the line that makes the compiler prove that the type for 2^4
is a subtype of 1*10+6

println((null:( _2 # ^ [_4] )):( _1 # * [_10] # + [_6] ))

to prove 2^5 <:< 3*10+2 instead

println((null:( _2 # ^ [_5] )):( _3 # * [_10] # + [_2] ))

and get some coffee...

$ time
/cygdrive/c/StandaloneApps/scala-2.10.0.r26037-b20111121023229/bin/scalac 1.scala

real 86m11.747s
user 0m0.151s
sys 0m0.442s

From 5 seconds to 86 minutes for 27 sloc!

In general, it can help to add type annotations in order to cut down on
compilation times by making it easier for the type inferencer to find
the right types. In this specific example, there's probably not much you
can do.

-sz

Thank you all for the advice; I've made significant progress,
especially thanks to Paul Phillips' suggestion to add [Any] in one
particular line, cutting compile time by 2/3rds!

Daniel -- if you are still interested, yes, you can pass that URL to
mercurial, "hg clone https://tqft.net/hg/metaphor".

best regards,
Scott

On Wed, Jan 11, 2012 at 21:27, Daniel Sobral wrote:
> On Tue, Jan 10, 2012 at 22:50, Scott Morrison wrote:
>> Dear all,
>>
>> I've been working on a project recently (related to category theory
>> and databases), where we've been observing the compile time
>> dramatically increasing, to the point it's a serious obstruction to
>> development. Our codebase is approximately 2000 LOC (rather small),
>> but it now takes about 3 minutes for "sbt clean compile" to run on a
>> 2011 Macbook Pro. We do use lots of abstract type members, and method
>> dependent types via -Xexperimental, if this is relevant.
>
> Experimental is experimental... :-)
>
>> Does this sound unusual? Can anyone suggest ways to reduce compile
>> time, or at least to work out what it is about our code that causes
>> compile time to be so large?
>
> It is my impression -- though I could be wrong! -- that some things
> about typer are worse than O(n). In fact, some prints I saw looked
> downright quadratic. So, if you have lot of types in scope at the same
> time, things could get really nasty -- assuming, again, that this is
> true.
>
>>
>> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
>> feeling extraordinarily helpful; "./sbt test" in the main directory
>> should compile everything and run the test suite.)
>
> How does one download from that? I saw no obvious link to do so. Or
> can I pass the URL to hg?
>
>
> --
> Daniel C. Sobral
>
> I travel to the future all the time.

Can you share that one particular line? I would be interested to see this.
The HG web interface is hard to use.
Thanks,Tommy
On Wednesday, January 11, 2012, Scott Morrison wrote:

Thank you all for the advice; I've made significant progress,
especially thanks to Paul Phillips' suggestion to add [Any] in one
particular line, cutting compile time by 2/3rds!

Daniel -- if you are still interested, yes, you can pass that URL to
mercurial, "hg clone https://tqft.net/hg/metaphor".

best regards,
Scott

On Wed, Jan 11, 2012 at 21:27, Daniel Sobral <dcsobral@gmail.com> wrote:
> On Tue, Jan 10, 2012 at 22:50, Scott Morrison <scott.morrison@gmail.com> wrote:
>> Dear all,
>>
>> I've been working on a project recently (related to category theory
>> and databases), where we've been observing the compile time
>> dramatically increasing, to the point it's a serious obstruction to
>> development. Our codebase is approximately 2000 LOC (rather small),
>> but it now takes about 3 minutes for "sbt clean compile" to run on a
>> 2011 Macbook Pro. We do use lots of abstract type members, and method
>> dependent types via -Xexperimental, if this is relevant.
>
> Experimental is experimental... :-)
>
>> Does this sound unusual? Can anyone suggest ways to reduce compile
>> time, or at least to work out what it is about our code that causes
>> compile time to be so large?
>
> It is my impression -- though I could be wrong! -- that some things
> about typer are worse than O(n). In fact, some prints I saw looked
> downright quadratic. So, if you have lot of types in scope at the same
> time, things could get really nasty -- assuming, again, that this is
> true.
>
>>
>> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
>> feeling extraordinarily helpful; "./sbt test" in the main directory
>> should compile everything and run the test suite.)
>
> How does one download from that? I saw no obvious link to do so. Or
> can I pass the URL to hg?
>
>
> --
> Daniel C. Sobral
>
> I travel to the future all the time.

--
Tommy Chheng

Just a note:  It takes Scala ~ 10 minutes to compiler/run tests.   Granted, it's doing *a lot* of building (since its tests are actually compilation tests.)  I think that's the worst-case project I've ever seen.
- Josh

On Wed, Jan 11, 2012 at 3:15 PM, Tommy Chheng <tommy.chheng@gmail.com> wrote:

Can you share that one particular line? I would be interested to see this.
The HG web interface is hard to use.
Thanks,Tommy
On Wednesday, January 11, 2012, Scott Morrison wrote:

Thank you all for the advice; I've made significant progress,
especially thanks to Paul Phillips' suggestion to add [Any] in one
particular line, cutting compile time by 2/3rds!

Daniel -- if you are still interested, yes, you can pass that URL to
mercurial, "hg clone https://tqft.net/hg/metaphor".

best regards,
Scott

On Wed, Jan 11, 2012 at 21:27, Daniel Sobral <dcsobral@gmail.com> wrote:
> On Tue, Jan 10, 2012 at 22:50, Scott Morrison <scott.morrison@gmail.com> wrote:
>> Dear all,
>>
>> I've been working on a project recently (related to category theory
>> and databases), where we've been observing the compile time
>> dramatically increasing, to the point it's a serious obstruction to
>> development. Our codebase is approximately 2000 LOC (rather small),
>> but it now takes about 3 minutes for "sbt clean compile" to run on a
>> 2011 Macbook Pro. We do use lots of abstract type members, and method
>> dependent types via -Xexperimental, if this is relevant.
>
> Experimental is experimental... :-)
>
>> Does this sound unusual? Can anyone suggest ways to reduce compile
>> time, or at least to work out what it is about our code that causes
>> compile time to be so large?
>
> It is my impression -- though I could be wrong! -- that some things
> about typer are worse than O(n). In fact, some prints I saw looked
> downright quadratic. So, if you have lot of types in scope at the same
> time, things could get really nasty -- assuming, again, that this is
> true.
>
>>
>> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
>> feeling extraordinarily helpful; "./sbt test" in the main directory
>> should compile everything and run the test suite.)
>
> How does one download from that? I saw no obvious link to do so. Or
> can I pass the URL to hg?
>
>
> --
> Daniel C. Sobral
>
> I travel to the future all the time.

--
Tommy Chheng

Dear Tommy,

yes, the problem was (quoting Paul) "finding the lub of a bunch of
sets whose types are themselves being inferred", and you can see the
change the shaved 60 seconds off compile time here:

http://tqft.net/hg/metaphor/diff/e005154ae030/src/main/scala/net/metapho...

(Of course, it's a silly line anyway; probably I should build a Tuple3
rather a List, at least.)

When I asked Paul how he tracked it down, he said that he passed the
scalac option -Ytyper-debug, and looked where it was when it paused
for 60 seconds.

best regards,
Scott Morrison

On Thu, Jan 12, 2012 at 05:15, Tommy Chheng wrote:
> Can you share that one particular line? I would be interested to see this.
>
> The HG web interface is hard to use.
>
> Thanks,
> Tommy
>
> On Wednesday, January 11, 2012, Scott Morrison wrote:
>>
>> Thank you all for the advice; I've made significant progress,
>> especially thanks to Paul Phillips' suggestion to add [Any] in one
>> particular line, cutting compile time by 2/3rds!
>>
>> Daniel -- if you are still interested, yes, you can pass that URL to
>> mercurial, "hg clone https://tqft.net/hg/metaphor".
>>
>> best regards,
>> Scott
>>
>> On Wed, Jan 11, 2012 at 21:27, Daniel Sobral wrote:
>> > On Tue, Jan 10, 2012 at 22:50, Scott Morrison
>> > wrote:
>> >> Dear all,
>> >>
>> >> I've been working on a project recently (related to category theory
>> >> and databases), where we've been observing the compile time
>> >> dramatically increasing, to the point it's a serious obstruction to
>> >> development. Our codebase is approximately 2000 LOC (rather small),
>> >> but it now takes about 3 minutes for "sbt clean compile" to run on a
>> >> 2011 Macbook Pro. We do use lots of abstract type members, and method
>> >> dependent types via -Xexperimental, if this is relevant.
>> >
>> > Experimental is experimental... :-)
>> >
>> >> Does this sound unusual? Can anyone suggest ways to reduce compile
>> >> time, or at least to work out what it is about our code that causes
>> >> compile time to be so large?
>> >
>> > It is my impression -- though I could be wrong! -- that some things
>> > about typer are worse than O(n). In fact, some prints I saw looked
>> > downright quadratic. So, if you have lot of types in scope at the same
>> > time, things could get really nasty -- assuming, again, that this is
>> > true.
>> >
>> >>
>> >> (The code is available at https://tqft.net/hg/metaphor/ if anyone is
>> >> feeling extraordinarily helpful; "./sbt test" in the main directory
>> >> should compile everything and run the test suite.)
>> >
>> > How does one download from that? I saw no obvious link to do so. Or
>> > can I pass the URL to hg?
>> >
>> >
>> > --
>> > Daniel C. Sobral
>> >
>> > I travel to the future all the time.
>
>
>
> --
> Tommy Chheng