compile speed

Hi all,
I'm noticing that my project (3,364 lines of code including comments) compiles (with either scalac or fsc) very slowly in general and particularly slowly on networked filesystems (the situation is even worse with -optimise).  Some details are listed below.  Is this expected?  I'm running scala 2.9.1.final on a intel i7 linux box.
thanks,wade

Local filesystem============
local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m18.409suser    1m39.777ssys     0m3.909s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m3.793s user    0m0.608ssys     0m0.154s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    0m34.715suser    0m0.566s sys     0m0.117s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala                                                                                                                       
real    0m17.423suser    0m0.569ssys     0m0.123s
Networked filesystem================
nfs$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    3m30.669suser    2m28.289ssys     0m2.392s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    4m7.908s user    0m0.400ssys     0m0.092s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m37.953suser    0m0.388s sys     0m0.084s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m6.757suser    0m0.328ssys     0m0.044s

On Wednesday 14 December 2011 21:58:18 S Wade wrote:
> I'm noticing that my project (3,364 lines of code including comments)
> compiles (with either scalac or fsc) very slowly in general and
> particularly slowly on networked filesystems (the situation is even worse
> with -optimise). Some details are listed below. Is this expected? I'm
> running scala 2.9.1.final on a intel i7 linux box.
>
> thanks,
> wade
>
>
> Local filesystem
> ============
>
> local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
>
> real 1m18.409s
> user 1m39.777s
> sys 0m3.909s
> ....

Insufficient RAM?

For ~18K lines of code (or ~24K with blank and comments) I get about 50
seconds on i5 at 3.2 GHz, 8GB RAM at 1600 MHz. Linux also.

On Wednesday 14 December 2011 23:19:18 Maxime Lévesque wrote:
> I gave up on compiling on anything on else than solid state drives,
> and my quality of life has increased dramatically.

With sufficient RAM (and normal operating system) HDD activity (after the
first project building) is almost zero due to filesystem caching.

Andrew

Hi all,
I'm noticing that my project (3,364 lines of code including comments) compiles (with either scalac or fsc) very slowly in general and particularly slowly on networked filesystems (the situation is even worse with -optimise).  Some details are listed below.  Is this expected?  I'm running scala 2.9.1.final on a intel i7 linux box.
thanks,wade

Local filesystem============
local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m18.409suser    1m39.777ssys     0m3.909s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m3.793s user    0m0.608ssys     0m0.154s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    0m34.715suser    0m0.566s sys     0m0.117s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala                                                                                                                       
real    0m17.423suser    0m0.569ssys     0m0.123s
Networked filesystem================
nfs$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    3m30.669suser    2m28.289ssys     0m2.392s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    4m7.908s user    0m0.400ssys     0m0.092s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m37.953suser    0m0.388s sys     0m0.084s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m6.757suser    0m0.328ssys     0m0.044s

On Wednesday 14 December 2011 23:35:25 Rex Kerr wrote:
> Although it is possible for complex pattern matching, among other things,
> to take surprising amounts of time to compile, I usually find that a slow
> compile is due to insufficient memory for the JVM. The default is
> something preposterously small for most modern computers, like 256M; give
> -J-Xmx2G or somesuch a try.
>
> --Rex

It seems JVM is more aggressive now. I have tried to build SBT project
starting sbt with and without

-Xmx2G -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m

options and got approximately the same result.

$ java -version
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)

Andrew

I have 128gb on that machine, so I don't think it's memory.

thanks,
wade

On Dec 14, 2011, at 11:34 PM, Andrew Gaydenko wrote:

> On Wednesday 14 December 2011 21:58:18 S Wade wrote:
>> I'm noticing that my project (3,364 lines of code including comments)
>> compiles (with either scalac or fsc) very slowly in general and
>> particularly slowly on networked filesystems (the situation is even worse
>> with -optimise). Some details are listed below. Is this expected? I'm
>> running scala 2.9.1.final on a intel i7 linux box.
>>
>> thanks,
>> wade
>>
>>
>> Local filesystem
>> ============
>>
>> local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
>> src/*.scala
>>
>> real 1m18.409s
>> user 1m39.777s
>> sys 0m3.909s
>> ....
>
> Insufficient RAM?
>
> For ~18K lines of code (or ~24K with blank and comments) I get about 50
> seconds on i5 at 3.2 GHz, 8GB RAM at 1600 MHz. Linux also.
>
> On Wednesday 14 December 2011 23:19:18 Maxime Lévesque wrote:
>> I gave up on compiling on anything on else than solid state drives,
>> and my quality of life has increased dramatically.
>
> With sufficient RAM (and normal operating system) HDD activity (after the
> first project building) is almost zero due to filesystem caching.
>
>
> Andrew

On Dec 15, 4:19 am, Maxime Lévesque wrote:
> I gave up on compiling on anything on else than solid state drives,
> and my quality of life has increased dramatically.

+1

ramdrive beats anything

-------- Original-Nachricht --------
> Datum: Thu, 15 Dec 2011 04:50:02 -0800 (PST)
> Von: bryan hunt
> An: scala-user
> Betreff: [scala-user] Re: compile speed

> On Dec 15, 4:19 am, Maxime Lévesque wrote:
> > I gave up on compiling on anything on else than solid state drives,
> > and my quality of life has increased dramatically.
>
> +1

Hi all,
I'm noticing that my project (3,364 lines of code including comments) compiles (with either scalac or fsc) very slowly in general and particularly slowly on networked filesystems (the situation is even worse with -optimise).  Some details are listed below.  Is this expected?  I'm running scala 2.9.1.final on a intel i7 linux box.
thanks,wade

Local filesystem============
local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m18.409suser    1m39.777ssys     0m3.909s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m3.793s user    0m0.608ssys     0m0.154s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    0m34.715suser    0m0.566s sys     0m0.117s
local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala                                                                                                                       
real    0m17.423suser    0m0.569ssys     0m0.123s
Networked filesystem================
nfs$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    3m30.669suser    2m28.289ssys     0m2.392s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    4m7.908s user    0m0.400ssys     0m0.092s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m37.953suser    0m0.388s sys     0m0.084s
nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build src/*.scala
real    1m6.757suser    0m0.328ssys     0m0.044s

i always imagine the compiler doing really awesome and complex stuff while compiling to kill the time

-------- Original-Nachricht --------
> Datum: Thu, 15 Dec 2011 17:12:02 +0100
> Von: Trond Olsen
> An: S Wade
> CC: scala-user@googlegroups.com
> Betreff: Re: [scala-user] compile speed

> One of my subprojects has about 7100 locs and compiles on cold-start
>
> compile:
> [mkdir] Created dir: \build\classes
> [scalac] Compiling 67 source files to \build\classes
> [stopwatch] [compile.timer: 24.616 sec]
>
> java = Java HotSpot(TM) Client VM 1.6.0_29
> java args = -Xms256m -Xss1M -Xmx768M -XX:MaxPermSize=256m
> -XX:+UseConcMarkSweepGC
>
> On an Intel P8700 (2.5Ghz), 3GB, Intel SSD (1.gen).
>
> For a general project try
> http://www.scala-lang.org/downloads/distrib/files/scala-2.9.1.final-sour....
> The results are:
>
> [stopwatch] [locker.lib.timer: 3:16.018 sec]
> [stopwatch] [locker.comp.timer: 3:21.567 sec]
> [stopwatch] [quick.lib.timer: 3:24.188 sec]
> [stopwatch] [quick.comp.timer: 2:56.572 sec]
> [stopwatch] [quick.plugins.timer: 15.006 sec]
> [stopwatch] [quick.scalap.timer: 17.960 sec]
> [stopwatch] [quick.partest.timer: 14.229 sec]
> Total time: 14 minutes 17 seconds
>
>
> On 15 December 2011 03:58, S Wade wrote:
>
> > Hi all,
> >
> > I'm noticing that my project (3,364 lines of code including comments)
> > compiles (with either scalac or fsc) very slowly in general and
> > particularly slowly on networked filesystems (the situation is even
> worse
> > with -optimise). Some details are listed below. Is this expected? I'm
> > running scala 2.9.1.final on a intel i7 linux box.
> >
> > thanks,
> > wade
> >
> >
> > Local filesystem
> > ============
> >
> > local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real 1m18.409s
> > user 1m39.777s
> > sys 0m3.909s
> >
> > local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real 1m3.793s
> > user 0m0.608s
> > sys 0m0.154s
> >
> > local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real 0m34.715s
> > user 0m0.566s
> > sys 0m0.117s
> >
> > local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> >
> > real 0m17.423s
> > user 0m0.569s
> > sys 0m0.123s
> >
> > Networked filesystem
> > ================
> >
> > nfs$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real 3m30.669s
> > user 2m28.289s
> > sys 0m2.392s
> >
> > nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
> >
> > real 4m7.908s
> > user 0m0.400s
> > sys 0m0.092s
> >
> > nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
> >
> > real 1m37.953s
> > user 0m0.388s
> > sys 0m0.084s
> >
> > nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
> >
> > real 1m6.757s
> > user 0m0.328s
> > sys 0m0.044s
> >
> >

lol!

i always imagine the compiler doing really awesome and complex stuff while compiling to kill the time

-------- Original-Nachricht --------
> Datum: Thu, 15 Dec 2011 17:12:02 +0100
> Von: Trond Olsen <trond@steinbit.org>
> An: S Wade <swadenator@gmail.com>
> CC: scala-user@googlegroups.com
> Betreff: Re: [scala-user] compile speed

> One of my subprojects has about 7100 locs and compiles on cold-start
>
> compile:
>     [mkdir] Created dir: \build\classes
>    [scalac] Compiling 67 source files to \build\classes
> [stopwatch] [compile.timer: 24.616 sec]
>
> java = Java HotSpot(TM) Client VM 1.6.0_29
> java args = -Xms256m -Xss1M -Xmx768M -XX:MaxPermSize=256m
> -XX:+UseConcMarkSweepGC
>
> On an Intel P8700 (2.5Ghz), 3GB, Intel SSD (1.gen).
>
> For a general project try
> http://www.scala-lang.org/downloads/distrib/files/scala-2.9.1.final-sources.tgz.
> The results are:
>
> [stopwatch] [locker.lib.timer: 3:16.018 sec]
> [stopwatch] [locker.comp.timer: 3:21.567 sec]
> [stopwatch] [quick.lib.timer: 3:24.188 sec]
> [stopwatch] [quick.comp.timer: 2:56.572 sec]
> [stopwatch] [quick.plugins.timer: 15.006 sec]
> [stopwatch] [quick.scalap.timer: 17.960 sec]
> [stopwatch] [quick.partest.timer: 14.229 sec]
> Total time: 14 minutes 17 seconds
>
>
> On 15 December 2011 03:58, S Wade <swadenator@gmail.com> wrote:
>
> > Hi all,
> >
> > I'm noticing that my project (3,364 lines of code including comments)
> > compiles (with either scalac or fsc) very slowly in general and
> > particularly slowly on networked filesystems (the situation is even
> worse
> > with -optimise).  Some details are listed below.  Is this expected?  I'm
> > running scala 2.9.1.final on a intel i7 linux box.
> >
> > thanks,
> > wade
> >
> >
> > Local filesystem
> > ============
> >
> > local$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real    1m18.409s
> > user    1m39.777s
> > sys     0m3.909s
> >
> > local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real    1m3.793s
> > user    0m0.608s
> > sys     0m0.154s
> >
> > local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real    0m34.715s
> > user    0m0.566s
> > sys     0m0.117s
> >
> > local$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> >
> > real    0m17.423s
> > user    0m0.569s
> > sys     0m0.123s
> >
> > Networked filesystem
> > ================
> >
> > nfs$ time scalac -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> > src/*.scala
> >
> > real    3m30.669s
> > user    2m28.289s
> > sys     0m2.392s
> >
> > nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
> >
> > real    4m7.908s
> > user    0m0.400s
> > sys     0m0.092s
> >
> > nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
> >
> > real    1m37.953s
> > user    0m0.388s
> > sys     0m0.084s
> >
> > nfs$ time fsc -cp lib/mongo-2.6.5.jar:Utils/Utils.jar -d build
> src/*.scala
> >
> > real    1m6.757s
> > user    0m0.328s
> > sys     0m0.044s
> >
> >

I'm using ANT_OPTS="-Xms1536M -Xmx4096M -Xss2M -XX:MaxPermSize=512M -XX:+UseParallelGC -XX:ReservedCodeCacheSize=128m"

thanks!

On 17/12/2011 18:16, Christian Parker wrote:
> Dennis, how do you use the ramdrive to improve compile time? Thanks!
>
> CP
A long time ago, I wrote an article explaining how to set-up a ramdisk
on Linux:

http://blog.normation.com/2010/07/27/set-up-eclipse-workspace-in-ram/

Cheers,

On 17/12/2011 20:05, Christian Parker wrote:
> Hmm, turns out to only save a couple seconds on a 1 minute 30
> second(ish) build.
>
> Geez, does the whole JVM need to run from a ramdisk?
>

The compiler is more CPU intensive than I/O intensive, so it may
explains what you are seeing.

Personnaly, I stopped using ramdisk. It seems that Linux is quite good
for putting things into caches, and eclipse 3.7 seems to be less bad
than previous one on that task, so all in all, the burden to have 25%
less RAM and don't forget to synchronize things (especially on non usual
tasks) was too expensive compared to the gain.

Thanks,

Am 17.12.2011 18:16, schrieb Christian Parker:
> Dennis, how do you use the ramdrive to improve compile time? Thanks!
>
> CP
it's my output directory. it's been ages since i used a regular hdd.

Here is an equivalent for Intellij, (require gksudo). Scala is not the
only language with a slow compile.

% cat ~/bin/startidea
#!/usr/bin/env zsh

~/bin/intelij10.tmpfs.optimised.sh
export IDEA_HOME="/big/idea-IU-111.69"
cd ${IDEA_HOME}/bin
./idea.sh &> $tmpfile

% cat ~/bin/intelij10.tmpfs.optimised.sh
#!/bin/sh

export MOUNTSIZE=600m

mkdir -p ~/.IntelliJIdea10/system/caches
mkdir -p ~/.IntelliJIdea10/system/index

#lets check to see if intellij cache directories are symbolic links to
my tmpfs
test -h ~/.IntelliJIdea10/system/index \
-a -d ~/.IntelliJIdea10/system/index \
-a -h ~/.IntelliJIdea10/system/caches \
-a -d ~/.IntelliJIdea10/system/caches

if [ $? -eq 0 ]
then
echo "found existing caches and index, will reuse..."
else
echo "didn't find, gotta create"
tmpdirname="/tmp/intellij-$(cat /dev/urandom | env LC_CTYPE=C tr -
cd 'a-f0-9' | head -c 32)"
mkdir $tmpdirname
gksudo -u root --message "mount tmpfs" "mount -t tmpfs tmpfs
$tmpdirname -o size=${MOUNTSIZE}"
#sudo mount -t tmpfs tmpfs $tmpdirname -o size=300m
rm -Rf ~/.IntelliJIdea10/system/index
rm -Rf ~/.IntelliJIdea10/system/caches
mkdir -p $tmpdirname/idea/caches
mkdir -p $tmpdirname/idea/index
ln -s $tmpdirname/idea/caches ~/.IntelliJIdea10/system/caches
ln -s $tmpdirname/idea/index ~/.IntelliJIdea10/system/index
fi