Speed of mutable vs immutable objects.

Hi Chris!

What JVM settings did you use?

On Tue, Feb 9, 2010 at 8:55 PM, Chris Dew <cmsdew@googlemail.com> wrote:

I've just done a little informal benchmark to see if purely functional
objects can compete with the speed of mutable objects, in Scala.

http://www.finalcog.com/speed-mutable-immutable-scala

Creating new immutable objects (instead of mutating mutable objects)
is one-to-two orders of magnitude slower.  (To put this in
perspective, you can still make 10's of millions of immutable objects
per second.)

Regards,

Chris.

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

Just the defaults for java-6-openjdk on Ubuntu 9.10 AMD64.

java -Xmx256M -Xms16M
-Xbootclasspath/a:/usr/share/java/scala-library.jar:/usr/share/java/scala-compiler.jar:/usr/share/java/jline.jar
-classpath .::/usr/share/java/scala-library.jar:/usr/share/java/scala-compiler.jar:/usr/share/java/jline.jar
com.finalcog.immutability_test

It takes a fraction of a second for the first two billion method calls
(on the mutable object) and a little less than a second for the 200M
immutable object instantiations.

Would you expect a different result?

All the best,

Chris.

On 9 February 2010 19:58, Viktor Klang wrote:
> Hi Chris!
>
> What JVM settings did you use?
>
> On Tue, Feb 9, 2010 at 8:55 PM, Chris Dew wrote:
>>
>> I've just done a little informal benchmark to see if purely functional
>> objects can compete with the speed of mutable objects, in Scala.
>>
>> http://www.finalcog.com/speed-mutable-immutable-scala
>>
>> Creating new immutable objects (instead of mutating mutable objects)
>> is one-to-two orders of magnitude slower.  (To put this in
>> perspective, you can still make 10's of millions of immutable objects
>> per second.)
>>
>> Regards,
>>
>> Chris.
>
>
>
> --
> Viktor Klang
> | "A complex system that works is invariably
> | found to have evolved from a simple system
> | that worked." - John Gall
>
> Akka - the Actor Kernel: Akkasource.org
> Twttr: twitter.com/viktorklang
>

Of course, a tight loop that simply instantiates gazillions of objects is going to perform much slower than one that simply changes two Ints.

It seems to me that the "immutable is slow" argument comes up fairly often, and often with a rather narrow view of things. The value to be gained in immutability is not to be found in simply churning out new objects and throwing away the old (though the JVM is getting much better, if not perfect yet, at dealing with this). In my mind, the gains from immutability are threefold:

1) Persistence. This is a key characteristic of immutable data structures: you can safely reuse/share/etc older "versions" of objects while continuing to "mutate" (ie, create new versions) them. For instance, in an immutable list, adding an item to a List does indeed create a new List object, but that new list shares all existing object references with the old one besides the single new head reference.

2) Concurrency. Since objects don't change, you don't even have to think twice about sharing them between threads. Look ma, no locks! Not only does this make (the very tricky area of) concurrency vastly simpler to comprehend, but usually offers nice performance gains. It would be very interesting to see how your metrics change if you modify your test so that many threads are all trying to (safely) update a small number of points at the same time....

3) Lastly, debugging. Since I've made a concerted effort towards a more functional style (including immutability), I've found my code much easier to reason about, and much more tractable to debugging: when something has the wrong value, there's really only one place to look. This isn't, of course, a measurable performance gain in the code itself, but it most definitely is a performance gain in getting the code written, and maintaining it thereafter.

I'm most certainly not trying to discredit anyone's findings, or argue for rigidly sticking to immutability. There is most certainly a time and place for both approaches, and both bring different offerings to the table. As an example, the scala compiler (and libraries) itself make use of both approaches (and shades in between).

I really just wanted to point out that the benefits of immutability can't always be measured with (endTime - startTime)... :)

- Colin

Sounds very much like what I've seen on similar tests; integer operations take a fraction of a nanosecond, while creation of small objects eats up about 4-10 ns apiece.  In some cases the object creation can be eliminated entirely (e.g. when the object is only being used transiently with an implicit conversion to effectively add a method), but for 2D integer vectors you're seeing the same thing I did.  (And I've tried pretty much every JVM option with essentially no change.)

  --Rex

On Tue, Feb 9, 2010 at 3:12 PM, Chris Dew <cmsdew@googlemail.com> wrote:

It takes a fraction of a second for the first two billion method calls
(on the mutable object) and a little less than a second for the 200M
immutable object instantiations.

Would you expect a different result?

Immutable data structures advantages results from not having to constantly copy them when passing them around, and reusing as much of an old version as possible   For an example of the later, a "shape" could be composed of points and offset, so "move" would produce a new offset and reuse the set of points.   If your code never needs to copy mutable data before passing it on, then the benchmark you provided is a sure indication that you should keep to mutable data. If not, then that benchmark doesn't have much value.

On Tue, Feb 9, 2010 at 5:55 PM, Chris Dew <cmsdew@googlemail.com> wrote:

I've just done a little informal benchmark to see if purely functional
objects can compete with the speed of mutable objects, in Scala.

http://www.finalcog.com/speed-mutable-immutable-scala

Creating new immutable objects (instead of mutating mutable objects)
is one-to-two orders of magnitude slower.  (To put this in
perspective, you can still make 10's of millions of immutable objects
per second.)

Regards,

Chris.

--
Daniel C. Sobral

I travel to the future all the time.

On Tue, Feb 9, 2010 at 9:12 PM, Chris Dew <cmsdew@googlemail.com> wrote:

Just the defaults for java-6-openjdk on Ubuntu 9.10 AMD64.

java -Xmx256M -Xms16M
-Xbootclasspath/a:/usr/share/java/scala-library.jar:/usr/share/java/scala-compiler.jar:/usr/share/java/jline.jar
-classpath .::/usr/share/java/scala-library.jar:/usr/share/java/scala-compiler.jar:/usr/share/java/jline.jar
com.finalcog.immutability_test

Any difference if you use: -XX:-DoEscapeAnalysis -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

It takes a fraction of a second for the first two billion method calls
(on the mutable object) and a little less than a second for the 200M
immutable object instantiations.

Would you expect a different result?

All the best,

Chris.

On 9 February 2010 19:58, Viktor Klang <viktor.klang@gmail.com> wrote:
> Hi Chris!
>
> What JVM settings did you use?
>
> On Tue, Feb 9, 2010 at 8:55 PM, Chris Dew <cmsdew@googlemail.com> wrote:
>>
>> I've just done a little informal benchmark to see if purely functional
>> objects can compete with the speed of mutable objects, in Scala.
>>
>> http://www.finalcog.com/speed-mutable-immutable-scala
>>
>> Creating new immutable objects (instead of mutating mutable objects)
>> is one-to-two orders of magnitude slower.  (To put this in
>> perspective, you can still make 10's of millions of immutable objects
>> per second.)
>>
>> Regards,
>>
>> Chris.
>
>
>
> --
> Viktor Klang
> | "A complex system that works is invariably
> | found to have evolved from a simple system
> | that worked." - John Gall
>
> Akka - the Actor Kernel: Akkasource.org
> Twttr: twitter.com/viktorklang
>



--

http://www.finalcog.com/

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

Yes, did you use the -server option at least?

In practice, I have found the difference much much less than an order of
magnitude (if using -server):

I did some comparisions some time ago between Java code that was written
with mutable objects and Scala code that was written with immutable ones:
http://www.box2d.org/forum/viewtopic.php?p=16297#p16297
I'm not sure if I've done the benchmarks the best way (I did let it warm
up a bit), but it's real code. Well, not extremely real, just a demo
scene of a 2D physics engine.

The difference is not that big, although the code is not exactly
identical (the Scala version is a port of the Java version, but some
things have been changed around and different types of collections used)

Erkki

Viktor Klang wrote:
> Hi Chris!
>
> What JVM settings did you use?
>
> On Tue, Feb 9, 2010 at 8:55 PM, Chris Dew > wrote:
>
> I've just done a little informal benchmark to see if purely functional
> objects can compete with the speed of mutable objects, in Scala.
>
> http://www.finalcog.com/speed-mutable-immutable-scala
>
> Creating new immutable objects (instead of mutating mutable objects)
> is one-to-two orders of magnitude slower. (To put this in
> perspective, you can still make 10's of millions of immutable objects
> per second.)
>
> Regards,
>
> Chris.
>
>
>
>

That benchmark is too small for those options to have any meaning. A warm up would be required before doing the measure.

On Tue, Feb 9, 2010 at 6:21 PM, Viktor Klang <viktor.klang@gmail.com> wrote:

On Tue, Feb 9, 2010 at 9:12 PM, Chris Dew <cmsdew@googlemail.com> wrote:

Just the defaults for java-6-openjdk on Ubuntu 9.10 AMD64.

java -Xmx256M -Xms16M
-Xbootclasspath/a:/usr/share/java/scala-library.jar:/usr/share/java/scala-compiler.jar:/usr/share/java/jline.jar
-classpath .::/usr/share/java/scala-library.jar:/usr/share/java/scala-compiler.jar:/usr/share/java/jline.jar
com.finalcog.immutability_test

Any difference if you use: -XX:-DoEscapeAnalysis -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

It takes a fraction of a second for the first two billion method calls
(on the mutable object) and a little less than a second for the 200M
immutable object instantiations.

Would you expect a different result?

All the best,

Chris.

On 9 February 2010 19:58, Viktor Klang <viktor.klang@gmail.com> wrote:
> Hi Chris!
>
> What JVM settings did you use?
>
> On Tue, Feb 9, 2010 at 8:55 PM, Chris Dew <cmsdew@googlemail.com> wrote:
>>
>> I've just done a little informal benchmark to see if purely functional
>> objects can compete with the speed of mutable objects, in Scala.
>>
>> http://www.finalcog.com/speed-mutable-immutable-scala
>>
>> Creating new immutable objects (instead of mutating mutable objects)
>> is one-to-two orders of magnitude slower.  (To put this in
>> perspective, you can still make 10's of millions of immutable objects
>> per second.)
>>
>> Regards,
>>
>> Chris.
>
>
>
> --
> Viktor Klang
> | "A complex system that works is invariably
> | found to have evolved from a simple system
> | that worked." - John Gall
>
> Akka - the Actor Kernel: Akkasource.org
> Twttr: twitter.com/viktorklang
>



--

http://www.finalcog.com/

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

--
Daniel C. Sobral

I travel to the future all the time.