Why Scala actors 15-20 times slower compare to Jetlang and/or Groovy++ messaging

reply-to-all-fail

Here's some somewhat dated analysis:L
http://erikengbrecht.blogspot.com/2009/02/scala-actors-versus-exceptionblob.html

On Wed, Feb 24, 2010 at 10:30 AM, HHB <hubaghdadi@yahoo.ca> wrote:

Hey,
I got this article and I would like to hear your opinions.
http://groovy.dzone.com/articles/why-scala-actors-15-20-times
I don't mean to start any war, I just want to hear the professional peoples.
Thanks.
--
View this message in context: http://old.nabble.com/Why-Scala-actors-15-20-times-slower-compare-to-Jetlang-and-or-Groovy%2B%2B-messaging-tp27714232p27714232.html
Sent from the Scala - User mailing list archive at Nabble.com.

--
http://erikengbrecht.blogspot.com/

Using loop/react really adds some overhead to message processing. All this benchmark is testing is pure message passing, which isn't really the expected usage for actors, particularly those using loop/react.   Generally speaking, actors that are message-heavy are better implemented with while(true)/receive, at the expense of hogging a thread, whereas actors who do non-trivial work -- and, therefore, _can't_ be message-heavy -- use loop/react, which trades some overhead in message passing in exchange for not hogging threads. I understand the latter are even further optimized with Reactors in Scala 2.8, with some further restrictions.   Given their setup, involving thousands of actors doing nothing but passing messages, there really isn't any good fit, though replacing "loop" with "while(true)" ought to speed up a bit. I'm running this thing right now from my work notebook, but it's taking a long, long time.

On Wed, Feb 24, 2010 at 12:30 PM, HHB <hubaghdadi@yahoo.ca> wrote:

Hey,
I got this article and I would like to hear your opinions.
http://groovy.dzone.com/articles/why-scala-actors-15-20-times
I don't mean to start any war, I just want to hear the professional peoples.
Thanks.
--
View this message in context: http://old.nabble.com/Why-Scala-actors-15-20-times-slower-compare-to-Jetlang-and-or-Groovy%2B%2B-messaging-tp27714232p27714232.html
Sent from the Scala - User mailing list archive at Nabble.com.

--
Daniel C. Sobral

I travel to the future all the time.

Daniel Sobral wrote:
> Using loop/react really adds some overhead to message processing. All
> this benchmark is testing is pure message passing, which isn't really
> the expected usage for actors, particularly those using loop/react.
>
> Generally speaking, actors that are message-heavy are better implemented
> with while(true)/receive, at the expense of hogging a thread, whereas
> actors who do non-trivial work -- and, therefore, _can't_ be
> message-heavy -- use loop/react, which trades some overhead in message
> passing in exchange for not hogging threads. I understand the latter are
> even further optimized with Reactors in Scala 2.8, with some further
> restrictions.

Indeed, I expect this benchmark to run much, much faster using Reactors
in Scala 2.8 (even normal actors).

I noticed that the benchmark creates actors using `actor`. This adds
some overhead since it means calls to `react` have to go through
`Actor.self` which requires looking up a thread-local variable-- slower
than just `this.react`. So, extending `Actor` and implementing `act`
would already help.

Also, I wouldn't be surprised if `Actor` instances (which `actor`
creates) had a different feature set than the channels of Jetlang or
Groovy++.
One example that also costs performance are implicitly transmitted
`sender`s. This is one feature that can be avoided in Scala 2.8 by
simply using a `Reactor`. Also, control-flow combinators such as `loop`
are much cheaper for `Reactor`s, since they don't require look-ups of
thread-local variables (which means using them requires `Reactor.this`
to be in scope, which is in contrast to `Actor`s that are as dynamic as
Erlang in that respect).

On Scala 2.8 both Actors and Reactors can be run on the new
`ForkJoinScheduler` (provided we are running on a Sun 1.6/1.7 JVM),
which is also a big advantage. However, in Beta1 this is not enabled by
default for Reactors (enabled for Actors). One can enable it as follows:

object Test {
lazy val fjSched = {
val s = new scala.actors.scheduler.ForkJoinScheduler
s.start()
s
}
}

trait FJReactor extends scala.actors.Reactor {
override def scheduler = Test.fjSched
}

We are planning to make this the default scheduler also for Reactors.

Anyways, only so much for now.

Cheers,
Philipp

> Given their setup, involving thousands of actors doing nothing but
> passing messages, there really isn't any good fit, though replacing
> "loop" with "while(true)" ought to speed up a bit. I'm running this
> thing right now from my work notebook, but it's taking a long, long time.
>
> On Wed, Feb 24, 2010 at 12:30 PM, HHB > wrote:
>
>
> Hey,
> I got this article and I would like to hear your opinions.
> http://groovy.dzone.com/articles/why-scala-actors-15-20-times
> I don't mean to start any war, I just want to hear the professional
> peoples.
> Thanks.
> --
> View this message in context:
> http://old.nabble.com/Why-Scala-actors-15-20-times-slower-compare-to-Jet...
> Sent from the Scala - User mailing list archive at Nabble.com.
>
>
>
>

On Wed, Feb 24, 2010 at 12:05 PM, Philipp Haller <philipp.haller@epfl.ch> wrote:

Daniel Sobral wrote:

  Generally speaking, actors that are message-heavy are better implemented with while(true)/receive, at the expense of hogging a thread, whereas actors who do non-trivial work -- and, therefore, _can't_ be message-heavy -- use loop/react, which trades some overhead in message passing in exchange for not hogging threads.

Fiber rings have so many threads that the overhead is impossibly large.  I scaled back the problem by 10x number of in-flight messages, which gives a nice 4x speedup over the original implementation (on a 4-core machine) with while(true)/receive.  At the normal number of messages, it's still about a 3x speedup _except_ the normal scheduler hangs on 500 messages--using the ForkJoinScheduler you need to increase the max pool size to higher than the number of in-flight messages (on 2.8.0 Beta1); max pool of 512 works with 500 in-flight messages.

This seems like a *very* bad idea for a default scheduler--to just hang if you have too many concurrent messages, rather than just have degrading performance.

(Whichever scheduler is used for reactors also chokes by default here.)

I understand the latter are even further optimized with Reactors in Scala 2.8, with some further restrictions.

Indeed, I expect this benchmark to run much, much faster using Reactors in Scala 2.8 (even normal actors).

I tested this also.  It's also about 3x faster on my system (10s vs. 30s).  I wouldn't call that "much much", though a factor of two is nice.
 

I noticed that the benchmark creates actors using `actor`. This adds some overhead since it means calls to `react` have to go through `Actor.self` which requires looking up a thread-local variable-- slower than just `this.react`. So, extending `Actor` and implementing `act` would already help.

This is negligible.
 

On Scala 2.8 both Actors and Reactors can be run on the new `ForkJoinScheduler` (provided we are running on a Sun 1.6/1.7 JVM), which is also a big advantage. However, in Beta1 this is not enabled by default for Reactors (enabled for Actors). One can enable it as follows:

object Test {
 lazy val fjSched = {
   val s = new scala.actors.scheduler.ForkJoinScheduler
   s.start()
   s
 }
}

trait FJReactor extends scala.actors.Reactor {
 override def scheduler = Test.fjSched
}

We are planning to make this the default scheduler also for Reactors.

This is a bad idea.  With the default pool size, this is 2x slower than the current scheduler for both Reactors and Actors in this test (for loop/react; as I mentioned above, both the default for FJ and whatever (thread pool?) is used for Reactor hangs idle if used with too many concurrent messages with while(true)/receive; 2.7.7 fails in this manner also).

  --Rex
 

I noticed that the benchmark creates actors using `actor`. This adds some overhead since it means calls to `react` have to go through `Actor.self` which requires looking up a thread-local variable-- slower than just `this.react`. So, extending `Actor` and implementing `act` would already help.

This is negligible.

I wouldn't be so sure.  The call itself may not be that expensive, but I've noticed that relatively small changes (particularly increasing the length of a call chain) in code can drastically effect the performance of code that relies on exceptions for control flow.  I think it has to do with the JVM's ability (or willingness) to inline methods.

On Wed, Feb 24, 2010 at 3:06 PM, Rex Kerr <ichoran@gmail.com> wrote:

On Wed, Feb 24, 2010 at 12:05 PM, Philipp Haller <philipp.haller@epfl.ch> wrote:

Daniel Sobral wrote:

  Generally speaking, actors that are message-heavy are better implemented with while(true)/receive, at the expense of hogging a thread, whereas actors who do non-trivial work -- and, therefore, _can't_ be message-heavy -- use loop/react, which trades some overhead in message passing in exchange for not hogging threads.

Fiber rings have so many threads that the overhead is impossibly large.  I scaled back the problem by 10x number of in-flight messages, which gives a nice 4x speedup over the original implementation (on a 4-core machine) with while(true)/receive.  At the normal number of messages, it's still about a 3x speedup _except_ the normal scheduler hangs on 500 messages--using the ForkJoinScheduler you need to increase the max pool size to higher than the number of in-flight messages (on 2.8.0 Beta1); max pool of 512 works with 500 in-flight messages.

This seems like a *very* bad idea for a default scheduler--to just hang if you have too many concurrent messages, rather than just have degrading performance.

(Whichever scheduler is used for reactors also chokes by default here.)

I understand the latter are even further optimized with Reactors in Scala 2.8, with some further restrictions.

Indeed, I expect this benchmark to run much, much faster using Reactors in Scala 2.8 (even normal actors).

I tested this also.  It's also about 3x faster on my system (10s vs. 30s).  I wouldn't call that "much much", though a factor of two is nice.
 

I noticed that the benchmark creates actors using `actor`. This adds some overhead since it means calls to `react` have to go through `Actor.self` which requires looking up a thread-local variable-- slower than just `this.react`. So, extending `Actor` and implementing `act` would already help.

This is negligible.
 

On Scala 2.8 both Actors and Reactors can be run on the new `ForkJoinScheduler` (provided we are running on a Sun 1.6/1.7 JVM), which is also a big advantage. However, in Beta1 this is not enabled by default for Reactors (enabled for Actors). One can enable it as follows:

object Test {
 lazy val fjSched = {
   val s = new scala.actors.scheduler.ForkJoinScheduler
   s.start()
   s
 }
}

trait FJReactor extends scala.actors.Reactor {
 override def scheduler = Test.fjSched
}

We are planning to make this the default scheduler also for Reactors.

This is a bad idea.  With the default pool size, this is 2x slower than the current scheduler for both Reactors and Actors in this test (for loop/react; as I mentioned above, both the default for FJ and whatever (thread pool?) is used for Reactor hangs idle if used with too many concurrent messages with while(true)/receive; 2.7.7 fails in this manner also).

  --Rex
 

--
http://erikengbrecht.blogspot.com/

Well, Rex, the problem in making decisions about scheduler and such with this example, is that it does nothing _besides_ send and receive messages. While, of course, that is important in its own right, I'd argue that in almost all cases at least some actors will be doing non-trivial work.

On Wed, Feb 24, 2010 at 5:06 PM, Rex Kerr <ichoran@gmail.com> wrote:

On Wed, Feb 24, 2010 at 12:05 PM, Philipp Haller <philipp.haller@epfl.ch> wrote:

Daniel Sobral wrote:

  Generally speaking, actors that are message-heavy are better implemented with while(true)/receive, at the expense of hogging a thread, whereas actors who do non-trivial work -- and, therefore, _can't_ be message-heavy -- use loop/react, which trades some overhead in message passing in exchange for not hogging threads.

Fiber rings have so many threads that the overhead is impossibly large.  I scaled back the problem by 10x number of in-flight messages, which gives a nice 4x speedup over the original implementation (on a 4-core machine) with while(true)/receive.  At the normal number of messages, it's still about a 3x speedup _except_ the normal scheduler hangs on 500 messages--using the ForkJoinScheduler you need to increase the max pool size to higher than the number of in-flight messages (on 2.8.0 Beta1); max pool of 512 works with 500 in-flight messages.

This seems like a *very* bad idea for a default scheduler--to just hang if you have too many concurrent messages, rather than just have degrading performance.

(Whichever scheduler is used for reactors also chokes by default here.)

I understand the latter are even further optimized with Reactors in Scala 2.8, with some further restrictions.

Indeed, I expect this benchmark to run much, much faster using Reactors in Scala 2.8 (even normal actors).

I tested this also.  It's also about 3x faster on my system (10s vs. 30s).  I wouldn't call that "much much", though a factor of two is nice.
 

I noticed that the benchmark creates actors using `actor`. This adds some overhead since it means calls to `react` have to go through `Actor.self` which requires looking up a thread-local variable-- slower than just `this.react`. So, extending `Actor` and implementing `act` would already help.

This is negligible.
 

On Scala 2.8 both Actors and Reactors can be run on the new `ForkJoinScheduler` (provided we are running on a Sun 1.6/1.7 JVM), which is also a big advantage. However, in Beta1 this is not enabled by default for Reactors (enabled for Actors). One can enable it as follows:

object Test {
 lazy val fjSched = {
   val s = new scala.actors.scheduler.ForkJoinScheduler
   s.start()
   s
 }
}

trait FJReactor extends scala.actors.Reactor {
 override def scheduler = Test.fjSched
}

We are planning to make this the default scheduler also for Reactors.

This is a bad idea.  With the default pool size, this is 2x slower than the current scheduler for both Reactors and Actors in this test (for loop/react; as I mentioned above, both the default for FJ and whatever (thread pool?) is used for Reactor hangs idle if used with too many concurrent messages with while(true)/receive; 2.7.7 fails in this manner also).

  --Rex
 

--
Daniel C. Sobral

I travel to the future all the time.

I would normally think so, except that there was that recent thread about Erlang-style I/O actors.  Any normal-size "println"-type statement actually takes _less_ time than the message pass.  It's just a bad idea to divide up your work into pieces that small with the concurrency facilities that we have.  (Simple tests of this give me a ~2x slowdown on printing strings of numbers or small classes which were generated using functional methods.)

So it seems to me that if you want granularity on the size of an single-line output statement, you want better message-sending.  (I don't know how to do it in Java, incidentally; I'm not sure what Groovy++ and Jetlang are doing.)

If you don't care about that, then I agree, the relatively high overhead is not much of a concern.

  --Rex

On Wed, Feb 24, 2010 at 3:23 PM, Daniel Sobral <dcsobral@gmail.com> wrote:

Well, Rex, the problem in making decisions about scheduler and such with this example, is that it does nothing _besides_ send and receive messages. While, of course, that is important in its own right, I'd argue that in almost all cases at least some actors will be doing non-trivial work.

You could try to buffer messages before actually sending them and then let the actor drain its message queue more eagerly.

-Stefan

2010/2/25 Rex Kerr <ichoran@gmail.com>

I would normally think so, except that there was that recent thread about Erlang-style I/O actors.  Any normal-size "println"-type statement actually takes _less_ time than the message pass.  It's just a bad idea to divide up your work into pieces that small with the concurrency facilities that we have.  (Simple tests of this give me a ~2x slowdown on printing strings of numbers or small classes which were generated using functional methods.)

So it seems to me that if you want granularity on the size of an single-line output statement, you want better message-sending.  (I don't know how to do it in Java, incidentally; I'm not sure what Groovy++ and Jetlang are doing.)

If you don't care about that, then I agree, the relatively high overhead is not much of a concern.

  --Rex

On Wed, Feb 24, 2010 at 3:23 PM, Daniel Sobral <dcsobral@gmail.com> wrote:

Well, Rex, the problem in making decisions about scheduler and such with this example, is that it does nothing _besides_ send and receive messages. While, of course, that is important in its own right, I'd argue that in almost all cases at least some actors will be doing non-trivial work.

The actor library can't let messages queue up before scheduling the actor because it doesn't know how many messages could safely be queued without introducing some weird starvation condition.  A lot of actors have to send a message before receiving their next message.

Having the actor hold the thread as long as it has messages to process can be an effective optimization, especially if you introduce a construct that prevents a control exception from being needed between messages (e.g. the reactWhile construct in my blog that I linked to earlier in this thread).  However, a naive implementation of such a construct (e.g. mine) introduces fairness issues, because it allows a single actor to hog a thread.  So in order to preserve fairness an actor framework that performs such an optimization would need a smarter scheduler that could kick the actor off the thread if it was needed.

On Thu, Feb 25, 2010 at 3:18 AM, Stefan Langer <mailtolanger@googlemail.com> wrote:

You could try to buffer messages before actually sending them and then let the actor drain its message queue more eagerly.

-Stefan

2010/2/25 Rex Kerr <ichoran@gmail.com>

I would normally think so, except that there was that recent thread about Erlang-style I/O actors.  Any normal-size "println"-type statement actually takes _less_ time than the message pass.  It's just a bad idea to divide up your work into pieces that small with the concurrency facilities that we have.  (Simple tests of this give me a ~2x slowdown on printing strings of numbers or small classes which were generated using functional methods.)

So it seems to me that if you want granularity on the size of an single-line output statement, you want better message-sending.  (I don't know how to do it in Java, incidentally; I'm not sure what Groovy++ and Jetlang are doing.)

If you don't care about that, then I agree, the relatively high overhead is not much of a concern.

  --Rex

On Wed, Feb 24, 2010 at 3:23 PM, Daniel Sobral <dcsobral@gmail.com> wrote:

Well, Rex, the problem in making decisions about scheduler and such with this example, is that it does nothing _besides_ send and receive messages. While, of course, that is important in its own right, I'd argue that in almost all cases at least some actors will be doing non-trivial work.

--
http://erikengbrecht.blogspot.com/