Profiling Actor-based server: Lots of time in FJTaskRunner

On Sat, Feb 14, 2009 at 6:13 PM, John Kalucki wrote:
> I'm profiling a 2.7.3 server that uses a modest number of Actors, some of
> which have dedicated threads, others yield. When under load, I notice nearly
> all of the cpu going to these sorts of stack traces:
>
> java.lang.Thread.yield(Thread.java:Unknown line)
> scala.actors.FJTaskRunner.scanWhileIdling(:Unknown
> line)
> scala.actors.FJTaskRunner.run(:Unknown line)
>
> and
>
> java.lang.Thread.setPriority0(Thread.java:Unknown line)
> java.lang.Thread.setPriority(Thread.java:1039)
> scala.actors.FJTaskRunner.scanWhileIdling(:Unknown
> line)
> scala.actors.FJTaskRunner.run(:Unknown line)
>
> I'm only vaguely familiar with the FJTaskRunner package. At first glance, tt
> appears that something is mis-tuned with how Actors are using this package.
> Nearly all of the samples, BTW, are in FJTaskRunner, or other Actor
> related-code, and few, if any, are doing direct application work. The
> application isn't performing particularly CPU intensive work, it's shuffling
> messages around.
>
> Is there anything that I can tune or look for that might be causing this
> behavior? Or am I looking at this in the wrong way?

I think that's called when a thread isn't doing anything, and wants to
give up it's CPU allotment for the moment. It's a bit strange that
it's using much CPU. Maybe your program is dominated by semi-idle
threads for some reason? I'm not too familiar with FJTaskRunner
either.

http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Thread.html#yield()

Cheers
Rich

On Fri, Feb 13, 2009 at 9:13 PM, John Kalucki <bulk@kalucki.com> wrote:

I'm profiling a 2.7.3 server that uses a modest number of Actors, some of
which have dedicated threads, others yield. When under load, I notice nearly
all of the cpu going to these sorts of stack traces:

       java.lang.Thread.yield(Thread.java:Unknown line)
       scala.actors.FJTaskRunner.scanWhileIdling(<Unknown Source>:Unknown
line)
       scala.actors.FJTaskRunner.run(<Unknown Source>:Unknown line)

and

       java.lang.Thread.setPriority0(Thread.java:Unknown line)
       java.lang.Thread.setPriority(Thread.java:1039)
       scala.actors.FJTaskRunner.scanWhileIdling(<Unknown Source>:Unknown
line)
       scala.actors.FJTaskRunner.run(<Unknown Source>:Unknown line)

I'm only vaguely familiar with the FJTaskRunner package. At first glance, tt
appears that something is mis-tuned with how Actors are using this package.
Nearly all of the samples, BTW, are in FJTaskRunner, or other Actor
related-code, and few, if any, are doing direct application work. The
application isn't performing particularly CPU intensive work, it's shuffling
messages around.

Is there anything that I can tune or look for that might be causing this
behavior? Or am I looking at this in the wrong way?

It's hard to tell.  Java profiling tools are good at dealing with stuff that lives in byte-code, but not so good with stuff that makes calls into native code.
You're going to see a lot of wall-clock time used in methods that yield their threads, but that does not translate into a lot of CPU utilization.
If you can put together an example of the message passing (e.g., create 1M Actors and randomly poke actors to message pass to other actors where the messages cascade for a couple of generations), I'll be happy to take a look at it after 2/26 (that's when we ship Lift 1.0 and the last chapter of Beginning Scala is due.)
You also might want to scare up a Solaris machine and look at things under dtrace.  That might give you a better idea if there's a real cost to yielding the threads. Finally, it might be useful to benchmark on the actual OS that you're running on.  Mac OS X's Java implementation is much different from Sun's Linux JVM, and to hear the things Linus has to say about Mach... well...
Hope this helps.

-John Kalucki

--
View this message in context: http://www.nabble.com/Profiling-Actor-based-server%3A-Lots-of-time-in-FJTaskRunner-tp22009334p22009334.html
Sent from the Scala - User mailing list archive at Nabble.com.

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

In this case, the JVM profiling tools are accurate, or rather, they are
accurate enough. FJTaskRunner.scanWhileIdling is indeed wasting copious
amounts of CPU. On an 8-core box, 3 CPUs are burned by the stack trace
described below.

If I tune the -Dactors.maxPoolSize down to ensure that excessive idle
threads aren't created, this issue is brought under control. My "unloaded",
but polling, application runs in 14% of a single CPU, approximately what I'd
expect. The JVM CPU profile changes accordingly.

If I get a moment, I hope to fiddle with the Actor library code in question
and see if I can pinpoint what the problem is. I suspect that the ratio and
duration of wait calls in FJTaskRunnerGroup.checkActive() to yield calls in
FJTaskRunner.scanWhileIdling() is inappropriate when several idle threads
are present. These ratios and constants should be tunable.

I also suspect that running on an 8 core box exacerbates the wasteful
behavior of scanWhileIdling, because 16 threads are created by default.
FJTaskScheduler2 creates two threads per core, which is a quite reasonable
default. But, if I really only need 10 (9 receive blocks, a few react
blocks, in this case), then 6 threads might be pounding through
scanWhileIdling. At first glance there doesn't seem to be any coordination
between idle threads. Perhaps they could coordinate via something as simple
as a semaphore that prevents more than 1 or 2 to scan, or perhaps by an
aggressive exponential backoff.

-John

David Pollak-4 wrote:
>
> On Fri, Feb 13, 2009 at 9:13 PM, John Kalucki wrote:
>
>>
>> I'm profiling a 2.7.3 server that uses a modest number of Actors, some of
>> which have dedicated threads, others yield. When under load, I notice
>> nearly
>> all of the cpu going to these sorts of stack traces:
>>
>> java.lang.Thread.yield(Thread.java:Unknown line)
>> scala.actors.FJTaskRunner.scanWhileIdling(:Unknown
>> line)
>> scala.actors.FJTaskRunner.run(:Unknown line)
>>
>> and
>>
>> java.lang.Thread.setPriority0(Thread.java:Unknown line)
>> java.lang.Thread.setPriority(Thread.java:1039)
>> scala.actors.FJTaskRunner.scanWhileIdling(:Unknown
>> line)
>> scala.actors.FJTaskRunner.run(:Unknown line)
>>
>> I'm only vaguely familiar with the FJTaskRunner package. At first glance,
>> tt
>> appears that something is mis-tuned with how Actors are using this
>> package.
>> Nearly all of the samples, BTW, are in FJTaskRunner, or other Actor
>> related-code, and few, if any, are doing direct application work. The
>> application isn't performing particularly CPU intensive work, it's
>> shuffling
>> messages around.
>>
>> Is there anything that I can tune or look for that might be causing this
>> behavior? Or am I looking at this in the wrong way?
>
>
> It's hard to tell. Java profiling tools are good at dealing with stuff
> that
> lives in byte-code, but not so good with stuff that makes calls into
> native
> code.
>
> You're going to see a lot of wall-clock time used in methods that yield
> their threads, but that does not translate into a lot of CPU utilization.
>
> If you can put together an example of the message passing (e.g., create 1M
> Actors and randomly poke actors to message pass to other actors where the
> messages cascade for a couple of generations), I'll be happy to take a
> look
> at it after 2/26 (that's when we ship Lift 1.0 and the last chapter of
> Beginning Scala is due.)
>
> You also might want to scare up a Solaris machine and look at things under
> dtrace. That might give you a better idea if there's a real cost to
> yielding the threads.
>
> Finally, it might be useful to benchmark on the actual OS that you're
> running on. Mac OS X's Java implementation is much different from Sun's
> Linux JVM, and to hear the things Linus has to say about Mach... well...
>
> Hope this helps.
>
>>
>>
>> -John Kalucki
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Profiling-Actor-based-server%3A-Lots-of-time-in-FJ...
>> Sent from the Scala - User mailing list archive at Nabble.com.
>>
>>
>
>

On 03/02/2009 11:39 AM, John Kalucki wrote:
> If I get a moment, I hope to fiddle with the Actor library code in question...
>
John,

If you have a bit of room to maneuver, you might like to have a look at
Erik Engbrecht's recent work on refactoring the actor library (starting
with
http://erikengbrecht.blogspot.com/2009/01/refactoring-scala-actors.html )

-0xe1a

FYI - My work still uses the same scheduler.  I've noticed the FJTaskScheduler2 is sometimes overly aggressive in spawning threads, especially if you have a lot of cores, but last time I poked at that particular piece of code it became too conservative with spawning threads and locked up.   I think it's relatively straight forward to use: http://www.scala-lang.org/docu/files/api/scala/actors/SchedulerAdapter.html   to wrap a java.util.concurrent executor: http://java.sun.com/j2se/1.5.0/docs/api/index.html?java/util/concurrent/ThreadPoolExecutor.html   ...and then you can gain a little more control over the behavior.

On Mon, Mar 2, 2009 at 2:47 PM, Alex Cruise <alex@cluonflux.com> wrote:

On 03/02/2009 11:39 AM, John Kalucki wrote:

If I get a moment, I hope to fiddle with the Actor library code in question...
 

John,

If you have a bit of room to maneuver, you might like to have a look at Erik Engbrecht's recent work on refactoring the actor library (starting with http://erikengbrecht.blogspot.com/2009/01/refactoring-scala-actors.html )

-0xe1a

--
http://erikengbrecht.blogspot.com/

Erik, Alex, John, et al,

i'm loathe to hijack this thread -- which is a good one -- but the experience with the lock up described below is really just the tip of the iceberg. Unless specific scheduling contracts and semantics can be worked out, a pluggable scheduling architecture is just asking for disaster. Worse, it means that a whole host of compiler support that could be provided to apps that use actor-based concurrency is null and void because those solutions (be they model-checking or types-based) will critically depend on specific ranges of scheduling semantics.

Actors and other higher level concurrency constructs are a major appeal of languages like scala. If they prove themselves to be "not ready for prime time" the impact on perception might not be contained to just that usage of the language.

Best wishes,

--greg

On Mon, Mar 2, 2009 at 11:55 AM, Erik Engbrecht <erik.engbrecht@gmail.com> wrote:

FYI - My work still uses the same scheduler.  I've noticed the FJTaskScheduler2 is sometimes overly aggressive in spawning threads, especially if you have a lot of cores, but last time I poked at that particular piece of code it became too conservative with spawning threads and locked up.   I think it's relatively straight forward to use: http://www.scala-lang.org/docu/files/api/scala/actors/SchedulerAdapter.html   to wrap a java.util.concurrent executor: http://java.sun.com/j2se/1.5.0/docs/api/index.html?java/util/concurrent/ThreadPoolExecutor.html   ...and then you can gain a little more control over the behavior.

On Mon, Mar 2, 2009 at 2:47 PM, Alex Cruise <alex@cluonflux.com> wrote:

On 03/02/2009 11:39 AM, John Kalucki wrote:

If I get a moment, I hope to fiddle with the Actor library code in question...
 

John,

If you have a bit of room to maneuver, you might like to have a look at Erik Engbrecht's recent work on refactoring the actor library (starting with http://erikengbrecht.blogspot.com/2009/01/refactoring-scala-actors.html )

-0xe1a

--
http://erikengbrecht.blogspot.com/

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105

+1 206.650.3740

http://biosimilarity.blogspot.com

Yes, it's very easy to cause lockups. Perhaps the configuration should
include "maxIdleThreads".

Erik Engbrecht wrote:
>
> FYI - My work still uses the same scheduler. I've noticed the
> FJTaskScheduler2 is sometimes overly aggressive in spawning threads,
> especially if you have a lot of cores, but last time I poked at that
> particular piece of code it became too conservative with spawning threads
> and locked up.
>
> I think it's relatively straight forward to use:
> http://www.scala-lang.org/docu/files/api/scala/actors/SchedulerAdapter.html
>
> to wrap a java.util.concurrent executor:
> http://java.sun.com/j2se/1.5.0/docs/api/index.html?java/util/concurrent/...
>
> ...and then you can gain a little more control over the behavior.
>
> On Mon, Mar 2, 2009 at 2:47 PM, Alex Cruise wrote:
>
>> On 03/02/2009 11:39 AM, John Kalucki wrote:
>>
>>> If I get a moment, I hope to fiddle with the Actor library code in
>>> question...
>>>
>>>
>> John,
>>
>> If you have a bit of room to maneuver, you might like to have a look at
>> Erik Engbrecht's recent work on refactoring the actor library (starting
>> with
>> http://erikengbrecht.blogspot.com/2009/01/refactoring-scala-actors.html )
>>
>> -0xe1a
>>
>
>
>

On Mon, 2009-03-02 at 14:55 -0500, Erik Engbrecht wrote:
> to wrap a java.util.concurrent executor:
> http://java.sun.com/j2se/1.5.0/docs/api/index.html?java/util/concurrent/...

I would first look at the fork join pool from here:

http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166y/

The plan is for it to be included in JDK7 and it should perform better
if you don't need blocking and the like.

Best,
Ismael

That's an excellent suggestion, as I think FJTaskScheduler2 is based on an early incarnation of the same code, so you won't lose features such as work stealing and the general semantics will be much more similar to the OOTB actor library.

On Mon, Mar 2, 2009 at 3:50 PM, Ismael Juma <mlists@juma.me.uk> wrote:

On Mon, 2009-03-02 at 14:55 -0500, Erik Engbrecht wrote:
> to wrap a java.util.concurrent executor:
> http://java.sun.com/j2se/1.5.0/docs/api/index.html?java/util/concurrent/ThreadPoolExecutor.html

I would first look at the fork join pool from here:

http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166y/

The plan is for it to be included in JDK7 and it should perform better
if you don't need blocking and the like.

Best,
Ismael

--
http://erikengbrecht.blogspot.com/