actors on opensolaris and os x

On 08/02/10 06:49, Claus Guttesen wrote:
> Hi.
>
> I'm writing a small scala-program using 2.8.0 beta 1 where I'm
> converting images using ImageMagick. Some images can be converted in
> parallel and I'm trying to wrap my head around to use actors "the
> right way" :-) .
>
> To test concurrency I convert x images on os x 10.6.2 and opensolaris
> rev. 131. Both are using jdk 1.6. Opensolaris is on a xeon E5520 with
> four cores with hyperthreading enabled (eight virtual cores). Os x is
> my macbook pro with a dual core.
>
> When I run the program on opensolaris 16 processes are running in
> parallel which (I guess) are two threads pr. core. On my macbook pro
> it starts as many processes as I wish making it crawl.
>
> Do I have to specify a flag on os x to limit concurrent processes?
>
> The scala-program is at http://dl.dropbox.com/u/2729115/Convert.scala.
> Starting 99 processes: scala Convert 99
>
> opensolaris: ps ax | grep -i "convert -resize" | wc -l
> 17 (including the ps-command)
>
> os x: ps ax | grep -i "convert -resize" | wc -l
> 78
>
> I had to force a shutdown when I had watched the beach-ball for some
> minutes so the program spawn as many processes as I request.

I'm afraid I'm only a novice at using Scala, but I have played with
Actors a little. It is my understanding that they will only ever process
one task at a time from their message queue, at least by default. (Can
you point me to how to enable the worker-pool type Actors?)

I wrote a simple version of your program, that sleeps instead of
shelling out to an external image processing program, and it
demonstrates that there's only ever one thing happening at once.

I'll try and attach them to this email..
After compilation, I run and get:

tobyc@arya:~/scala-2.8.0.Beta1-prerelease/actors
$ scala toby.example.Convert 5
Hi convert!
Starting job: 1 on img1.tiff
Done job # 1 on img1.tiff
Starting job: 2 on img2.tiff
Done job # 2 on img2.tiff
Starting job: 3 on img3.tiff
Done job # 3 on img3.tiff
Starting job: 4 on img4.tiff
Done job # 4 on img4.tiff
Starting job: 5 on img5.tiff
Done job # 5 on img5.tiff
Exiting

On 08/02/10 17:31, Toby Corkindale wrote:
> On 08/02/10 06:49, Claus Guttesen wrote:
>> Hi.
>>
>> I'm writing a small scala-program using 2.8.0 beta 1 where I'm
>> converting images using ImageMagick. Some images can be converted in
>> parallel and I'm trying to wrap my head around to use actors "the
>> right way" :-) .
>>
>> To test concurrency I convert x images on os x 10.6.2 and opensolaris
>> rev. 131. Both are using jdk 1.6. Opensolaris is on a xeon E5520 with
>> four cores with hyperthreading enabled (eight virtual cores). Os x is
>> my macbook pro with a dual core.
>>
>> When I run the program on opensolaris 16 processes are running in
>> parallel which (I guess) are two threads pr. core. On my macbook pro
>> it starts as many processes as I wish making it crawl.
>>
>> Do I have to specify a flag on os x to limit concurrent processes?
>>
>> The scala-program is at http://dl.dropbox.com/u/2729115/Convert.scala.
>> Starting 99 processes: scala Convert 99
>>
>> opensolaris: ps ax | grep -i "convert -resize" | wc -l
>> 17 (including the ps-command)
>>
>> os x: ps ax | grep -i "convert -resize" | wc -l
>> 78
>>
>> I had to force a shutdown when I had watched the beach-ball for some
>> minutes so the program spawn as many processes as I request.
>
> I'm afraid I'm only a novice at using Scala, but I have played with
> Actors a little. It is my understanding that they will only ever process
> one task at a time from their message queue, at least by default. (Can
> you point me to how to enable the worker-pool type Actors?)
>
> I wrote a simple version of your program, that sleeps instead of
> shelling out to an external image processing program, and it
> demonstrates that there's only ever one thing happening at once.

Sorry, forgot to say - I couldn't work out why you were passing self to
the actor then calling conv() on it.. Seemed wrong to me, so I removed
it. Was it done that way for a reason?

Also, I'm not convinced that calling exit() inside the Actor is the
Right Way To Do It; I think you should just leave the actor alone and
let it get cleaned up when it goes out of scope, and call exit() from
inside your main subroutine. (Or just return out of it.)

On Feb 8, 2010, at 7:36 AM, Toby Corkindale wrote:

On 08/02/10 17:31, Toby Corkindale wrote:
Sorry, forgot to say - I couldn't work out why you were passing self to the actor then calling conv() on it.. Seemed wrong to me, so I removed it. Was it done that way for a reason?

Also, I'm not convinced that calling exit() inside the Actor is the Right Way To Do It; I think you should just leave the actor alone and let it get cleaned up when it goes out of scope, and call exit() from inside your main subroutine. (Or just return out of it.)

And his main class is not an actor, so he tries to send a message to something that is not an actor it seems.Process() here would be nonblocking, and I guess that is why there are so many convert processes spawned, check out waitFor in java.lang.Process.

>> On 08/02/10 17:31, Toby Corkindale wrote:
>> Sorry, forgot to say - I couldn't work out why you were passing self to the
>> actor then calling conv() on it.. Seemed wrong to me, so I removed it. Was
>> it done that way for a reason?

I modified an actor-example in 'Programming in Scala' on page 618.

> And his main class is not an actor, so he tries to send a message to
> something that is not an actor it seems.
> Process() here would be nonblocking, and I guess that is why there are so
> many convert processes spawned, check out waitFor in java.lang.Process.

I instantiate Actor as module and send filename and self as a
message. In actor module I "spawn" a new actor which does the actual
convert of the file. It does process up to 16 simultaneous processes
on opensolaris but all on os x.

It does seems to be one way to create new actors when they can be run
in parallel but this is one of my first attempts to utilize actors so
I might overlook an important aspect. :-)

Hi,

> When I run the program on opensolaris 16 processes are running in
> parallel which (I guess) are two threads pr. core. On my macbook pro
> it starts as many processes as I wish making it crawl.

In your code all the actors you create execute a blocking operation,
therefore you need as many threads as there are actors. I suspect that
OpenSolaris uses a hybrid scheme to map JVM threads to OS processes
where not every thread gets its own process. You could obtain a JVM
thread dump when running the code to see how many threads are created
(or just use VisualVM).

> Do I have to specify a flag on os x to limit concurrent processes?

The number of worker threads that are created is limited to 256 by
default. But you can change that using the `actors.maxPoolSize` property.

Usually, it's best to use non-blocking I/O where completion events are
turned into messages for actors. However, in your case you are running a
potentially large number of external processes. Actors can't help you
with that out-of-the-box. You'd need some I/O library that has a smart
way of executing external processes where the creation of a new process
is delayed until enough old ones have finished. On the other hand, if
you set the maximum size of the actors worker pool you get this delaying
automatically (a new actor won't be started until an older one has
finished).

Cheers,
Philipp