(yet another) Swarm question for you

In moveTo:
...
if (Swarm.isLocal(location)) {
log("Is local")
c()
NoBee()
} else {
...
}

shouldn't you return the result of c instead of NoBee?

- Tiark

On 02.09.2009, at 01:55, Ian Clarke wrote:

> Tiark,
>
> Another problem for you to ponder (again, you can find all relevant
> source at http://github.com/sanity/Swarm/tree/master).
>
> I'm trying a new experiment:
>
> def exp2(u : Unit) = {
> val vLoc = Reference.create[String](myLocation, "test local
> string");
> val vRem = Reference.create[String](new Location(myLocation.address,
> 9997), "test remote string");
> moveTo(vLoc.location)
> println("moved 1");
> moveTo(vRem.location)
> println("moved 2");
> NoBee()
> }
>
> The line beginning vRem should result in the continuation being moved
> to the node with port 9997, but it never reaches the "Transmitting
> task to..." line in execute():
>
> 9998 : Waiting for connection
> 9998 : Running task
> 9998 : Move to
> 9998 : Is local
> 9998 : Move to
> 9998 : Moving task to 9997
> No more continuations to execute
>
> Any ideas?
>
> Ian.
>

You just saved my bacon, again, thank you!

My problem is that I'm still stumbling around in the dark a bit with
the continuations stuff, it is surprisingly difficult for me to get my
head around it.

How did you initially learn about and come to understand delimited
continuations?

Ian.

On Thu, Sep 3, 2009 at 8:57 AM, Tiark Rompf wrote:
> In moveTo:
> ...
> if (Swarm.isLocal(location)) {
>        log("Is local")
>        c()
>        NoBee()
> } else {
> ...
> }
>
> shouldn't you return the result of c instead of NoBee?
>
> - Tiark
>
>
> On 02.09.2009, at 01:55, Ian Clarke wrote:
>
>> Tiark,
>>
>> Another problem for you to ponder (again, you can find all relevant
>> source at http://github.com/sanity/Swarm/tree/master).
>>
>> I'm trying a new experiment:
>>
>>        def exp2(u : Unit) = {
>>                val vLoc = Reference.create[String](myLocation, "test local
>> string");
>>                val vRem = Reference.create[String](new
>> Location(myLocation.address,
>> 9997), "test remote string");
>>                moveTo(vLoc.location)
>>                println("moved 1");
>>                moveTo(vRem.location)
>>                println("moved 2");
>>                NoBee()
>>        }
>>
>> The line beginning vRem should result in the continuation being moved
>> to the node with port 9997, but it never reaches the "Transmitting
>> task to..." line in execute():
>>
>> 9998 : Waiting for connection
>> 9998 : Running task
>> 9998 : Move to
>> 9998 : Is local
>> 9998 : Move to
>> 9998 : Moving task to 9997
>> No more continuations to execute
>>
>> Any ideas?
>>
>> Ian.
>>
>> --
>> Ian Clarke
>> CEO, Uprizer Labs
>> Email: ian@uprizer.com
>> Ph: +1 512 422 3588
>> Fax: +1 512 276 6674
>
>

By the way, would you mind explaining why what I was doing above was
wrong? Why did it matter what I returned? moveTo() doesn't need to
return anything?

Ian.

On Thu, Sep 3, 2009 at 10:13 AM, Ian Clarke wrote:
> You just saved my bacon, again, thank you!
>
> My problem is that I'm still stumbling around in the dark a bit with
> the continuations stuff, it is surprisingly difficult for me to get my
> head around it.
>
> How did you initially learn about and come to understand delimited
> continuations?
>
> Ian.
>
> On Thu, Sep 3, 2009 at 8:57 AM, Tiark Rompf wrote:
>> In moveTo:
>> ...
>> if (Swarm.isLocal(location)) {
>>        log("Is local")
>>        c()
>>        NoBee()
>> } else {
>> ...
>> }
>>
>> shouldn't you return the result of c instead of NoBee?
>>
>> - Tiark
>>
>>
>> On 02.09.2009, at 01:55, Ian Clarke wrote:
>>
>>> Tiark,
>>>
>>> Another problem for you to ponder (again, you can find all relevant
>>> source at http://github.com/sanity/Swarm/tree/master).
>>>
>>> I'm trying a new experiment:
>>>
>>>        def exp2(u : Unit) = {
>>>                val vLoc = Reference.create[String](myLocation, "test local
>>> string");
>>>                val vRem = Reference.create[String](new
>>> Location(myLocation.address,
>>> 9997), "test remote string");
>>>                moveTo(vLoc.location)
>>>                println("moved 1");
>>>                moveTo(vRem.location)
>>>                println("moved 2");
>>>                NoBee()
>>>        }
>>>
>>> The line beginning vRem should result in the continuation being moved
>>> to the node with port 9997, but it never reaches the "Transmitting
>>> task to..." line in execute():
>>>
>>> 9998 : Waiting for connection
>>> 9998 : Running task
>>> 9998 : Move to
>>> 9998 : Is local
>>> 9998 : Move to
>>> 9998 : Moving task to 9997
>>> No more continuations to execute
>>>
>>> Any ideas?
>>>
>>> Ian.
>>>
>>> --
>>> Ian Clarke
>>> CEO, Uprizer Labs
>>> Email: ian@uprizer.com
>>> Ph: +1 512 422 3588
>>> Fax: +1 512 276 6674
>>
>>
>
>
>
> --
> Ian Clarke
> CEO, Uprizer Labs
> Email: ian@uprizer.com
> Ph: +1 512 422 3588
> Fax: +1 512 276 6674
>

> My problem is that I'm still stumbling around in the dark a bit with
> the continuations stuff, it is surprisingly difficult for me to get my
> head around it.

Comment from the peanut gallery, and disclaimer that I've done
absolutely nothing with Scala continuations, but have you considered
decompiling the compiler/plugin-produced bytecode back into Java source?

My hope would be that looking at the internal guts of the output would
reduce the perceived magic of the continuation plugin's transformations.

Speaking of which, has anyone done this and posted the results? A quick
google didn't turn up anything.

- Stephen

Running

scalac -Xprint:selectivecps

will print out the transformed code.

- Tiark

On 03.09.2009, at 16:33, Stephen Haberman wrote:

>
>> My problem is that I'm still stumbling around in the dark a bit with
>> the continuations stuff, it is surprisingly difficult for me to get
>> my
>> head around it.
>
> Comment from the peanut gallery, and disclaimer that I've done
> absolutely nothing with Scala continuations, but have you considered
> decompiling the compiler/plugin-produced bytecode back into Java
> source?
>
> My hope would be that looking at the internal guts of the output would
> reduce the perceived magic of the continuation plugin's
> transformations.
>
> Speaking of which, has anyone done this and posted the results? A
> quick
> google didn't turn up anything.
>
> - Stephen
>

Function c at that point is the remaining code you want to execute, so
it can contain further invocations of moveTo. And if you encounter a
remote moveTo, then you want to stop execution and return the
continuation (i.e. the code you want to move) wrapped in an IsBee
object to the top level. So this is not about the return value of
moveTo (i.e. the value passed to moveTo's continuation), but the
result of the shift's body inside moveTo (i.e. what will be the result
of the enclosing reset). Now, if c calls moveTo with a remote
location, c() will return an IsBee object. Instead of passing it back
up to the caller of reset, you just replaced it with a NoBee, which
means the remaining program was discarded (that's why you got the "no
more things to execute" message).

Nobody said delimited continuations are easy ;-)) There is a set of
annotated slides by Oleg Kiselyov, which I think give a good
explanation (http://okmij.org/ftp/Computation/Fest2008-talk-
notes.pdf). Also, it helps to look for analogies to simpler examples.
Can you figure out what the following code will print and why?

def foo() = {
1 + shift(k => k(k(k(8))))
}
def bar() = {
foo() * 2
}
def baz() = {
reset(bar())
}
println(baz())

- Tiark

On 03.09.2009, at 16:15, Ian Clarke wrote:

> By the way, would you mind explaining why what I was doing above was
> wrong? Why did it matter what I returned? moveTo() doesn't need to
> return anything?
>
> Ian.
>
> On Thu, Sep 3, 2009 at 10:13 AM, Ian Clarke
> wrote:
>> You just saved my bacon, again, thank you!
>>
>> My problem is that I'm still stumbling around in the dark a bit with
>> the continuations stuff, it is surprisingly difficult for me to get
>> my
>> head around it.
>>
>> How did you initially learn about and come to understand delimited
>> continuations?
>>
>> Ian.
>>
>> On Thu, Sep 3, 2009 at 8:57 AM, Tiark Rompf
>> wrote:
>>> In moveTo:
>>> ...
>>> if (Swarm.isLocal(location)) {
>>> log("Is local")
>>> c()
>>> NoBee()
>>> } else {
>>> ...
>>> }
>>>
>>> shouldn't you return the result of c instead of NoBee?
>>>
>>> - Tiark
>>>
>>>
>>> On 02.09.2009, at 01:55, Ian Clarke wrote:
>>>
>>>> Tiark,
>>>>
>>>> Another problem for you to ponder (again, you can find all relevant
>>>> source at http://github.com/sanity/Swarm/tree/master).
>>>>
>>>> I'm trying a new experiment:
>>>>
>>>> def exp2(u : Unit) = {
>>>> val vLoc = Reference.create[String](myLocation,
>>>> "test local
>>>> string");
>>>> val vRem = Reference.create[String](new
>>>> Location(myLocation.address,
>>>> 9997), "test remote string");
>>>> moveTo(vLoc.location)
>>>> println("moved 1");
>>>> moveTo(vRem.location)
>>>> println("moved 2");
>>>> NoBee()
>>>> }
>>>>
>>>> The line beginning vRem should result in the continuation being
>>>> moved
>>>> to the node with port 9997, but it never reaches the "Transmitting
>>>> task to..." line in execute():
>>>>
>>>> 9998 : Waiting for connection
>>>> 9998 : Running task
>>>> 9998 : Move to
>>>> 9998 : Is local
>>>> 9998 : Move to
>>>> 9998 : Moving task to 9997
>>>> No more continuations to execute
>>>>
>>>> Any ideas?
>>>>
>>>> Ian.

Tiark,

Thanks for the reference, I'll read it just as soon as I can, I'll get
back to you on the puzzle :-)

A more general question though that I think will be of interest to all
potential users of the delimited continuations plugin:

If I'm creating a framework based on the plugin, and my users will be
writing code that (most often unknown to them) will run inside a
reset{} and may call a shift{} - correct me if I'm wrong but it looks
like they can't simply use vanilla Scala and vanilla Scala API calls
(like foreach) and expect it to work. I assume this is the reason you
need to define a custom foreach in ContinuationContext.

So the question is, are there a simple set of guidelines or
constraints that users of such a library can operate in which will
allow them to write "continuation friendly" code, without having to
understand the complexities of the continuations plugin?

Would it be possible to create a compiler plugin that enforces these
constraints at compile-time, while printing user-friendly error
messages if the user does something wrong?

I think it would be very problematic if the users of any
library/framework that relied on the continuations plugin had to
understand the continuations plugin in order to use the library, given
that delimited continuations have such a steep learning curve.

Ian.

I don't think users of your framework will have to be aware of how
continuations work. However, there's no free lunch. On the plus side,
you gain that your users can write

moveTo(anotherLocation)

moveTo(yetAnotherLocation)
...

and it all just works. On the minus side, you have to explain to them
that when they want to write

for (x <- listOfLocations) {
moveTo(x)
...
}

they have to write

for (x <- listOfLocations.suspendable) {
moveTo(x)
...
}

instead. If you wouldn't use shift/reset inside your library
functions, your users would have to write

moveTo(anotherLocation) {

moveTo(yetAnotherLocation) {
...
}
}

(or maybe for (_ <- moveTo(location) ...) and, more importantly, they
couldn't use Iterable.foreach/map at all!

So bottom line, it looks like a clear win to me. To make it even
easier you could introduce a type alias to hide the @cps[Bee, Bee]
annotations, so your users would only see e.g. @swarm. Then the
guideline would be: if you use one of the functions that return
Something @swarm in a for comprehension, make sure you
use .suspendable on the traversed collection.

What other Scala language feature or API besides collection traversal
do you have in mind that you can't use? If it is something important
we can see whether we find a way to support it. The general limitation
is that you cannot place an expression that will call shift in a
context that is not prepared for it. So you have to be careful with
anonymous functions that are passed as arguments to library code (like
foreach or map).

In the case of your framework, moveTo migrates the computation to a
different machine, so I think your users would understand that there
are some restrictions on where in a program you can do migrations and
where not.

In any event, the compiler will complain if something is wrong (if it
does not it's a bug that should be fixed). I do agree that the error
messages should be improved although I'm not sure to what extent this
is possible.

- Tiark

> If I'm creating a framework based on the plugin, and my users will be
> writing code that (most often unknown to them) will run inside a
> reset{} and may call a shift{} - correct me if I'm wrong but it looks
> like they can't simply use vanilla Scala and vanilla Scala API calls
> (like foreach) and expect it to work. I assume this is the reason you
> need to define a custom foreach in ContinuationContext.
>
> So the question is, are there a simple set of guidelines or
> constraints that users of such a library can operate in which will
> allow them to write "continuation friendly" code, without having to
> understand the complexities of the continuations plugin?
>
> Would it be possible to create a compiler plugin that enforces these
> constraints at compile-time, while printing user-friendly error
> messages if the user does something wrong?
>
> I think it would be very problematic if the users of any
> library/framework that relied on the continuations plugin had to
> understand the continuations plugin in order to use the library, given
> that delimited continuations have such a steep learning curve.
>
> Ian.
>

Thanks once again for your response.

On Thu, Sep 3, 2009 at 2:15 PM, Tiark Rompf wrote:
> To make it even easier you could introduce a type alias to hide the @cps[Bee, Bee]
> annotations, so your users would only see e.g. @swarm

Hmm, you can define an alias for an annotation? Could you explain
further how one would do this?

> What other Scala language feature or API besides collection traversal do you
> have in mind that you can't use? If it is something important we can see
> whether we find a way to support it. The general limitation is that you
> cannot place an expression that will call shift in a context that is not
> prepared for it. So you have to be careful with anonymous functions that are
> passed as arguments to library code (like foreach or map).

I guess the other thing that springs to mind is if the user is
defining their own functions which contain calls to moveTo(). What
general instruction can they be given about the return types of those
functions? I assume that if they don't specify a return type, Scala
will figure it out, but many people like to specify return types
explicitly.

> In the case of your framework, moveTo migrates the computation to a
> different machine, so I think your users would understand that there are
> some restrictions on where in a program you can do migrations and where not.

Agreed.

> In any event, the compiler will complain if something is wrong (if it does
> not it's a bug that should be fixed). I do agree that the error messages
> should be improved although I'm not sure to what extent this is possible.

I was thinking that at some point I could create a Swarm-specific
compiler plugin that would enforce various guidelines like only using
@serializable objects where they might need to be transmitted across
the network, and other guidelines related to ensuring that the
continuations work. Of course, this is bluesky stuff considering the
rudimentary stage I'm at today :-)

Ian.

Dude,

The more you talk about your implementation, the more i am reminded of Lucian's implementation of the fusion machine.

Best wishes,

--greg

On Fri, Sep 4, 2009 at 6:57 AM, Ian Clarke <ian.clarke@gmail.com> wrote:

Thanks once again for your response.

On Thu, Sep 3, 2009 at 2:15 PM, Tiark Rompf<tiark.rompf@epfl.ch> wrote:
> To make it even easier you could introduce a type alias to hide the @cps[Bee, Bee]
> annotations, so your users would only see e.g. @swarm

Hmm, you can define an alias for an annotation?  Could you explain
further how one would do this?

> What other Scala language feature or API besides collection traversal do you
> have in mind that you can't use? If it is something important we can see
> whether we find a way to support it. The general limitation is that you
> cannot place an expression that will call shift in a context that is not
> prepared for it. So you have to be careful with anonymous functions that are
> passed as arguments to library code (like foreach or map).

I guess the other thing that springs to mind is if the user is
defining their own functions which contain calls to moveTo().  What
general instruction can they be given about the return types of those
functions?  I assume that if they don't specify a return type, Scala
will figure it out, but many people like to specify return types
explicitly.

> In the case of your framework, moveTo migrates the computation to a
> different machine, so I think your users would understand that there are
> some restrictions on where in a program you can do migrations and where not.

Agreed.

> In any event, the compiler will complain if something is wrong (if it does
> not it's a bug that should be fixed). I do agree that the error messages
> should be improved although I'm not sure to what extent this is possible.

I was thinking that at some point I could create a Swarm-specific
compiler plugin that would enforce various guidelines like only using
@serializable objects where they might need to be transmitted across
the network, and other guidelines related to ensuring that the
continuations work.  Of course, this is bluesky stuff considering the
rudimentary stage I'm at today :-)

Ian.

--
Ian Clarke
CEO, Uprizer Labs
Email: ian@uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
1219 NW 83rd St
Seattle, WA 98117

+1 206.650.3740

http://biosimilarity.blogspot.com

On 04.09.2009, at 15:57, Ian Clarke wrote:

> Thanks once again for your response.
>
> On Thu, Sep 3, 2009 at 2:15 PM, Tiark Rompf
> wrote:
>> To make it even easier you could introduce a type alias to hide the
>> @cps[Bee, Bee]
>> annotations, so your users would only see e.g. @swarm
>
> Hmm, you can define an alias for an annotation? Could you explain
> further how one would do this?

Annotations are just classes inheriting from scala.Annotation, so all
you have to do is define

type swarm = cps[Bee, Bee]

somewhere and make sure it's imported where you want to use it.

>> What other Scala language feature or API besides collection
>> traversal do you
>> have in mind that you can't use? If it is something important we
>> can see
>> whether we find a way to support it. The general limitation is that
>> you
>> cannot place an expression that will call shift in a context that
>> is not
>> prepared for it. So you have to be careful with anonymous functions
>> that are
>> passed as arguments to library code (like foreach or map).
>
> I guess the other thing that springs to mind is if the user is
> defining their own functions which contain calls to moveTo(). What
> general instruction can they be given about the return types of those
> functions? I assume that if they don't specify a return type, Scala
> will figure it out, but many people like to specify return types
> explicitly.

Any function that will call moveTo without enclosing reset will need
to have the same @cps[Bee,Bee] annotation on its return type as
moveTo. In general, the @cps annotations are always propagated
outwards. WIth the above type alias an example would be:

def computeThreePlusFourRemotely(): Int @swarm = {
moveTo(remoteSite)
3+4
}

Omitting the return type will also work, but declaring just Int as
return type will produce a 'found cps expression in non-cps position'
error.

>
>> In the case of your framework, moveTo migrates the computation to a
>> different machine, so I think your users would understand that
>> there are
>> some restrictions on where in a program you can do migrations and
>> where not.
>
> Agreed.
>
>> In any event, the compiler will complain if something is wrong (if
>> it does
>> not it's a bug that should be fixed). I do agree that the error
>> messages
>> should be improved although I'm not sure to what extent this is
>> possible.
>
> I was thinking that at some point I could create a Swarm-specific
> compiler plugin that would enforce various guidelines like only using
> @serializable objects where they might need to be transmitted across
> the network, and other guidelines related to ensuring that the
> continuations work. Of course, this is bluesky stuff considering the
> rudimentary stage I'm at today :-)

It's not too hard to write simple compiler plugins that check specific
things but I have the impression that checking for serializability
would be difficult. Imagine, e.g. an object that is declared as
serializable but has fields pointing to objects which are not.

- Tiark