Scala Automatic Resource Management Incubator Project

All,

I believe that before (or while) introducing a good I/O library, Scala should include a way to perform automated resource management (open/closing a file before reading lines, etc.).

To that effect, I've pulled ideas/reworked scalax + some applications of it we used at my work place along with integrated to Paul Phillips's Exception DSL to create the following ARM library: http://github.com/jsuereth/scala-arm

The goal if this library is as follows

• Easy to use
• Extendability to various concepts (Transactions and Sockets/Streams for now)
• Functional
• Works well in for expressions

I'd like to start work on a SID, possibly by using a github WIKI page to kick it off.  That way contributors can keep track of revisions and work concurrently.   I'm only going to open up the project to users who have filled out/submitted the Scala Contributor Agreement and are willing to release their works under the Scala license.

The library requires SBT 5.5 to run (along with a custom built version against a latest scala nightly).  If you're interesting in joining/contributing to this project, please let me know!

- Josh Suereth

I briefly looked at the wiki, and it isn't clear when opening/closing happens. Could you make it more clear?
I like the opt/either thing, but I'd prefer toOption, toEither. Then again, I was on the losing side of Seq/Sequence. :-)

On Sat, Oct 3, 2009 at 10:31 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

All,

I believe that before (or while) introducing a good I/O library, Scala should include a way to perform automated resource management (open/closing a file before reading lines, etc.).

To that effect, I've pulled ideas/reworked scalax + some applications of it we used at my work place along with integrated to Paul Phillips's Exception DSL to create the following ARM library: http://github.com/jsuereth/scala-arm

The goal if this library is as follows

• Easy to use
• Extendability to various concepts (Transactions and Sockets/Streams for now)
• Functional
• Works well in for expressions

I'd like to start work on a SID, possibly by using a github WIKI page to kick it off.  That way contributors can keep track of revisions and work concurrently.   I'm only going to open up the project to users who have filled out/submitted the Scala Contributor Agreement and are willing to release their works under the Scala license.

The library requires SBT 5.5 to run (along with a custom built version against a latest scala nightly).  If you're interesting in joining/contributing to this project, please let me know!

- Josh Suereth

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

On Sun, Oct 4, 2009 at 3:17 AM, Josh Suereth wrote:
> I'll see if I can upgrade the doucmentation.  I'm also going to see if I can
> get SBT to build a scaladoc site and push it to a github pages branch so you
> can go to http://jsuereth.github.com/scala-arm and see the appropriate
> scala-doc APIs.

I could do with seeing a few not-quite-so-toy examples. How about a
simple socket based server where the resources being managed are the
listening socket and the accepted socket.

Cheers,

Miles

No problem.

I have a (similar) library being used to manage J2EE UserTransacrion's
at work. I can port that over quickly, along with file parsing code.

I'll post an example echo server with sockets. Perhaps someone can
upgrade this to http at some point?

Also, the biggest issue with ARM in scala that I've experienced is
that if you do not have an explicit "actually execute this block in an
arm" functions, then things can get confusing with returns.

Right now you can try to "acquire and get" an iterator that works on
an open resource. This is why I added the ManagedTraversable (which
ensures the resource is open while traversing). I have explicit "opt"
and "either" methods to pull from a "TranslatedResource", and only
foreach and the "acquireAndGet*" methods will execute blocks on
ManagedResource.

Anyone have any ideas on how to require "safe" return values from arm
blocks?

Currently if I enable flatMap on ManagedResource it is not obvious how
to correctly for-yield results from said resource.

- Josh

Sent from my iPhone

On Oct 4, 2009, at 7:12 AM, Miles Sabin wrote:

>
> On Sun, Oct 4, 2009 at 3:17 AM, Josh Suereth
> wrote:
>> I'll see if I can upgrade the doucmentation. I'm also going to see
>> if I can
>> get SBT to build a scaladoc site and push it to a github pages
>> branch so you
>> can go to http://jsuereth.github.com/scala-arm and see the
>> appropriate
>> scala-doc APIs.
>
> I could do with seeing a few not-quite-so-toy examples. How about a
> simple socket based server where the resources being managed are the
> listening socket and the accepted socket.
>
> Cheers,
>
>
> Miles
>

Hi,

you might want to have a look at the continuation-based socket server
example:

http://lampsvn.epfl.ch/trac/scala/browser/compiler-plugins/continuations...

The basic pattern is as follows:

for (socketChannel <- acceptConnections(80)) {
println("Connect: " + socketChannel)
for (req <- readRequests(socketChannel)) {
...
}
println("Disconnect: " + socketChannel)
}

Despite the seemingly sequential loops, the implementation uses NIO
and task runners underneath and is fully asynchronous. Resource
management is hidden from the user inside the foreach methods, using
continuations to orchestrate the underlying NIO primitives.

- Tiark

On 04.10.2009, at 15:15, Josh Suereth wrote:

> No problem.
>
> I have a (similar) library being used to manage J2EE UserTransacrion's
> at work. I can port that over quickly, along with file parsing code.
>
> I'll post an example echo server with sockets. Perhaps someone can
> upgrade this to http at some point?
>
> Also, the biggest issue with ARM in scala that I've experienced is
> that if you do not have an explicit "actually execute this block in an
> arm" functions, then things can get confusing with returns.
>
> Right now you can try to "acquire and get" an iterator that works on
> an open resource. This is why I added the ManagedTraversable (which
> ensures the resource is open while traversing). I have explicit "opt"
> and "either" methods to pull from a "TranslatedResource", and only
> foreach and the "acquireAndGet*" methods will execute blocks on
> ManagedResource.
>
> Anyone have any ideas on how to require "safe" return values from arm
> blocks?
>
> Currently if I enable flatMap on ManagedResource it is not obvious how
> to correctly for-yield results from said resource.
>
> - Josh
>
> Sent from my iPhone
>
> On Oct 4, 2009, at 7:12 AM, Miles Sabin wrote:
>
>>
>> On Sun, Oct 4, 2009 at 3:17 AM, Josh Suereth
>> wrote:
>>> I'll see if I can upgrade the doucmentation. I'm also going to see
>>> if I can
>>> get SBT to build a scaladoc site and push it to a github pages
>>> branch so you
>>> can go to http://jsuereth.github.com/scala-arm and see the
>>> appropriate
>>> scala-doc APIs.
>>
>> I could do with seeing a few not-quite-so-toy examples. How about a
>> simple socket based server where the resources being managed are the
>> listening socket and the accepted socket.
>>
>> Cheers,
>>
>>
>> Miles
>>

Hi,

you might want to have a look at the continuation-based socket server example:

       http://lampsvn.epfl.ch/trac/scala/browser/compiler-plugins/continuations/trunk/doc/examples/continuations/Test17Webserver.scala

The basic pattern is as follows:

       for (socketChannel <- acceptConnections(80)) {
               println("Connect: " + socketChannel)
               for (req <- readRequests(socketChannel)) {
                       ...
               }
               println("Disconnect: " + socketChannel)
       }

Despite the seemingly sequential loops, the implementation uses NIO and task runners underneath and is fully asynchronous. Resource management is hidden from the user inside the foreach methods, using continuations to orchestrate the underlying NIO primitives.

- Tiark

On 04.10.2009, at 15:15, Josh Suereth wrote:

No problem.

I have a (similar) library being used to manage J2EE UserTransacrion's
at work.  I can port that over quickly, along with file parsing code.

I'll post an example echo server with sockets.  Perhaps someone can
upgrade this to http at some point?

Also,  the biggest issue with ARM in scala that I've experienced is
that if you do not have an explicit "actually execute this block in an
arm" functions, then things can get confusing with returns.

Right now you can try to "acquire and get" an iterator that works on
an open resource.  This is why I added the ManagedTraversable (which
ensures the resource is open while traversing).  I have explicit "opt"
and "either" methods to pull from a "TranslatedResource", and only
foreach and the "acquireAndGet*" methods will execute blocks on
ManagedResource.

Anyone have any ideas on how to require "safe" return values from arm
blocks?

Currently if I enable flatMap on ManagedResource it is not obvious how
to correctly for-yield results from said resource.

- Josh

Sent from my iPhone

On Oct 4, 2009, at 7:12 AM, Miles Sabin <miles@milessabin.com> wrote:

On Sun, Oct 4, 2009 at 3:17 AM, Josh Suereth
<joshua.suereth@gmail.com> wrote:

I'll see if I can upgrade the doucmentation.  I'm also going to see
if I can
get SBT to build a scaladoc site and push it to a github pages
branch so you
can go to http://jsuereth.github.com/scala-arm and see the
appropriate
scala-doc APIs.

I could do with seeing a few not-quite-so-toy examples. How about a
simple socket based server where the resources being managed are the
listening socket and the accepted socket.

Cheers,


Miles

On Sun, Oct 4, 2009 at 3:17 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> I'll see if I can upgrade the doucmentation.  I'm also going to see if I can
> get SBT to build a scaladoc site and push it to a github pages branch so you
> can go to http://jsuereth.github.com/scala-arm and see the appropriate
> scala-doc APIs.

I could do with seeing a few not-quite-so-toy examples. How about a
simple socket based server where the resources being managed are the
listening socket and the accepted socket.

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth wrote:
> Here's a socket example.

To be honest, that looks surprisingly heavyweight and like Daniel it's
also not immediately clear to me where the open/close bracketing
comes. Could you add comments to the code indicating those points?

Cheers,

Miles

Part of this excercise is to make this lighter weight, hence my earler
questions about how to ensure open/close are performed correctly.

I can add comments noting where close happens.

Sent from my iPhone

On Oct 4, 2009, at 11:26 AM, Miles Sabin wrote:

>
> On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth
> wrote:
>> Here's a socket example.
>
> To be honest, that looks surprisingly heavyweight and like Daniel it's
> also not immediately clear to me where the open/close bracketing
> comes. Could you add comments to the code indicating those points?
>
> Cheers,
>
>
> Miles
>

On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> Here's a socket example.

To be honest, that looks surprisingly heavyweight and like Daniel it's
also not immediately clear to me where the open/close bracketing
comes. Could you add comments to the code indicating those points?

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

Updated:  http://wiki.github.com/jsuereth/scala-arm/socketexample


Also remember that this library is just a starting point for discussions.  I'm hoping to outline the issue areas in an ARM API so we can address them and make a final solution.

So far I think we've identified that demarking the opening/closing of resources is important, as well as finding a way to ensure the user only extracts "non-resource-bound" items from the ARM block.

Anyway, please propose alternate solutions.  Would you like access to the wiki?


- Josh

On Sun, Oct 4, 2009 at 11:26 AM, Miles Sabin <miles@milessabin.com> wrote:

On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> Here's a socket example.

To be honest, that looks surprisingly heavyweight and like Daniel it's
also not immediately clear to me where the open/close bracketing
comes. Could you add comments to the code indicating those points?

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

for(pi <-ManagedResource(new PipedInputStream)){

    for(po<-ManagedResource(new PipedOutputStream(pi))) {
  po.write('A');
  Console.println((pi.read).asInstanceOf[Char]);
    }
}

for(pi<-new ManagedResource(new PipedInputStream);

    po<-new ManagedResource(new PipedOutputStream(pi))) {
  po.write('A');
  Console.println((pi.read).asInstanceOf[Char]);
}

Hi Josh,
you might find this useful:http://www.saager.org/2007/12/30/automatic-resource-management-blocks.html
p

On Sun, Oct 4, 2009 at 5:48 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

Updated:  http://wiki.github.com/jsuereth/scala-arm/socketexample


Also remember that this library is just a starting point for discussions.  I'm hoping to outline the issue areas in an ARM API so we can address them and make a final solution.

So far I think we've identified that demarking the opening/closing of resources is important, as well as finding a way to ensure the user only extracts "non-resource-bound" items from the ARM block.

Anyway, please propose alternate solutions.  Would you like access to the wiki?


- Josh

On Sun, Oct 4, 2009 at 11:26 AM, Miles Sabin <miles@milessabin.com> wrote:

On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> Here's a socket example.

To be honest, that looks surprisingly heavyweight and like Daniel it's
also not immediately clear to me where the open/close bracketing
comes. Could you add comments to the code indicating those points?

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

Josh,

why have you stripped all IO stuff that exists in scalax.io? For
example, InputStreamSource abstract class. I find it more useful then
basic ARM stuff.

S.

On Sun, Oct 4, 2009 at 05:31, Josh Suereth wrote:
> All,
>
> I believe that before (or while) introducing a good I/O library, Scala
> should include a way to perform automated resource management (open/closing
> a file before reading lines, etc.).
>
> To that effect, I've pulled ideas/reworked scalax + some applications of it
> we used at my work place along with integrated to Paul Phillips's Exception
> DSL to create the following ARM library:
> http://github.com/jsuereth/scala-arm
>
> The goal if this library is as follows
>
> Easy to use
> Extendability to various concepts (Transactions and Sockets/Streams for now)
> Functional
> Works well in for expressions
>
> I'd like to start work on a SID, possibly by using a github WIKI page to
> kick it off.  That way contributors can keep track of revisions and work
> concurrently.   I'm only going to open up the project to users who have
> filled out/submitted the Scala Contributor Agreement and are willing to
> release their works under the Scala license.
>
> The library requires SBT 5.5 to run (along with a custom built version
> against a latest scala nightly).  If you're interesting in
> joining/contributing to this project, please let me know!
>
> - Josh Suereth
>

Josh,

why have you stripped all IO stuff that exists in scalax.io? For
example, InputStreamSource abstract class. I find it more useful then
basic ARM stuff.

S.


On Sun, Oct 4, 2009 at 05:31, Josh Suereth <joshua.suereth@gmail.com> wrote:
> All,
>
> I believe that before (or while) introducing a good I/O library, Scala
> should include a way to perform automated resource management (open/closing
> a file before reading lines, etc.).
>
> To that effect, I've pulled ideas/reworked scalax + some applications of it
> we used at my work place along with integrated to Paul Phillips's Exception
> DSL to create the following ARM library:
> http://github.com/jsuereth/scala-arm
>
> The goal if this library is as follows
>
> Easy to use
> Extendability to various concepts (Transactions and Sockets/Streams for now)
> Functional
> Works well in for expressions
>
> I'd like to start work on a SID, possibly by using a github WIKI page to
> kick it off.  That way contributors can keep track of revisions and work
> concurrently.   I'm only going to open up the project to users who have
> filled out/submitted the Scala Contributor Agreement and are willing to
> release their works under the Scala license.
>
> The library requires SBT 5.5 to run (along with a custom built version
> against a latest scala nightly).  If you're interesting in
> joining/contributing to this project, please let me know!
>
> - Josh Suereth
>

On Mon, Oct 5, 2009 at 04:12, Josh Suereth wrote:
> I'm proposing a separate library from I/O.  i.e. these are the fundamental
> resource traits from scalax.resource with a few twists I made for use in
> Scala 2.8.0 along with some things I learned from production use.
>
> If we only want ARM in the context of I/O then that's fine.  I thought it
> might deserve it's own lirbary/SID as it had the same in scalax.  It's also
> a fairly small library, so I was hoping this could be a real test of the new
> incubator process and vet out how things should work on this list.
>
> So I guess the first question here really should be:   Do you think
> Automatic-Resource-Management deserves its own library?

No, because IO stuff depends on ARM.

As I said before, for me, ARM itself is not a critical missing part of
the standard library, but IO is.

Impementation of IO worth inclusion into the standard library, ARM
alone is not. IMHO.

S.

> On Sun, Oct 4, 2009 at 7:49 PM, Stepan Koltsov
> wrote:
>>
>> Josh,
>>
>> why have you stripped all IO stuff that exists in scalax.io? For
>> example, InputStreamSource abstract class. I find it more useful then
>> basic ARM stuff.
>>
>> S.
>>
>>
>> On Sun, Oct 4, 2009 at 05:31, Josh Suereth
>> wrote:
>> > All,
>> >
>> > I believe that before (or while) introducing a good I/O library, Scala
>> > should include a way to perform automated resource management
>> > (open/closing
>> > a file before reading lines, etc.).
>> >
>> > To that effect, I've pulled ideas/reworked scalax + some applications of
>> > it
>> > we used at my work place along with integrated to Paul Phillips's
>> > Exception
>> > DSL to create the following ARM library:
>> > http://github.com/jsuereth/scala-arm
>> >
>> > The goal if this library is as follows
>> >
>> > Easy to use
>> > Extendability to various concepts (Transactions and Sockets/Streams for
>> > now)
>> > Functional
>> > Works well in for expressions
>> >
>> > I'd like to start work on a SID, possibly by using a github WIKI page to
>> > kick it off.  That way contributors can keep track of revisions and work
>> > concurrently.   I'm only going to open up the project to users who have
>> > filled out/submitted the Scala Contributor Agreement and are willing to
>> > release their works under the Scala license.
>> >
>> > The library requires SBT 5.5 to run (along with a custom built version
>> > against a latest scala nightly).  If you're interesting in
>> > joining/contributing to this project, please let me know!
>> >
>> > - Josh Suereth
>> >
>
>

On Mon, Oct 5, 2009 at 04:12, Josh Suereth <joshua.suereth@gmail.com> wrote:
> I'm proposing a separate library from I/O.  i.e. these are the fundamental
> resource traits from scalax.resource with a few twists I made for use in
> Scala 2.8.0 along with some things I learned from production use.
>
> If we only want ARM in the context of I/O then that's fine.  I thought it
> might deserve it's own lirbary/SID as it had the same in scalax.  It's also
> a fairly small library, so I was hoping this could be a real test of the new
> incubator process and vet out how things should work on this list.
>
> So I guess the first question here really should be:   Do you think
> Automatic-Resource-Management deserves its own library?


No, because IO stuff depends on ARM.

As I said before, for me, ARM itself is not a critical missing part of
the standard library, but IO is.

Impementation of IO worth inclusion into the standard library, ARM
alone is not. IMHO.

S.


> On Sun, Oct 4, 2009 at 7:49 PM, Stepan Koltsov <stepan.koltsov@gmail.com>
> wrote:
>>
>> Josh,
>>
>> why have you stripped all IO stuff that exists in scalax.io? For
>> example, InputStreamSource abstract class. I find it more useful then
>> basic ARM stuff.
>>
>> S.
>>
>>
>> On Sun, Oct 4, 2009 at 05:31, Josh Suereth <joshua.suereth@gmail.com>
>> wrote:
>> > All,
>> >
>> > I believe that before (or while) introducing a good I/O library, Scala
>> > should include a way to perform automated resource management
>> > (open/closing
>> > a file before reading lines, etc.).
>> >
>> > To that effect, I've pulled ideas/reworked scalax + some applications of
>> > it
>> > we used at my work place along with integrated to Paul Phillips's
>> > Exception
>> > DSL to create the following ARM library:
>> > http://github.com/jsuereth/scala-arm
>> >
>> > The goal if this library is as follows
>> >
>> > Easy to use
>> > Extendability to various concepts (Transactions and Sockets/Streams for
>> > now)
>> > Functional
>> > Works well in for expressions
>> >
>> > I'd like to start work on a SID, possibly by using a github WIKI page to
>> > kick it off.  That way contributors can keep track of revisions and work
>> > concurrently.   I'm only going to open up the project to users who have
>> > filled out/submitted the Scala Contributor Agreement and are willing to
>> > release their works under the Scala license.
>> >
>> > The library requires SBT 5.5 to run (along with a custom built version
>> > against a latest scala nightly).  If you're interesting in
>> > joining/contributing to this project, please let me know!
>> >
>> > - Josh Suereth
>> >
>
>

On Mon, Oct 5, 2009 at 04:12, Josh Suereth <joshua.suereth@gmail.com> wrote:
> I'm proposing a separate library from I/O.  i.e. these are the fundamental
> resource traits from scalax.resource with a few twists I made for use in
> Scala 2.8.0 along with some things I learned from production use.
>
> If we only want ARM in the context of I/O then that's fine.  I thought it
> might deserve it's own lirbary/SID as it had the same in scalax.  It's also
> a fairly small library, so I was hoping this could be a real test of the new
> incubator process and vet out how things should work on this list.
>
> So I guess the first question here really should be:   Do you think
> Automatic-Resource-Management deserves its own library?


No, because IO stuff depends on ARM.

As I said before, for me, ARM itself is not a critical missing part of
the standard library, but IO is.

Impementation of IO worth inclusion into the standard library, ARM
alone is not. IMHO.

S.


> On Sun, Oct 4, 2009 at 7:49 PM, Stepan Koltsov <stepan.koltsov@gmail.com>
> wrote:
>>
>> Josh,
>>
>> why have you stripped all IO stuff that exists in scalax.io? For
>> example, InputStreamSource abstract class. I find it more useful then
>> basic ARM stuff.
>>
>> S.
>>
>>
>> On Sun, Oct 4, 2009 at 05:31, Josh Suereth <joshua.suereth@gmail.com>
>> wrote:
>> > All,
>> >
>> > I believe that before (or while) introducing a good I/O library, Scala
>> > should include a way to perform automated resource management
>> > (open/closing
>> > a file before reading lines, etc.).
>> >
>> > To that effect, I've pulled ideas/reworked scalax + some applications of
>> > it
>> > we used at my work place along with integrated to Paul Phillips's
>> > Exception
>> > DSL to create the following ARM library:
>> > http://github.com/jsuereth/scala-arm
>> >
>> > The goal if this library is as follows
>> >
>> > Easy to use
>> > Extendability to various concepts (Transactions and Sockets/Streams for
>> > now)
>> > Functional
>> > Works well in for expressions
>> >
>> > I'd like to start work on a SID, possibly by using a github WIKI page to
>> > kick it off.  That way contributors can keep track of revisions and work
>> > concurrently.   I'm only going to open up the project to users who have
>> > filled out/submitted the Scala Contributor Agreement and are willing to
>> > release their works under the Scala license.
>> >
>> > The library requires SBT 5.5 to run (along with a custom built version
>> > against a latest scala nightly).  If you're interesting in
>> > joining/contributing to this project, please let me know!
>> >
>> > - Josh Suereth
>> >
>
>