Re: Scala Automatic Resource Management Incubator Project

Based on my understanding of the original Scalax ManagedResource stuff, ManagedResource was not really meant to be a user-facing library. It was rather intended to be a library-writer's library. A user would rarely wrap resources inside of a ManagedResource themselves. Instead, user-facing libraries (like IO, Transactions, Locks, and anything else that needs to open a resource and ensure that it is closed after it's been used) would provide access to objects that were Managed. For a user to get access to a Managed object, they'd have to use it inside a for-comprehension.

For example, in scalax.io, printing the lines of a file was as simple as:

  val file: java.io.File = ...
  for (line <- file.lines)
    println(line)

Of course, a lot of this relied on implicit magic at the level of scalax.io, but the ARM component ensured that the file was properly opened and closed. At the user-level, there was no need to create a Managed* object, just to know that file.lines returned a ManagedSequence (now ManagedTraversable).

Now, as I understand the scalax ARM stuff, a ManagedResource has two central concepts: a Resource and a Handle. The Handle is the thing that needs to be opened/closed, and the Resource is the thing that a user actually wants to work with. In the basic, most common case, the Handle and the Resource are the same thing. The thing that needs to be open/closed is the same thing that the users wants to work with. This is called an UntranslatedManagedResource, for reasons that will become clear in a second.

A ManagedResource has three methods that are meant to be used by the library, not directly by a user. To open a ManagedResource means to obtain a Handle, to close a ManagedResource means to dispose of a Handle, and to translate means to go from a Handle to a raw Resource (hence Untranslated when a Handle and a Resource are the same thing). There is one method which is actually meant to be used by the user of a ManagedResource. It takes a function on a raw Resource and performs the full chain of opening  the ManagedResource to obtain the Handle, translating the handle to a raw Resource, invoke the user's function on the raw Resource, and closing the ManagedResource (disposing of the Handle).

As mentioned above, most ManagedResources are in fact UntranslatedManagedResources. This is the case for InputStreams, FileReaders, etc. In fact, in the scalax code, there were very few translated ManagedResources. An example of where you would want a translated resource was FileChannels. In this case, the Handle is an InputStream and the Resource is a FileChannel. The InputStream (Handle) needs to be opened and closed safely, but the user doesn't ever need to know that the InputStream even exists, the user only ever deals with the FileChannel. The user passes functions that operate on a FileChannel (the Resource), and the library takes care of opening an InputStream, translating the InputStream to a FileChannel, invoking the user's function on the FileChannel, and closing the InputStream.

Now, this is my understanding of the ARM code in the original scalax. I'm not sure exactly how much this applies to the new code Josh has in scala-arm. From a cursory inspection it seems like Josh is adding further laziness, such that you can stack user operations on ManagedResources and "force" them only once you're done. I'm also not sure how well the terminology I've described also applies to Josh's library. Josh, care to comment?

--j

On Sun, Oct 4, 2009 at 9:46 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

No problem!  Communication is what this is all about.

So...   That line is unecessarily complex due to the bloat of the java.io library.  Assuming a sane Scala.io library, it would look more like the following:

val input = ManagedResource(file.reader).toTraversable(_.lines)

input.foreach(println) //Prints all lines in the file...

Basically, the ManagedResource is defined on the inputstream from a connection (in this case to a file's "reader").  The API provides not only a mechanism of opening/closing resources but a "ManagedTraversable".  The idea here is that you have a collection that lives inside a resource.   The ManagedTraversable opens the resource before iteration and closes it afterwards.  Because Traversable implements off a "foreach" method, any kind of traversing method performed on ManagedTraversbale will open/close the resource.   This can be done lazily by using a view.

What I've tried to do is keep the simple while adding more powerful features.  i.e. for( handle <- ManagedResource(openResource)) { foo(handle) }   still works just fine.   However you have some additional advanced features if needed.

Another feature I tried to add is appending work to a resource to be performed when it is finally opened.   The map operator will defer a function to be applied against the resource when it is opened.  This allows you to convert between various java.io  classes before actually opening your resource.   My JavaBufferedreaderLineIterator requires a java.io.BufferedReader.   A Socket returns an InputStream.   Hence the "map" before the call to "lines".  

The library also has a notion of extracting data from within a resource.   The ManagedTraversable is a decent way to do this, but will continually try to open the resource if you use it more than once.   The "opt" and "either" methods on a TranslatedResource (i.e. after a map function) are used to extract whatever results the mapping function may generate.   This should be used to extract *safe* items from the resource (i.e. not iterators tied to the resource). 


Anyway, I hope this helps outline the thought process and how the additions to the library came to be.


- Josh


On Mon, Oct 5, 2009 at 12:33 AM, phausel <peter.hausel@gmail.com> wrote:

Hi Josh,

Thanks for getting back to me.

I understand this one below covers a more complex use case but for me
it does look heavyweight (especially compared to C#'s or java7's
solution) and also it's hard to tell where opening/closing is
happening:
val input = ManagedResource(connection.getInputStream).map( is => new
BufferedReader(new InputStreamReader(is))).toTraversable( br => new
JavaBufferedReaderLineIterator(br))

just a thought
Thanks Peter


On Oct 5, 1:24 am, Josh Suereth <joshua.suer...@gmail.com> wrote:
> Actually, you can accomplish the same thing in the API I've started as a
> sample discussion point.   The code is adapted from Scalax.IO and some
> production libraries I use at work, so I have some ideas of the complexities
> involved with ARM.
>
> The simple lightweight version is still there in my API, however I wanted to
> have depth when you need it.   Here's what the above blog sample would look
> like in my API
>
> for(pi <-ManagedResource(new PipedInputStream)){
>     for(po<-ManagedResource(new PipedOutputStream(pi))) {
>   po.write('A');
>   Console.println((pi.read).asInstanceOf[Char]);
>     }
>
> }
>
> vs. the original
>
> for(pi<-new ManagedResource(new PipedInputStream);
>     po<-new ManagedResource(new PipedOutputStream(pi))) {
>   po.write('A');
>   Console.println((pi.read).asInstanceOf[Char]);
>
> }
>
> So now my question is this:  What's "heavyweight" about my API?
>
>
>
> On Sun, Oct 4, 2009 at 2:09 PM, Peter Hausel <peter.hau...@gmail.com> wrote:
> > Hi Josh,
> > you might find this useful:
> >http://www.saager.org/2007/12/30/automatic-resource-management-blocks...
>
> > p
>
> > On Sun, Oct 4, 2009 at 5:48 PM, Josh Suereth <joshua.suer...@gmail.com>wrote:
>
> >> Updated:  http://wiki.github.com/jsuereth/scala-arm/socketexample
>
> >> Also remember that this library is just a starting point for discussions.
> >> I'm hoping to outline the issue areas in an ARM API so we can address them
> >> and make a final solution.
>
> >> So far I think we've identified that demarking the opening/closing of
> >> resources is important, as well as finding a way to ensure the user only
> >> extracts "non-resource-bound" items from the ARM block.
>
> >> Anyway, please propose alternate solutions.  Would you like access to the
> >> wiki?
>
> >> - Josh
>
> >> On Sun, Oct 4, 2009 at 11:26 AM, Miles Sabin <mi...@milessabin.com>wrote:
>
> >>> On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth <joshua.suer...@gmail.com>
> >>> wrote:
> >>> > Here's a socket example.
>
> >>> To be honest, that looks surprisingly heavyweight and like Daniel it's
> >>> also not immediately clear to me where the open/close bracketing
> >>> comes. Could you add comments to the code indicating those points?
>
> >>> Cheers,
>
> >>> Miles
>
> >>> --
> >>> Miles Sabin
> >>> tel: +44 (0)7813 944 528
> >>> skype:  milessabin
> >>>http://www.chuusai.com/
> >>>http://twitter.com/milessabin

Jorge, 

I think my translation needs some work/thinking.  I had the idea of handle/resource differently (see the ServerSocket example where the resource is the "ServerSocket" but the handle and what you want to execute against is the "Socket").

So we have two cases:

1) The opening/closing of a resource is done from a separate entity (the handle)
2) The resource is obtained from some other entity (and returned to the same)

My initial thought is that #2 could be done with appropriate use of closures, and the handle stuff can be reworked to look like #1 again.

Anyone have any other ideas on how to merge these concepts? 

- Josh

On Mon, Oct 5, 2009 at 5:16 AM, Jorge Ortiz <jorge.ortiz@gmail.com> wrote:

Based on my understanding of the original Scalax ManagedResource stuff, ManagedResource was not really meant to be a user-facing library. It was rather intended to be a library-writer's library. A user would rarely wrap resources inside of a ManagedResource themselves. Instead, user-facing libraries (like IO, Transactions, Locks, and anything else that needs to open a resource and ensure that it is closed after it's been used) would provide access to objects that were Managed. For a user to get access to a Managed object, they'd have to use it inside a for-comprehension.

For example, in scalax.io, printing the lines of a file was as simple as:

  val file: java.io.File = ...
  for (line <- file.lines)
    println(line)

Of course, a lot of this relied on implicit magic at the level of scalax.io, but the ARM component ensured that the file was properly opened and closed. At the user-level, there was no need to create a Managed* object, just to know that file.lines returned a ManagedSequence (now ManagedTraversable).

Now, as I understand the scalax ARM stuff, a ManagedResource has two central concepts: a Resource and a Handle. The Handle is the thing that needs to be opened/closed, and the Resource is the thing that a user actually wants to work with. In the basic, most common case, the Handle and the Resource are the same thing. The thing that needs to be open/closed is the same thing that the users wants to work with. This is called an UntranslatedManagedResource, for reasons that will become clear in a second.

A ManagedResource has three methods that are meant to be used by the library, not directly by a user. To open a ManagedResource means to obtain a Handle, to close a ManagedResource means to dispose of a Handle, and to translate means to go from a Handle to a raw Resource (hence Untranslated when a Handle and a Resource are the same thing). There is one method which is actually meant to be used by the user of a ManagedResource. It takes a function on a raw Resource and performs the full chain of opening  the ManagedResource to obtain the Handle, translating the handle to a raw Resource, invoke the user's function on the raw Resource, and closing the ManagedResource (disposing of the Handle).

As mentioned above, most ManagedResources are in fact UntranslatedManagedResources. This is the case for InputStreams, FileReaders, etc. In fact, in the scalax code, there were very few translated ManagedResources. An example of where you would want a translated resource was FileChannels. In this case, the Handle is an InputStream and the Resource is a FileChannel. The InputStream (Handle) needs to be opened and closed safely, but the user doesn't ever need to know that the InputStream even exists, the user only ever deals with the FileChannel. The user passes functions that operate on a FileChannel (the Resource), and the library takes care of opening an InputStream, translating the InputStream to a FileChannel, invoking the user's function on the FileChannel, and closing the InputStream.

Now, this is my understanding of the ARM code in the original scalax. I'm not sure exactly how much this applies to the new code Josh has in scala-arm. From a cursory inspection it seems like Josh is adding further laziness, such that you can stack user operations on ManagedResources and "force" them only once you're done. I'm also not sure how well the terminology I've described also applies to Josh's library. Josh, care to comment?

--j

On Sun, Oct 4, 2009 at 9:46 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

No problem!  Communication is what this is all about.

So...   That line is unecessarily complex due to the bloat of the java.io library.  Assuming a sane Scala.io library, it would look more like the following:

val input = ManagedResource(file.reader).toTraversable(_.lines)

input.foreach(println) //Prints all lines in the file...

Basically, the ManagedResource is defined on the inputstream from a connection (in this case to a file's "reader").  The API provides not only a mechanism of opening/closing resources but a "ManagedTraversable".  The idea here is that you have a collection that lives inside a resource.   The ManagedTraversable opens the resource before iteration and closes it afterwards.  Because Traversable implements off a "foreach" method, any kind of traversing method performed on ManagedTraversbale will open/close the resource.   This can be done lazily by using a view.

What I've tried to do is keep the simple while adding more powerful features.  i.e. for( handle <- ManagedResource(openResource)) { foo(handle) }   still works just fine.   However you have some additional advanced features if needed.

Another feature I tried to add is appending work to a resource to be performed when it is finally opened.   The map operator will defer a function to be applied against the resource when it is opened.  This allows you to convert between various java.io  classes before actually opening your resource.   My JavaBufferedreaderLineIterator requires a java.io.BufferedReader.   A Socket returns an InputStream.   Hence the "map" before the call to "lines".  

The library also has a notion of extracting data from within a resource.   The ManagedTraversable is a decent way to do this, but will continually try to open the resource if you use it more than once.   The "opt" and "either" methods on a TranslatedResource (i.e. after a map function) are used to extract whatever results the mapping function may generate.   This should be used to extract *safe* items from the resource (i.e. not iterators tied to the resource). 


Anyway, I hope this helps outline the thought process and how the additions to the library came to be.


- Josh


On Mon, Oct 5, 2009 at 12:33 AM, phausel <peter.hausel@gmail.com> wrote:

Hi Josh,

Thanks for getting back to me.

I understand this one below covers a more complex use case but for me
it does look heavyweight (especially compared to C#'s or java7's
solution) and also it's hard to tell where opening/closing is
happening:
val input = ManagedResource(connection.getInputStream).map( is => new
BufferedReader(new InputStreamReader(is))).toTraversable( br => new
JavaBufferedReaderLineIterator(br))

just a thought
Thanks Peter


On Oct 5, 1:24 am, Josh Suereth <joshua.suer...@gmail.com> wrote:
> Actually, you can accomplish the same thing in the API I've started as a
> sample discussion point.   The code is adapted from Scalax.IO and some
> production libraries I use at work, so I have some ideas of the complexities
> involved with ARM.
>
> The simple lightweight version is still there in my API, however I wanted to
> have depth when you need it.   Here's what the above blog sample would look
> like in my API
>
> for(pi <-ManagedResource(new PipedInputStream)){
>     for(po<-ManagedResource(new PipedOutputStream(pi))) {
>   po.write('A');
>   Console.println((pi.read).asInstanceOf[Char]);
>     }
>
> }
>
> vs. the original
>
> for(pi<-new ManagedResource(new PipedInputStream);
>     po<-new ManagedResource(new PipedOutputStream(pi))) {
>   po.write('A');
>   Console.println((pi.read).asInstanceOf[Char]);
>
> }
>
> So now my question is this:  What's "heavyweight" about my API?
>
>
>
> On Sun, Oct 4, 2009 at 2:09 PM, Peter Hausel <peter.hau...@gmail.com> wrote:
> > Hi Josh,
> > you might find this useful:
> >http://www.saager.org/2007/12/30/automatic-resource-management-blocks...
>
> > p
>
> > On Sun, Oct 4, 2009 at 5:48 PM, Josh Suereth <joshua.suer...@gmail.com>wrote:
>
> >> Updated:  http://wiki.github.com/jsuereth/scala-arm/socketexample
>
> >> Also remember that this library is just a starting point for discussions.
> >> I'm hoping to outline the issue areas in an ARM API so we can address them
> >> and make a final solution.
>
> >> So far I think we've identified that demarking the opening/closing of
> >> resources is important, as well as finding a way to ensure the user only
> >> extracts "non-resource-bound" items from the ARM block.
>
> >> Anyway, please propose alternate solutions.  Would you like access to the
> >> wiki?
>
> >> - Josh
>
> >> On Sun, Oct 4, 2009 at 11:26 AM, Miles Sabin <mi...@milessabin.com>wrote:
>
> >>> On Sun, Oct 4, 2009 at 4:16 PM, Josh Suereth <joshua.suer...@gmail.com>
> >>> wrote:
> >>> > Here's a socket example.
>
> >>> To be honest, that looks surprisingly heavyweight and like Daniel it's
> >>> also not immediately clear to me where the open/close bracketing
> >>> comes. Could you add comments to the code indicating those points?
>
> >>> Cheers,
>
> >>> Miles
>
> >>> --
> >>> Miles Sabin
> >>> tel: +44 (0)7813 944 528
> >>> skype:  milessabin
> >>>http://www.chuusai.com/
> >>>http://twitter.com/milessabin

On Mon, Oct 5, 2009 at 10:16 AM, Jorge Ortiz wrote:
> Now, this is my understanding of the ARM code in the original scalax. I'm
> not sure exactly how much this applies to the new code Josh has in
> scala-arm. From a cursory inspection it seems like Josh is adding further
> laziness, such that you can stack user operations on ManagedResources and
> "force" them only once you're done.

That's how it looks to me too. I don't understand what that initial
stack of operations adds over and above moving them into the body of
the corresponding for comprehension. Eg. from the echo server example,
why this,

val input = ManagedResource(connection.getInputStream).
map( is => new BufferedReader(new InputStreamReader(is))).
toTraversable( br => new JavaBufferedReaderLineIterator(br))
val output = ManagedResource(connection.getOutputStream).
map( os => new PrintWriter(new BufferedWriter(new OutputStreamWriter(os))))
for (outStream <- output; line <- input) {
println("Server returning: " + line)
outStream.println(line)
outStream.flush()
}

rather than this,

for {
os <- managed(connection.getOutputStream)
outStream = new PrintWriter(new BufferedWriter(new OutputStreamWriter(os))
is <- managed(connection.getInputStream)
line <- new JavaBufferedReaderLineIterator(new BufferedReader(new
InputStreamReader(is)))
} {
println("Server returning: " + line)
outStream.println(line)
outStream.flush()
}

where managed() gives us an appropriately pimped wrapper around the
underlying stream (I assume this second version is quite close to the
original in scalax.io).

Josh, could you explain the relative merits of these two?

Cheers,

Miles

They're actually mostly the same, it's just a matter of how you write the sample.  There's no particular reason I chose my version besides that's how I wrote it.   The library actually would support your syntax (just with ManagedResource instead of managed).

I was more trying to show off all the features at once instead of aiming for the most concise example.  Also I really want a good demonstration of ManagedTraversable (or I can add in File.lines) so that you can see how it's safe to pass this "collection" outside of an ARM block.

- Josh

On Mon, Oct 5, 2009 at 11:01 AM, Miles Sabin <miles@milessabin.com> wrote:

On Mon, Oct 5, 2009 at 10:16 AM, Jorge Ortiz <jorge.ortiz@gmail.com> wrote:
> Now, this is my understanding of the ARM code in the original scalax. I'm
> not sure exactly how much this applies to the new code Josh has in
> scala-arm. From a cursory inspection it seems like Josh is adding further
> laziness, such that you can stack user operations on ManagedResources and
> "force" them only once you're done.

That's how it looks to me too. I don't understand what that initial
stack of operations adds over and above moving them into the body of
the corresponding for comprehension. Eg. from the echo server example,
why this,

 val input = ManagedResource(connection.getInputStream).
   map( is => new BufferedReader(new InputStreamReader(is))).
   toTraversable( br => new JavaBufferedReaderLineIterator(br))
 val output = ManagedResource(connection.getOutputStream).
   map( os => new PrintWriter(new BufferedWriter(new OutputStreamWriter(os))))
 for (outStream <- output; line <- input) {
   println("Server returning: " + line)
   outStream.println(line)
   outStream.flush()
 }

rather than this,

 for {
   os   <- managed(connection.getOutputStream)
   outStream = new PrintWriter(new BufferedWriter(new OutputStreamWriter(os))
   is   <- managed(connection.getInputStream)
   line <- new JavaBufferedReaderLineIterator(new BufferedReader(new
InputStreamReader(is)))
 } {
   println("Server returning: " + line)
   outStream.println(line)
   outStream.flush()
 }

where managed() gives us an appropriately pimped wrapper around the
underlying stream (I assume this second version is quite close to the
original in scalax.io).

Josh, could you explain the relative merits of these two?

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin