Scala IO report on current state of the API implemented by Paul P.

On Sun, Oct 11, 2009 at 7:15 PM, Jesse Eichar wrote:
> File/Directory Split
>
> At the moment the biggest departure from the NIO filesystem API is that in
> Scala IO there is a concept of Directory and File.  Where Directory can
> contain Files and Directories but not data and Files can contain data but
> not Files or Directories. In NIO there is only Path.  I personally like the
> split but I believe that is must be possible to be both a File and a
> Directory, simply to support filesystems that may have that concept.

I'd like to hear something about how such a split copes with changes
on the underlying filesystem, eg.,

touch foo
rm foo
mkdir foo

As far as the filesystem is concerned foo was a file and is now a directory.

Should this be reflected in a File/Directory API and if so how?

Cheers,

Miles

On Sun, Oct 11, 2009 at 1:15 PM, Jesse Eichar <jeichar.w@gmail.com> wrote:

must be possible to be both a File and a Directory, simply to support filesystems that may have that concept.  

Like zip/jar files, which can be viewed as both?

On Sun, Oct 11, 2009 at 10:14 PM, Miles Sabin wrote:
> On Sun, Oct 11, 2009 at 7:15 PM, Jesse Eichar wrote:
>> File/Directory Split
>>
>> At the moment the biggest departure from the NIO filesystem API is that in
>> Scala IO there is a concept of Directory and File.  Where Directory can
>> contain Files and Directories but not data and Files can contain data but
>> not Files or Directories. In NIO there is only Path.  I personally like the
>> split but I believe that is must be possible to be both a File and a
>> Directory, simply to support filesystems that may have that concept.
>
> I'd like to hear something about how such a split copes with changes
> on the underlying filesystem, eg.,
>
>  touch foo
>  rm foo
>  mkdir foo
>
> As far as the filesystem is concerned foo was a file and is now a directory.
>
> Should this be reflected in a File/Directory API and if so how?
>
> Cheers,
>
>
> Miles
>
> --
> Miles Sabin
> tel: +44 (0)7813 944 528
> skype:  milessabin
> http://www.chuusai.com/
> http://twitter.com/milessabin
>

It gets worse though, directories are not the only container. In many
cases, it's possible for something to be both a file and a container
at the same time.

Off the top of my head, some non-directory containers are:

zip/rar/gz/tar/etc.
alternate file streams in NTFS and HFS
shell folders (such as control panel) in MS Windows

Once you get beyond the most basic scenarios, the distinction between
file and directory becomes increasingly artificial. All we can really
say with any guarantees of consistency is that we have a path and a
number of possible roles that may or may not be applicable.

All this doesn't invalidate Paul's API. At any point, you probably
want to work with a path as if it represented a file, or as if it
represented a directory, but exceedingly rarely both. I don't think
Paul's API prevents you from having both a File and a Directory
containing the same path, so there's nothing you can't do, and it's
probably simpler in the 99.9% case.

2009/10/11 Kevin Wright :
> On Sun, Oct 11, 2009 at 10:14 PM, Miles Sabin wrote:
>> On Sun, Oct 11, 2009 at 7:15 PM, Jesse Eichar wrote:
>>> File/Directory Split
>>>
>>> At the moment the biggest departure from the NIO filesystem API is that in
>>> Scala IO there is a concept of Directory and File.  Where Directory can
>>> contain Files and Directories but not data and Files can contain data but
>>> not Files or Directories. In NIO there is only Path.  I personally like the
>>> split but I believe that is must be possible to be both a File and a
>>> Directory, simply to support filesystems that may have that concept.
>>
>> I'd like to hear something about how such a split copes with changes
>> on the underlying filesystem, eg.,
>>
>>  touch foo
>>  rm foo
>>  mkdir foo
>>
>> As far as the filesystem is concerned foo was a file and is now a directory.
>>
>> Should this be reflected in a File/Directory API and if so how?
>>
>> Cheers,
>>
>>
>> Miles
>>
>> --
>> Miles Sabin
>> tel: +44 (0)7813 944 528
>> skype:  milessabin
>> http://www.chuusai.com/
>> http://twitter.com/milessabin
>>
>
> It gets worse though, directories are not the only container.  In many
> cases, it's possible for something to be both a file and a container
> at the same time.
>
> Off the top of my head, some non-directory containers are:
>
> zip/rar/gz/tar/etc.
> alternate file streams in NTFS and HFS
> shell folders (such as control panel) in MS Windows
>
> Once you get beyond the most basic scenarios, the distinction between
> file and directory becomes increasingly artificial.  All we can really
> say with any guarantees of consistency is that we have a path and a
> number of possible roles that may or may not be applicable.
>

On Sun, Oct 11, 2009 at 11:15 PM, Ricky Clarkson
wrote:
> All this doesn't invalidate Paul's API.  At any point, you probably
> want to work with a path as if it represented a file, or as if it
> represented a directory, but exceedingly rarely both.  I don't think
> Paul's API prevents you from having both a File and a Directory
> containing the same path, so there's nothing you can't do, and it's
> probably simpler in the 99.9% case.

I didn't say it was impossible, I said I wanted to see the details.

Cheers,

Miles

On Sun, Oct 11, 2009 at 11:15 PM, Ricky Clarkson
wrote:
> All this doesn't invalidate Paul's API.  At any point, you probably
> want to work with a path as if it represented a file, or as if it
> represented a directory, but exceedingly rarely both.  I don't think
> Paul's API prevents you from having both a File and a Directory
> containing the same path, so there's nothing you can't do, and it's
> probably simpler in the 99.9% case.

At the application level I'd agree, but I think a library has to
seriously consider the other 0.1% and do so in a consistent fashion.

Files and directories are also not the only things you'll find in a
filesystem; there's also symlinks and devices. Not to mention other
uses of a path, such as the root of a backup set, or as a mount-point.

Of course, you can translate between any of these things be extracting
the path as a string from one type of reference and using it to
construct another kind, but then we're dropping back to use strings as
the lowest common denominator - which kind of defeats the point behind
static typing.

All of these are simply roles that can be allocated to a path, so it
makes a lot of sense to build any IO library with paths as the central
concept and other functionality as layers over this.

> 2009/10/11 Kevin Wright :
>> On Sun, Oct 11, 2009 at 10:14 PM, Miles Sabin wrote:
>>> On Sun, Oct 11, 2009 at 7:15 PM, Jesse Eichar wrote:
>>>> File/Directory Split
>>>>
>>>> At the moment the biggest departure from the NIO filesystem API is that in
>>>> Scala IO there is a concept of Directory and File.  Where Directory can
>>>> contain Files and Directories but not data and Files can contain data but
>>>> not Files or Directories. In NIO there is only Path.  I personally like the
>>>> split but I believe that is must be possible to be both a File and a
>>>> Directory, simply to support filesystems that may have that concept.
>>>
>>> I'd like to hear something about how such a split copes with changes
>>> on the underlying filesystem, eg.,
>>>
>>>  touch foo
>>>  rm foo
>>>  mkdir foo
>>>
>>> As far as the filesystem is concerned foo was a file and is now a directory.
>>>
>>> Should this be reflected in a File/Directory API and if so how?
>>>
>>> Cheers,
>>>
>>>
>>> Miles
>>>
>>> --
>>> Miles Sabin
>>> tel: +44 (0)7813 944 528
>>> skype:  milessabin
>>> http://www.chuusai.com/
>>> http://twitter.com/milessabin
>>>
>>
>> It gets worse though, directories are not the only container.  In many
>> cases, it's possible for something to be both a file and a container
>> at the same time.
>>
>> Off the top of my head, some non-directory containers are:
>>
>> zip/rar/gz/tar/etc.
>> alternate file streams in NTFS and HFS
>> shell folders (such as control panel) in MS Windows
>>
>> Once you get beyond the most basic scenarios, the distinction between
>> file and directory becomes increasingly artificial.  All we can really
>> say with any guarantees of consistency is that we have a path and a
>> number of possible roles that may or may not be applicable.
>>
>
>
>
> --
> Ricky Clarkson
> Java Programmer, AD Holdings
> +44 1565 770804
> Skype: ricky_clarkson
> Google Talk: ricky.clarkson@gmail.com
>

Addressing Miles:
An excellent point about File/Directory.  This is the most controversial part of the API.  For now lets assume we are going to keep the File/Directory abstraction and see if we can answer satisfy our questions using this abstraction.  If not we will probably go with a NIO style API with Path only. 

For purposes of this discussion let us define a File as a filesystem object that can contain bytes and can optionally be read and/or written to.  A Directory can contain other filesystem objects.  Additionally, File, Directory and Path are simply views on a concrete filesystem object.  As a consequence a filesystem object is not a File per-say, rather it can be viewed as a File. 

Paul has kindly added an isValid method on Path which can be used to indicate whether a particular view isValid. 

Now with regards to your example suppose you do:

val path = Path("foo")  -> this gives a path view
val file = path.toFile  -> this gives a file view (a file is a subclass of Path)
val dir = path.toDirectory -> this gives a file view.

So before touch is called, path, file and dir are valid because the file does not exist and therefore all views are valid.  (That is current implementation.  I can also see a case for path isValid but not file or dir.)

Along come touch foo.  Now path and file are valid but not dir.

rm foo
mkdir foo

No surprise that path and dir are valid but not dir.

Now to Kevin Wrights point about having to consider Symlinks and Devices. 

Now assuming we keep Directory and Path there are two main possibilities>\:

1.  Both of these can be represented as either a File or Directory.  A symlink can be either a file or directory and treated as such.  With the current implementation that is how a symlink would be treated.  Path has an isSymlink method and when we go Java 7 we can have a createSymlink method as well.
    As for Devices.  Again I think it can be viewed as a Directory.  Even the NIO API (as far as I know) does not permit mounting and unmounting of devices. 
    In the NIO API all Path objects have FileAttributes.  These are designed to be extensible and platform specific.  I had planned on adding FileAttributes to future versions of Path.  This version cannot have it because we need the full NIO API to be able to support them.  Any metadata or information beyond the basic Path API would be accessible through the FileAttributes API.  While not the nicest API it does cover the rest of the 1% use-cases and
2.  We could add other alternate views (a symlink object and device object).  My problem with this is that we have to create a completely new framework for being able to define new views for a given filesystem.  So my feeling is to reusethe NIO FileAttributes framework for these sorts of corner cases.

If we have file and directory views we can add more in the future but I suspect that it is unnecessary since they encapsulate the vast majority of use-cases and FileAttributes cover the rest.

Advantages of having File and Directory
- The extra classes allow one to be more expressive when defining APIs.  For example by saying a return type is a File you are specifying additional metadata that the returned object will have data associated with it.
- We can have extractors for matching files/directories in a very clean manner.
- File and Directory are well understood by most people and in the vast majority of filesystems are work as expected

Cons
- Cases where a Path be both a File and a Directory is slightly more difficult to understand because you have to think of them as views not representations of the underlying filesystem object.  (Similar consideration for files that are deleted and have directories created for them.)
- File/Directory are commonly expected to be representative objects instead of views.  (This is easily fixed by a name change, FileView for example).

Jesse

In my first email I had several open questions.  Are there any opinions on them yet?

Open Questions:
- Do we need File and Directory to be traits? We can convert between different views by using the toFile and toDirectory if a Path is both a File and a Directory.
  My Answer: No they can be classes for binary compatibility.
- Do we want Java 6 Scala IO to be binary compatible with Java 7 Scala IO?
  My Answer:  Yes
- Shall we have a general Exception that is used (maybe IOException) for Java 6 Scala IO and in Java 7 use the narrower exception or have return codes in Either objects
  My Answer:  Use a general Exception where we can no way of determining the problem.  I have found some cases where we can have a more specific Exception in Java 6 so I can throw some specific Exceptions under certain circumstances.

Jesse



On Sun, Oct 11, 2009 at 11:15 AM, Jesse Eichar <jeichar.w@gmail.com> wrote:

Hi,

I have been reviewing the Scala IO implementation in the most recent version I have been able to find and have made several observations about the current state but more about how we can move forward with the implementation.  I have several open issues in a bullet list at the end.  

I finished several days ago but have limited access to the internet so I have only now emailed it.  Since that time I have been going through the Scala IO implementation published by Paul and am adding javadocs to all methods.  I am also adding notes and questions in the javadocs where I think more discussion is needed.  I want to get a draft API finished and will publish the javadocs for discussion in the next week.  

File/Directory Split

At the moment the biggest departure from the NIO filesystem API is that in Scala IO there is a concept of Directory and File.  Where Directory can contain Files and Directories but not data and Files can contain data but not Files or Directories. In NIO there is only Path.  I personally like the split but I believe that is must be possible to be both a File and a Directory, simply to support filesystems that may have that concept.  Further I think that when a Path object is created (especially in later versions when many filesystems are supported) the object must not require any filesystem access on creation.  The reason for this requirement is for slow filesystems to be efficiently accessed.  The attributes on path and resolution to a File and/or Directory must be lazy.  At least this must be the case in the spec.  For version 1 I think we can base it on File and just go with it but for more advanced implementations and the API spec we need this realization.  

This last point is a small problem with a current implemenation.  In the current implementation when a Path object is instantiated a Directory or File object is instantiated.  That is fine for "normal" filesystems but can be problematic for certain filesystems.  I would like to change that implementation detail so bad habits are not created.  Instead perhaps we could define extractors so that we could match on File and Directory.  Something like:

path match {

 case Hybrid(file,directory) => file.lines.length + directory.files.length

 case File(file) => file.lines.length

 case Directory(directory) => directory.files.length

}

Do these extractors add very much? It does cut down on boiler plate a bit:

path match {

 case path if(path.isFile && path.isDirectory) => {

   val (file, directory) = (path.toFile, path.toDirectory)

    file.lines.length + directory.files.length

 }

 case path if(path.isFile) => {

   val file = path.toFile

   file.lines.length

 }

 case path if(path.isDirectory) => {

   val directory = path.toDirectory

   directory.files.length

 }

}

Comparing those two examples I would say the extractors are worth it.

Backwards Compatibility

I personally envision at least 2 versions of the API a Java 6 version and a Java 7 version.  The Java 6 API will be reduced in scope because of technical limits imposed by Java 6.  The question is do we want the two versions to be binary compatible.  If that is a desire then File, Path and Directory must be abstract classes and not traits because (as I understand it) adding methods to traits breaks binary compatibility.  A consequence of this would be to remove the possibility of having a File be both a Directory and a File.

Exceptions

Exceptions will be an issue in the Java 6 Scala IO.  The java.io.File API does not indicate what problem has occurred normally when an operation fails.  The normal failure condition is indicated by a return value being false.  The result is in the Java 6 API we cannot provide the same exceptions as we can in Java 7.  I can see two solutions.  

If a failure occurs throw a super class of all the NIO exceptions so that we can throw a refinement of the exception when we provide the full Java 7 based API

We return a failure code (possibly as part of an Either object) and in Java 6 API we have a generic return code which we remove in Java 7 API in favour of the full set of code.

I am in favour of the first option for several reasons.  

It allows seemless migration from Java 6 API and 

It allows scripts to be simpler because the failure case can simply be ignored

It allows blocks of IO to be handled and all failure cases to be handled together.

ARM

We need it :).  I am going to defer this conversation a little bit until the ARM API is closer to completion.  I am watching that conversation closely and certainly want to incorporate that into my API.

Open Questions: 

• Is there a need for the hybrid extractor?  I think that "hybrid" is so rare that if you need to match on that you should use a guard that checks if the path is both a File and a Directory.
• Do we need File and Directory to be traits?  Can we just ignore the case where a Path can be both a File and a Directory and use the Java API if we want to use that kind of filesystem?
• Do we want Java 6 Scala IO to be binary compatible with Java 7 Scala IO?
• Shall we have a general Exception that is used (maybe IOException) for Java 6 Scala IO and in Java 7 use the narrower exception or have return codes in Either objects

Summary

• A Filesystem element must be able to be both a File and a Directory if supported by the filesystem type
• A Path object must be very light and lazy for performance.  
• Extractors can be created to assist in converting a Path to File and Directory
• We will have 2 APIs, Java 6 Scala IO and Java 7 Scala IO. 
• In the Java 6 Scala IO we can use a general Exception to indicate failure that is a superclass of all the Java 7 narrower exceptions.

Jesse

On Wed, Oct 14, 2009 at 6:59 AM, Jesse Eichar wrote:
> Addressing Miles:
> Paul has kindly added an isValid method on Path which can be used to
> indicate whether a particular view isValid.

I'm don't see how an isValid method could be used in the way that it's
presumably intended.

There's an inherent race condition here with other processes which
might be modifying the filesystem independently, so, for example,

val p : Path = ...
val d = p.toDirectory
if (d.isValid) {
// Filesystem is mutated externally here, p now denotes a file

// Do stuff assuming d is a valid directory ... boom
}

(there's a whole class of application security issues which exploit
this kind of filesystem time-of-check/time-of-use race condition).

Given that we can't portably synchronize wrt filesystem mutation, it
follows that all file-specific operations on File and all
directory-specific operations on Directory would have to check for
validity, and users of those operations would have to code for the
possiblity of failure due to the path being of the wrong kind.

But if that's the case, then I really don't see what value there is to
be gained by splitting these operations off into separate traits.

Cheers,

Miles

On Wed, Oct 14, 2009 at 7:00 AM, Jesse Eichar wrote:
> - Shall we have a general Exception that is used (maybe IOException) for
> Java 6 Scala IO and in Java 7 use the narrower exception or have return
> codes in Either objects

We should reuse Java exceptions whereever possible,
java.io.IOException as the general case.

Cheers,

Miles

Hi Miles,

I agree that there is an inherent race condition but I don't think that using only Path helps that in any way.  Consider the equivalent snippet when are using only path:


val p:Path = ...
if(p.isDirectory) { 
// Filesystem is mutated external an now is denotes a file

// do stuff assuming p is a valid file.
}

There is no way around this without using Java 7's NIO SecureDirectoryStream API.

So for me this is not an argument to remove the File/Directory classes.  In my mind they make more explicit certain assumptions made in the program.  Assumptions that would be more difficult to recognize if we only have Path. 

Example:

def saveTo(string:String, file:File) = file.write(string)

or with only Path

def saveTo(string:String, file:Path) = file.write(string)

The first alternative offers the compiler and the developer with more information about assumptions of what 'file' must be.  If we have only path there is more likely to have to be checks to verify that file is actually a file.

An argument I see is:  Why only File and Directory?  Why not:

ReadOnlyFile, WritableFile, Directory, FileSymlink, DirectorySymlink, ExecutableFile, etc...

The answer: Because File and Directory encapsulate the most useful set of semantics without an explosion of little classes with limited usefulness.

Jesse

On Wed, Oct 14, 2009 at 1:48 AM, Miles Sabin <miles@milessabin.com> wrote:

On Wed, Oct 14, 2009 at 6:59 AM, Jesse Eichar <jeichar.w@gmail.com> wrote:
> Addressing Miles:
> Paul has kindly added an isValid method on Path which can be used to
> indicate whether a particular view isValid.

I'm don't see how an isValid method could be used in the way that it's
presumably intended.

There's an inherent race condition here with other processes which
might be modifying the filesystem independently, so, for example,

 val p : Path = ...
 val d = p.toDirectory
 if (d.isValid) {
   // Filesystem is mutated externally here, p now denotes a file

   // Do stuff assuming d is a valid directory ... boom
 }

(there's a whole class of application security issues which exploit
this kind of filesystem time-of-check/time-of-use race condition).

Given that we can't portably synchronize wrt filesystem mutation, it
follows that all file-specific operations on File and all
directory-specific operations on Directory would have to check for
validity, and users of those operations would have to code for the
possiblity of failure due to the path being of the wrong kind.

But if that's the case, then I really don't see what value there is to
be gained by splitting these operations off into separate traits.

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

On Wed, Oct 14, 2009 at 1:52 AM, Miles Sabin <miles@milessabin.com> wrote:

On Wed, Oct 14, 2009 at 7:00 AM, Jesse Eichar <jeichar.w@gmail.com> wrote:
> - Shall we have a general Exception that is used (maybe IOException) for
> Java 6 Scala IO and in Java 7 use the narrower exception or have return
> codes in Either objects

We should reuse Java exceptions whereever possible,
java.io.IOException as the general case.

I agree. In Java 7 NIO API there are several new exceptions all of which inherit from the IOException, IllegalStateException or UnsupportedOperationException.  For the Java 7 version I will limit the exceptions to these three versions and the Java 7 version will have the full suite of Exceptions.

Jesse

I think that's the right way to go for java/scala interoperability.  If we need new exceptions, we can extends from java.io.Exception as needed.

Jesse, I also think the current split file/directory approach you're outlining is a good way to go.  We want things to simple in the simple case and possible in the difficult cases.

In terms of binary compatability... Good luck!   Scala seems to be taking the C++ approach of making binary compatabilty tough and forcing interoperability through the "less powerful" language (i.e. Java).  This means we should code to interfaces written in Java (i.e. have a scala-io-compatability.jar artifact that contains the interfaces you code to, and then implementations for java6/java7).

- Josh

On Wed, Oct 14, 2009 at 1:34 PM, Jesse Eichar <jeichar.w@gmail.com> wrote:

On Wed, Oct 14, 2009 at 1:52 AM, Miles Sabin <miles@milessabin.com> wrote:

On Wed, Oct 14, 2009 at 7:00 AM, Jesse Eichar <jeichar.w@gmail.com> wrote:
> - Shall we have a general Exception that is used (maybe IOException) for
> Java 6 Scala IO and in Java 7 use the narrower exception or have return
> codes in Either objects

We should reuse Java exceptions whereever possible,
java.io.IOException as the general case.

I agree. In Java 7 NIO API there are several new exceptions all of which inherit from the IOException, IllegalStateException or UnsupportedOperationException.  For the Java 7 version I will limit the exceptions to these three versions and the Java 7 version will have the full suite of Exceptions.

Jesse

On Wed, Oct 14, 2009 at 6:31 PM, Jesse Eichar wrote:
> I agree that there is an inherent race condition but I don't think that
> using only Path helps that in any way.  Consider the equivalent snippet when
> are using only path:
>
>
> val p:Path = ...
> if(p.isDirectory) {
> // Filesystem is mutated external an now is denotes a file
>
> // do stuff assuming p is a valid file.
> }

Well, this isn't quite what I had in mind.

To be clear, I'm not opposed in principle to the idea of there being
distinct types for files and directories: what I'm opposed to is
either of these extending the path type (which I believe both do
currently).

The problem with File <: Path and Directory <: Path is that it
encourages the view that a path can be fixed as a file path or
directory path (or a hybrid), and in general, on most filesystems,
that's simply not true.

If we remove that subtype relationship then we're just fine: the
file/dir (via their underlying OS handles/descriptors) and the path
can go their separate ways.

So, for your example above, I'd like to see something like,

val p : Path = ...
val optDir = p.directory
optDir match {
case Some(d) => // It was a directory, we now have a path-independent handle
case None =>
}

This is more or less what NIO2 does, with DirectoryStreams for
Directory and {Input,Output}Streams and Channels for File.

NIO2's Path has a slightly confusingly named FileRef as it's
superclass, but if you look at it closely you'll see that it only
defines path-related methods (ie. attribute access and file handle
creation).

Cheers,

Miles