Re: scala.io Path/File/Directory hierarchy

On Mon, 2009-09-07 at 19:13 +0100, Miles Sabin wrote:
> On Mon, Sep 7, 2009 at 7:07 PM, Ismael Juma wrote:
> > On Mon, 2009-09-07 at 18:34 +0100, Miles Sabin wrote:
> >> I'm saying that it should use NIO2. We want an API that's idiomatic
> >> for Scala, obviously.
> >
> > What is your suggestion for Scala 2.8.0 then? Do nothing?
>
> Given that the Java APIs are available,

The cumbersome Java APIs, you mean (not even NIO2).

> then yes, I think that waiting
> is preferable do doing something half-baked and having to redo it (or
> having it ignored in favour of the Java APIs forever).

Thankfully no-one is proposing the introduction of half-baked APIs. :)

> I don't think we can afford many false starts on a standard library.

I don't recommend it, but seems to have worked well for Java,
traditional IO, NIO and NIO2. ;)

Best,
Ismael

On Mon, Sep 7, 2009 at 7:23 PM, Ismael Juma wrote:
>> then yes, I think that waiting
>> is preferable do doing something half-baked and having to redo it (or
>> having it ignored in favour of the Java APIs forever).
>
> Thankfully no-one is proposing the introduction of half-baked APIs. :)

The scala.io.{ Directory, Path } that we've just acquired makes both
of the performance mistakes that I mentioned earlier. I would have
preferred it if that code hadn't been committed.

Cheers,

Miles

Has anyone here got strong experience in IO from other functional languages that could be relevant here?  Probably not the extreme pure monad-based stuff though... that might be overkill :)

On Mon, Sep 7, 2009 at 7:30 PM, Miles Sabin <miles@milessabin.com> wrote:

On Mon, Sep 7, 2009 at 7:23 PM, Ismael Juma<mlists@juma.me.uk> wrote:
>> then yes, I think that waiting
>> is preferable do doing something half-baked and having to redo it (or
>> having it ignored in favour of the Java APIs forever).
>
> Thankfully no-one is proposing the introduction of half-baked APIs. :)

The scala.io.{ Directory, Path } that we've just acquired makes both
of the performance mistakes that I mentioned earlier. I would have
preferred it if that code hadn't been committed.

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

On Mon, 2009-09-07 at 19:30 +0100, Miles Sabin wrote:
> On Mon, Sep 7, 2009 at 7:23 PM, Ismael Juma wrote:
> >> then yes, I think that waiting
> >> is preferable do doing something half-baked and having to redo it (or
> >> having it ignored in favour of the Java APIs forever).
> >
> > Thankfully no-one is proposing the introduction of half-baked APIs. :)
>
> The scala.io.{ Directory, Path } that we've just acquired makes both
> of the performance mistakes that I mentioned earlier.

Well, we're still some time away from Scala 2.8.0, right? As far as I
know, Paul is still working on those APIs.

And to comment directly on the performance mistakes

- "iterator-style API for directory contents": there is no way to avoid
that with the current Java APIs as far as I know (it's important to
expose it in a way that can be fixed once NIO2 becomes available, of
course). In other words, no worse than Java at this point.

- "For instance NIO2 distinguishes files from directories via separately
queryable attributes rather than via Path subtypes because because,
eg., network filesystems make attribute queries extremely expensive.":
There are no checks when Directory instances are created, so there is no
performance mistake here.

> I would have preferred it if that code hadn't been committed.

Out of curiosity, how is that worse than the new collections API or the
refactoring of the Eclipse plugin? Both reached trunk before they were
working fully as far as I remember. Given Subversion's weak support for
branching and merging, it's not surprising really.

Best,
Ismael

On Mon, Sep 7, 2009 at 8:20 PM, Ismael Juma wrote:
> Out of curiosity, how is that worse than the new collections API or the
> refactoring of the Eclipse plugin? Both reached trunk before they were
> working fully as far as I remember.

The new collections API grew out of months of experimentation on a
branch and came with a design document. Neither it not the process
were perfect, but they're closer to the model I'd like to see for
things which we're going to be relying on for years to come.

Cheers,

Miles

On Mon, Sep 07, 2009 at 07:30:51PM +0100, Miles Sabin wrote:
> The scala.io.{ Directory, Path } that we've just acquired makes both
> of the performance mistakes that I mentioned earlier. I would have
> preferred it if that code hadn't been committed.

I lack the spirit to resist this kind of stop energy. I will revert
what's checked in to the degree that I can do so without too much pain,
and we can all look forward to miles's implementation (not for 2.8 it
sounds like, but whenever he can find the time.)

On Mon, Sep 7, 2009 at 9:38 PM, Paul Phillips wrote:
> On Mon, Sep 07, 2009 at 07:30:51PM +0100, Miles Sabin wrote:
>> The scala.io.{ Directory, Path } that we've just acquired makes both
>> of the performance mistakes that I mentioned earlier. I would have
>> preferred it if that code hadn't been committed.
>
> I lack the spirit to resist this kind of stop energy.  I will revert
> what's checked in to the degree that I can do so without too much pain,
> and we can all look forward to miles's implementation (not for 2.8 it
> sounds like, but whenever he can find the time.)

Paul, for goodness sake stop threatening to throw your toys out of the
pram whenever you're on the receiving end of the slightest bit of
(constructive) criticism.

A standard library isn't just a technical exercise, it's also an
exercise in compromise and consensus building.

Cheers,

Miles

Hi,

Hopefully I don't come across as a pompous ass in the email :).  Its not my intention.  I really like the work I have seen done in scala.io and am using all of it and I can tell you all that it is a dramatic improvement over the standard Java API which is one of the biggest messes in the Java library.  (Ok maybe overstated but I really don't like using it).  I would really like to see this work continue a pace.   I have read a bit on JSR-208 (far from an expert) and the current APIs are not so far off.

What I would like to see is a detailed criticism of the API as it stands now and where the problems are.   Some of the issues I have seen raised are:

* Iterators need to be used so that the API can handle very large directories with many files. 
 * This is not a problem with the API as it uses iterators.  Perhaps it is a problem with the implementation but implementation can change overtime, the API cannot (so much)
* NIO2 distinguishes files from directories via separately queryable attributes rather than via Path subtypes because because, eg., network filesystems make attribute queries extremely expensive.
  * This I think is a fair criticism,  it is also an ugly API.  Is there a comprimise?

Those are the main issues I have seen raised.  I like those and think they are good.  I take it on myself to read the full 203 spec tomorrow and comeback with similar issues in a couple days.

Hoping to be helpful :)

Jesse

On Mon, Sep 7, 2009 at 11:00 PM, Miles Sabin <miles@milessabin.com> wrote:

On Mon, Sep 7, 2009 at 9:38 PM, Paul Phillips<paulp@improving.org> wrote:
> On Mon, Sep 07, 2009 at 07:30:51PM +0100, Miles Sabin wrote:
>> The scala.io.{ Directory, Path } that we've just acquired makes both
>> of the performance mistakes that I mentioned earlier. I would have
>> preferred it if that code hadn't been committed.
>
> I lack the spirit to resist this kind of stop energy.  I will revert
> what's checked in to the degree that I can do so without too much pain,
> and we can all look forward to miles's implementation (not for 2.8 it
> sounds like, but whenever he can find the time.)

Paul, for goodness sake stop threatening to throw your toys out of the
pram whenever you're on the receiving end of the slightest bit of
(constructive) criticism.

A standard library isn't just a technical exercise, it's also an
exercise in compromise and consensus building.

Cheers,


Miles

--
Miles Sabin
tel: +44 (0)7813 944 528
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

It's not a threat miles. I don't know which part of what you do you
believe is constructive, but IO is not interesting enough to me to fade
your endless supply of unspecific sniping. You are far better equipped
to outfit scala with an IO library than I am, and I'm sure it'll be
received with at least as much enthusiasm as NIO was.

On Mon, Sep 7, 2009 at 10:23 PM, Paul Phillips wrote:
> I don't know which part of what you do you believe is constructive,

The bit referring to prior work on NIO2 which we should be taking more
notice of. And the emphasis on inclusiveness, compromise and consensus
building.

Cheers,

Miles

On Mon, 2009-09-07 at 23:22 +0200, Jesse Eichar wrote:
> * Iterators need to be used so that the API can handle very large
> directories with many files.
> * This is not a problem with the API as it uses iterators. Perhaps
> it is a problem with the implementation but implementation can change
> overtime, the API cannot (so much)

Right, but as I stated in the other email, there is no way to change
this now without either using native code or the unreleased NIO2. As you
say, the implementation can easily be changed in the future once we have
NIO2 as a dependency (even if simply as an optional dependency).

> * NIO2 distinguishes files from directories via separately queryable
> attributes rather than via Path subtypes because because, eg., network
> filesystems make attribute queries extremely expensive.
> * This I think is a fair criticism, it is also an ugly API. Is
> there a comprimise?

As I stated in the other email, the performance criticism is not
appropriate because no checks are done when a Directory instance is
created. Given the current implementation, more appropriate criticism
would be that it's misleading to call it a Directory since we don't know
whether it is or it is not. I think there is room for improvement over
the current solution, as there is room for improvement over the NIO2
solution.

> Those are the main issues I have seen raised. I like those and think
> they are good. I take it on myself to read the full 203 spec tomorrow
> and comeback with similar issues in a couple days.
>
> Hoping to be helpful :)

That would be helpful, particularly if you co-ordinate with the people
who are working in this area.

Best,
Ismael

On Mon, Sep 7, 2009 at 10:55 PM, Ismael Juma wrote:
> On Mon, 2009-09-07 at 23:22 +0200, Jesse Eichar wrote:
>> Those are the main issues I have seen raised.  I like those and think
>> they are good.  I take it on myself to read the full 203 spec tomorrow
>> and comeback with similar issues in a couple days.
>>
>> Hoping to be helpful :)
>
> That would be helpful, particularly if you co-ordinate with the people
> who are working in this area.

Agreed, ideally if this can be done somewhere relatively public and
archived where interested parties can find the discussion and
contribute.

Maybe we need a scala-libraries@ list? Toni?

Cheers,

Miles

Miles Sabin wrote:
> Agreed, ideally if this can be done somewhere relatively public and
> archived where interested parties can find the discussion and
> contribute.
>
> Maybe we need a scala-libraries@ list? Toni?
>

You guys are doing just fine on this thread, for the time being.

Before creating yet another mailing list, I'd prefer to observe these
discussions lead to something concrete first, meaning at least a draft
design that most people find satisfactory to some degree.

In the meantime, please feel free to use this thread for further
discussion. Thread length is no object, as long as the discussion is
productive.
Toni

IIRC, the last time this issue came up, the conclusion was to set-up
an external project where the community could contribute and refine
library improvements. When those libraries had matured enough, they
could migrate to the standard library. While scalax languished for
some time, and so didn't really fill up to its promise, Josh and
Erik's fork seems to be an appropriate forum for this work.

PS: My only opinion as an user is that leaving Scala without a decent
IO library until it can demand JDK-7 support would be a real shame.

Just my 2 cents.
--
Rafael de F. Ferreira.
http://www.rafaelferreira.net/

(resending. damn reply-to headers)

On Mon, Sep 7, 2009 at 7:55 PM, Antonio Cunei wrote:
> Miles Sabin wrote:
>>
>> Agreed, ideally if this can be done somewhere relatively public and
>> archived where interested parties can find the discussion and
>> contribute.
>>
>> Maybe we need a scala-libraries@ list? Toni?
>>
>
> You guys are doing just fine on this thread, for the time being.
>
> Before creating yet another mailing list, I'd prefer to observe these
> discussions lead to something concrete first, meaning at least a draft
> design that most people find satisfactory to some degree.
>
> In the meantime, please feel free to use this thread for further discussion.
> Thread length is no object, as long as the discussion is productive.
> Toni
>

Anyone who wants commit access to either Scalax.IO on github or my fork for Paul's Scala project on github, please email me off list.  Or of course you are welcome to setup your own projects.  It doesn't really matter.  I think what's important is that is that we have some real code to review, so we have something tangible to discuss.

On Mon, Sep 7, 2009 at 8:37 PM, Rafael de F. Ferreira <rafaeldff@gmail.com> wrote:

IIRC, the last time this issue came up, the conclusion was to set-up
an external project where the community could contribute and refine
library improvements. When those libraries had matured enough, they
could migrate to the standard library. While scalax languished for
some time, and so didn't really fill up to its promise, Josh and
Erik's fork seems to be an appropriate forum for this work.

PS: My only opinion as an user is that leaving Scala without a decent
IO library until it can demand JDK-7 support would be a real shame.

Just my 2 cents.
--
Rafael de F. Ferreira.
http://www.rafaelferreira.net/

Wow.... quite the discussion.

I encourage everyone to put their best efforts forward in promoting this library.  I also think code/documents are more productive then emails.  Let's also not assume other people aren't doing their jobs just because there's an API in scala trunk.  I know for a fact that nio2 was looked at and discussed.


Also, Erik,  I was thinking about trying to expand our internal iterators in scalax.io.Directory/Path.  Right now everything that you can iterate over returns a traversable (foreach => internal iterator).  

I was trying to think of whether or not we could provide this but assume a partial function.   We use the partial function's "isDefinedAt" method to limit our iterations somehow.


def walkAndFoldTree[A](a : A)(  f : PartialFunction[(Location, A), A]) : A

path("/").walkAndFoldTree(0) {
    case (dir : Directory, result) if "some/important/path".startsWith(dir.absolutePathname) =>
          result
    case (file : File, result) => result + 1
}

I'm concerned that this would be unintuitive for users.

I was trying to implement Glob-Mappers in such a way the their performance could be optimized similarly...

i.e. glob("**/scala/*.scala") would not look below directories under one named "scala".  

You could then make globs applied to files return traversables and delay iteration.

Anyway, hopefully soon I'll have time to finish prototyping the idea.

- Josh


Anyway, you have any thoughts?  (See nio2's Files.walkFileTree)




- Josh

On Mon, Sep 7, 2009 at 11:42 PM, Erik Engbrecht <erik.engbrecht@gmail.com> wrote:

Anyone who wants commit access to either Scalax.IO on github or my fork for Paul's Scala project on github, please email me off list.  Or of course you are welcome to setup your own projects.  It doesn't really matter.  I think what's important is that is that we have some real code to review, so we have something tangible to discuss.

On Mon, Sep 7, 2009 at 8:37 PM, Rafael de F. Ferreira <rafaeldff@gmail.com> wrote:

IIRC, the last time this issue came up, the conclusion was to set-up
an external project where the community could contribute and refine
library improvements. When those libraries had matured enough, they
could migrate to the standard library. While scalax languished for
some time, and so didn't really fill up to its promise, Josh and
Erik's fork seems to be an appropriate forum for this work.

PS: My only opinion as an user is that leaving Scala without a decent
IO library until it can demand JDK-7 support would be a real shame.

Just my 2 cents.
--
Rafael de F. Ferreira.
http://www.rafaelferreira.net/