feedback on scala.io.Source

On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
> I found few things about scala.io.Source that keep me stay away from
> it, and I think it can be improved upon.

I think "can be improved upon" understates the case, but yes. I have a
branch with some of this done and other useful things in progress, but
it's on my desktop machine, still packed since moving waiting for me to
figure out somewhere to work. I'll note these suggestions.

We actually have some more flexible getLines methods in the Scalax.IO project.  There is a ReaderStream abstraction that allows you to pull lines given various "LineEndingStyle"'s.   One of which could be "Any" which is java.io.BufferedReader's default or "UNIX" which is scala.io.Source's default.


The foreachBlockBytes idea is a really good one!

- Josh

On Wed, Aug 12, 2009 at 12:32 PM, Zemian Deng <thebugslayer@gmail.com> wrote:

Hello list,   I found few things about scala.io.Source that keep me stay away from it, and I think it can be improved upon.    * Source#getLines does not handle '\r' char   * Source#getLines loads all lines in memory. Problematic with large text file   * Source#getLines is slower compare to using java.io.BufferedReader#readLine   * Source doesn't deal with binary stream where I need block array quickly   Is there reason not to just use BufferedReader#readLine in a loop to provide getLines implementation? Or even better yet allow user to pass in a higher function to process the each line? These are used so frequently and I think it ought to be included in standard API. Something like these for 2.8 would be nice:     scala.io.File.foreachLine{ line => process(line) }     scala.io.File.foreachBlockBytes(512){ block => process(block) }   If you guys need implementation on these, I would be happy to contribute it.   Thanks, Zemian
--
A scala web framework http://code.google.com/p/sweetscala
My programming blog http://www.jroller.com/thebugslayer

Funny, I just recently implemented a forLineInFile method that wraps around scala.io.Source. It's an extremely useful pattern, one I would love to see in the core library.
http://gist.github.com/166650

On Wed, Aug 12, 2009 at 09:32, Zemian Deng <thebugslayer@gmail.com> wrote:

Hello list,   I found few things about scala.io.Source that keep me stay away from it, and I think it can be improved upon.    * Source#getLines does not handle '\r' char   * Source#getLines loads all lines in memory. Problematic with large text file   * Source#getLines is slower compare to using java.io.BufferedReader#readLine   * Source doesn't deal with binary stream where I need block array quickly   Is there reason not to just use BufferedReader#readLine in a loop to provide getLines implementation? Or even better yet allow user to pass in a higher function to process the each line? These are used so frequently and I think it ought to be included in standard API. Something like these for 2.8 would be nice:     scala.io.File.foreachLine{ line => process(line) }     scala.io.File.foreachBlockBytes(512){ block => process(block) }   If you guys need implementation on these, I would be happy to contribute it.   Thanks, Zemian
--
A scala web framework http://code.google.com/p/sweetscala
My programming blog http://www.jroller.com/thebugslayer

--
Alex Payne
http://twitter.com/al3x

+1 for inclusion. Much easier to read than typical for comprehension impl.
High utility for a common operation.

Alex Payne wrote:
>
> Funny, I just recently implemented a forLineInFile method that wraps
> around
> scala.io.Source. It's an extremely useful pattern, one I would love to see
> in the core library.
> http://gist.github.com/166650
>

+1. But foreachLine is a better name.

On Wed, Aug 12, 2009 at 5:20 PM, jherber <jimherber@gmail.com> wrote:

+1 for inclusion.  Much easier to read than typical for comprehension impl.
High utility for a common operation.


Alex Payne wrote:
>
> Funny, I just recently implemented a forLineInFile method that wraps
> around
> scala.io.Source. It's an extremely useful pattern, one I would love to see
> in the core library.
> http://gist.github.com/166650
>

--
View this message in context: http://www.nabble.com/feedback-on-scala.io.Source-tp24940271p24943408.html
Sent from the Scala - User mailing list archive at Nabble.com.

--
Jonhnny Weslley Sousa Silva
MSc. Candidate in Computer Science
Distributed Systems Laboratory
Federal University of Campina Grande
------------------------------------------------------------------
Desktop Data Grid File System (DDGfs)
http://www.lsd.ufcg.edu.br/ddg
------------------------------------------------------------------
www.lsd.ufcg.edu.br/~jonhnny
jonhnny-weslley.blogspot.com
jynwyle.blogspot.com

I checked in some work on Source and File.

On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
> * Source#getLines does not handle '\r' char

getLines() now takes an optional argument specifying the line separator.
If you pass none it defaults to platform default (not '\n' as before.)
Unfortunately this approach means there's no way to say '\r|\n|\r\n' so
I'll probably have to change it to use an enum like josh's.

> * Source doesn't deal with binary stream where I need block array quickly

Look at the definition of Source, emphasis on the square brackets:

abstract class Source extends Iterator[Char]

For better or worse, it is not at all designed for reading bytes. It
was like that when I found it and that assumption has strengthened.

> scala.io.File.foreachLine{ line => process(line) }

Here is what you can do with this latest checkin:

// creates a file
scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")

// iterates over the lines of a file
scala> io.File("/tmp/newfile.txt").lines foreach println
one
two

On Sat, Aug 15, 2009 at 10:30:22AM -0300, Jonhnny Weslley wrote:
> +1. But foreachLine is a better name.

When you are tempted to name a method fooBar where foo is a fundamental
method and Bar is something very specific, it's improbable you have hit
upon the optimal interface.

Paul,
Maybe it's just me, but the examples you given doesn't seem intuitive to me.
For:

// creates a file
scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")

Forcing user to provide \n on each string is too much. Why not just use the platform default to write it all to save user some grief? And the name "writeLines" sounds better fit for it. 
If you want to be complete, I recommend add two more methods like   io.File#writeText(s: String): Unit // let user specify line feed or whatever.  io.File#readText : String // read all into single string including line feed unmodified.
As For:

// iterates over the lines of a file
scala> io.File("/tmp/newfile.txt").lines foreach println

Are you not still loading all lines into memory then using foreach on them? My suggestion earlier was that you let user pass in a user function and your implementation calls as you loop through the line. no need to load all text into memory.
If you are to add new methods into scala.io.File anyway, I don't see why you cannot add foreachBlockBytes method as suggested to handle byte array.
-Zemian
On Sun, Aug 16, 2009 at 2:13 PM, Paul Phillips <paulp@improving.org> wrote:

I checked in some work on Source and File.

On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
>   * Source#getLines does not handle '\r' char

getLines() now takes an optional argument specifying the line separator.
If you pass none it defaults to platform default (not '\n' as before.)
Unfortunately this approach means there's no way to say '\r|\n|\r\n' so
I'll probably have to change it to use an enum like josh's.

>   * Source doesn't deal with binary stream where I need block array quickly

Look at the definition of Source, emphasis on the square brackets:

 abstract class Source extends Iterator[Char]

For better or worse, it is not at all designed for reading bytes.  It
was like that when I found it and that assumption has strengthened.

>   scala.io.File.foreachLine{ line => process(line) }

Here is what you can do with this latest checkin:

// creates a file
scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")

// iterates over the lines of a file
scala> io.File("/tmp/newfile.txt").lines foreach println
one
two

On Sat, Aug 15, 2009 at 10:30:22AM -0300, Jonhnny Weslley wrote:
> +1. But foreachLine is a better name.

When you are tempted to name a method fooBar where foo is a fundamental
method and Bar is something very specific, it's improbable you have hit
upon the optimal interface.

--
Paul Phillips      | Every election is a sort of advance auction sale
Future Perfect     | of stolen goods.
Empiricist         |     -- H. L. Mencken
up hill, pi pals!  |----------* http://www.improving.org/paulp/ *----------

--
A scala web framework http://code.google.com/p/sweetscala
My programming blog http://www.jroller.com/thebugslayer

p, li { white-space: pre-wrap; }It's possible to make File.lines and File.bytes lazy, making it possible to use foreach and such things on it without wasting memory. You only need to implement Iterable to get the foreach method, so there's no need to load everything into memory as you claim.

Also, I think a lot of people would prefer if you'd use existing constructs in the IO library and not invent new ones (like "foreachBlockBytes"). The collections library was rewritten to be more modular for a reason.

Just my €0.02.

On Sunday 16 August 2009 22.59.26 Zemian Deng wrote:
> Paul,
> Maybe it's just me, but the examples you given doesn't seem intuitive to
> me.
>
> For:
>
> // creates a file
> scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
>
>
> Forcing user to provide \n on each string is too much. Why not just use the
> platform default to write it all to save user some grief? And the name
> "writeLines" sounds better fit for it.
>
> If you want to be complete, I recommend add two more methods like
> io.File#writeText(s: String): Unit // let user specify line feed or
> whatever.
> io.File#readText : String // read all into single string including line
> feed unmodified.
>
> As For:
>
> // iterates over the lines of a file
> scala> io.File("/tmp/newfile.txt").lines foreach println
>
> Are you not still loading all lines into memory then using foreach on them?
> My suggestion earlier was that you let user pass in a user function and
> your implementation calls as you loop through the line. no need to load all
> text into memory.
>
> If you are to add new methods into scala.io.File anyway, I don't see why
> you cannot add foreachBlockBytes method as suggested to handle byte array.
>
> -Zemian
>
> On Sun, Aug 16, 2009 at 2:13 PM, Paul Phillips <paulp@improving.org> wrote:
> > I checked in some work on Source and File.
> >
> > On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
> > > * Source#getLines does not handle '\r' char
> >
> > getLines() now takes an optional argument specifying the line separator.
> > If you pass none it defaults to platform default (not '\n' as before.)
> > Unfortunately this approach means there's no way to say '\r|\n|\r\n' so
> > I'll probably have to change it to use an enum like josh's.
> >
> > > * Source doesn't deal with binary stream where I need block array
> >
> > quickly
> >
> > Look at the definition of Source, emphasis on the square brackets:
> >
> > abstract class Source extends Iterator[Char]
> >
> > For better or worse, it is not at all designed for reading bytes. It
> > was like that when I found it and that assumption has strengthened.
> >
> > > scala.io.File.foreachLine{ line => process(line) }
> >
> > Here is what you can do with this latest checkin:
> >
> > // creates a file
> > scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
> >
> > // iterates over the lines of a file
> > scala> io.File("/tmp/newfile.txt").lines foreach println
> > one
> > two
> >
> > On Sat, Aug 15, 2009 at 10:30:22AM -0300, Jonhnny Weslley wrote:
> > > +1. But foreachLine is a better name.
> >
> > When you are tempted to name a method fooBar where foo is a fundamental
> > method and Bar is something very specific, it's improbable you have hit
> > upon the optimal interface.
> >
> > --
> > Paul Phillips | Every election is a sort of advance auction sale
> > Future Perfect | of stolen goods.
> > Empiricist | -- H. L. Mencken
> > up hill, pi pals! |----------*
> > http://www.improving.org/paulp/*----------

On Sun, Aug 16, 2009 at 04:59:26PM -0400, Zemian Deng wrote:
> Forcing user to provide \n on each string is too much.

What, you never write strings without newlines? Sorry, you can write
your own methods all the normal ways if you don't like the selection of
convenience methods.

> Are you not still loading all lines into memory then using foreach on
> them? My suggestion earlier was that you let user pass in a user
> function and your implementation calls as you loop through the line.
> no need to load all text into memory.

I haven't looked that hard but I see no reason it would be loading all
the lines into memory. What you are describing is what "foreach" does.

> If you are to add new methods into scala.io.File anyway, I don't see
> why you cannot add foreachBlockBytes method as suggested to handle
> byte array.

Because it doesn't belong there. The trick with APIs is knowing what
not to add, but this one's not even close since (at present, perhaps now
to be re-evaluted) Source iterates over characters, not bytes.

How do you propose to make it lazy and still guarantee timely resource cleanup?

2009/8/16 David Flemström :
> It's possible to make File.lines and File.bytes lazy, making it possible to
> use foreach and such things on it without wasting memory. You only need to
> implement Iterable to get the foreach method, so there's no need to load
> everything into memory as you claim.
>
> Also, I think a lot of people would prefer if you'd use existing constructs
> in the IO library and not invent new ones (like "foreachBlockBytes"). The
> collections library was rewritten to be more modular for a reason.
>
> Just my €0.02.
>
> On Sunday 16 August 2009 22.59.26 Zemian Deng wrote:
>> Paul,
>> Maybe it's just me, but the examples you given doesn't seem intuitive to
>> me.
>>
>> For:
>>
>> // creates a file
>> scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
>>
>>
>> Forcing user to provide \n on each string is too much. Why not just use
>> the
>> platform default to write it all to save user some grief? And the name
>> "writeLines" sounds better fit for it.
>>
>> If you want to be complete, I recommend add two more methods like
>> io.File#writeText(s: String): Unit // let user specify line feed or
>> whatever.
>> io.File#readText : String // read all into single string including line
>> feed unmodified.
>>
>> As For:
>>
>> // iterates over the lines of a file
>> scala> io.File("/tmp/newfile.txt").lines foreach println
>>
>> Are you not still loading all lines into memory then using foreach on
>> them?
>> My suggestion earlier was that you let user pass in a user function and
>> your implementation calls as you loop through the line. no need to load
>> all
>> text into memory.
>>
>> If you are to add new methods into scala.io.File anyway, I don't see why
>> you cannot add foreachBlockBytes method as suggested to handle byte array.
>>
>> -Zemian
>>
>> On Sun, Aug 16, 2009 at 2:13 PM, Paul Phillips
>> wrote:
>> > I checked in some work on Source and File.
>> >
>> > On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
>> > > * Source#getLines does not handle '\r' char
>> >
>> > getLines() now takes an optional argument specifying the line separator.
>> > If you pass none it defaults to platform default (not '\n' as before.)
>> > Unfortunately this approach means there's no way to say '\r|\n|\r\n' so
>> > I'll probably have to change it to use an enum like josh's.
>> >
>> > > * Source doesn't deal with binary stream where I need block array
>> >
>> > quickly
>> >
>> > Look at the definition of Source, emphasis on the square brackets:
>> >
>> > abstract class Source extends Iterator[Char]
>> >
>> > For better or worse, it is not at all designed for reading bytes. It
>> > was like that when I found it and that assumption has strengthened.
>> >
>> > > scala.io.File.foreachLine{ line => process(line) }
>> >
>> > Here is what you can do with this latest checkin:
>> >
>> > // creates a file
>> > scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
>> >
>> > // iterates over the lines of a file
>> > scala> io.File("/tmp/newfile.txt").lines foreach println
>> > one
>> > two
>> >
>> > On Sat, Aug 15, 2009 at 10:30:22AM -0300, Jonhnny Weslley wrote:
>> > > +1. But foreachLine is a better name.
>> >
>> > When you are tempted to name a method fooBar where foo is a fundamental
>> > method and Bar is something very specific, it's improbable you have hit
>> > upon the optimal interface.
>> >
>> > --
>> > Paul Phillips | Every election is a sort of advance auction sale
>> > Future Perfect | of stolen goods.
>> > Empiricist | -- H. L. Mencken
>> > up hill, pi pals! |----------*
>> > http://www.improving.org/paulp/*----------
>
>

I didn't really mean literally 'lazy'; I rather meant that the whole data
isn't loaded at the same time, but only on demand.

Also, if you use an Iterable/Stream etc, you won't have to care about random
access, and can therefore mark the resource as consumed when the iterator has
gotten to the EOF.

On Monday 17 August 2009 00.46.55 you wrote:
> How do you propose to make it lazy and still guarantee timely resource
> cleanup?
>
> 2009/8/16 David Flemström :
> > It's possible to make File.lines and File.bytes lazy, making it possible
> > to use foreach and such things on it without wasting memory. You only
> > need to implement Iterable to get the foreach method, so there's no need
> > to load everything into memory as you claim.
> >
> > Also, I think a lot of people would prefer if you'd use existing
> > constructs in the IO library and not invent new ones (like
> > "foreachBlockBytes"). The collections library was rewritten to be more
> > modular for a reason.
> >
> > Just my €0.02.
> >
> > On Sunday 16 August 2009 22.59.26 Zemian Deng wrote:
> >> Paul,
> >> Maybe it's just me, but the examples you given doesn't seem intuitive to
> >> me.
> >>
> >> For:
> >>
> >> // creates a file
> >> scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
> >>
> >>
> >> Forcing user to provide \n on each string is too much. Why not just use
> >> the
> >> platform default to write it all to save user some grief? And the name
> >> "writeLines" sounds better fit for it.
> >>
> >> If you want to be complete, I recommend add two more methods like
> >> io.File#writeText(s: String): Unit // let user specify line feed or
> >> whatever.
> >> io.File#readText : String // read all into single string including line
> >> feed unmodified.
> >>
> >> As For:
> >>
> >> // iterates over the lines of a file
> >> scala> io.File("/tmp/newfile.txt").lines foreach println
> >>
> >> Are you not still loading all lines into memory then using foreach on
> >> them?
> >> My suggestion earlier was that you let user pass in a user function and
> >> your implementation calls as you loop through the line. no need to load
> >> all
> >> text into memory.
> >>
> >> If you are to add new methods into scala.io.File anyway, I don't see why
> >> you cannot add foreachBlockBytes method as suggested to handle byte
> >> array.
> >>
> >> -Zemian
> >>
> >> On Sun, Aug 16, 2009 at 2:13 PM, Paul Phillips
> >>
> >> wrote:
> >> > I checked in some work on Source and File.
> >> >
> >> > On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
> >> > > * Source#getLines does not handle '\r' char
> >> >
> >> > getLines() now takes an optional argument specifying the line
> >> > separator. If you pass none it defaults to platform default (not '\n'
> >> > as before.) Unfortunately this approach means there's no way to say
> >> > '\r|\n|\r\n' so I'll probably have to change it to use an enum like
> >> > josh's.
> >> >
> >> > > * Source doesn't deal with binary stream where I need block array
> >> >
> >> > quickly
> >> >
> >> > Look at the definition of Source, emphasis on the square brackets:
> >> >
> >> > abstract class Source extends Iterator[Char]
> >> >
> >> > For better or worse, it is not at all designed for reading bytes. It
> >> > was like that when I found it and that assumption has strengthened.
> >> >
> >> > > scala.io.File.foreachLine{ line => process(line) }
> >> >
> >> > Here is what you can do with this latest checkin:
> >> >
> >> > // creates a file
> >> > scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
> >> >
> >> > // iterates over the lines of a file
> >> > scala> io.File("/tmp/newfile.txt").lines foreach println
> >> > one
> >> > two
> >> >
> >> > On Sat, Aug 15, 2009 at 10:30:22AM -0300, Jonhnny Weslley wrote:
> >> > > +1. But foreachLine is a better name.
> >> >
> >> > When you are tempted to name a method fooBar where foo is a
> >> > fundamental method and Bar is something very specific, it's improbable
> >> > you have hit upon the optimal interface.
> >> >
> >> > --
> >> > Paul Phillips | Every election is a sort of advance auction sale
> >> > Future Perfect | of stolen goods.
> >> > Empiricist | -- H. L. Mencken
> >> > up hill, pi pals! |----------*
> >> > http://www.improving.org/paulp/*----------

Actually, I think Traversable fits a lot better when you only want foreach.  If you look at Scalax.IO, that's exactly what we did with File.lines and File.bytes methods.   We also include default parameters you can do things like for { line <- file.lines(LineEndingStyle.MAC) } yield ...



On Sun, Aug 16, 2009 at 5:15 PM, David Flemström <david.flemstrom@gmail.com> wrote:

It's possible to make File.lines and File.bytes lazy, making it possible to use foreach and such things on it without wasting memory. You only need to implement Iterable to get the foreach method, so there's no need to load everything into memory as you claim.

Also, I think a lot of people would prefer if you'd use existing constructs in the IO library and not invent new ones (like "foreachBlockBytes"). The collections library was rewritten to be more modular for a reason.

Just my €0.02.

On Sunday 16 August 2009 22.59.26 Zemian Deng wrote:
> Paul,
> Maybe it's just me, but the examples you given doesn't seem intuitive to
> me.
>
> For:
>
> // creates a file
> scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
>
>
> Forcing user to provide \n on each string is too much. Why not just use the
> platform default to write it all to save user some grief? And the name
> "writeLines" sounds better fit for it.
>
> If you want to be complete, I recommend add two more methods like
> io.File#writeText(s: String): Unit // let user specify line feed or
> whatever.
> io.File#readText : String // read all into single string including line
> feed unmodified.
>
> As For:
>
> // iterates over the lines of a file
> scala> io.File("/tmp/newfile.txt").lines foreach println
>
> Are you not still loading all lines into memory then using foreach on them?
> My suggestion earlier was that you let user pass in a user function and
> your implementation calls as you loop through the line. no need to load all
> text into memory.
>
> If you are to add new methods into scala.io.File anyway, I don't see why
> you cannot add foreachBlockBytes method as suggested to handle byte array.
>
> -Zemian
>
> On Sun, Aug 16, 2009 at 2:13 PM, Paul Phillips <paulp@improving.org> wrote:
> > I checked in some work on Source and File.
> >
> > On Wed, Aug 12, 2009 at 12:32:18PM -0400, Zemian Deng wrote:
> > > * Source#getLines does not handle '\r' char
> >
> > getLines() now takes an optional argument specifying the line separator.
> > If you pass none it defaults to platform default (not '\n' as before.)
> > Unfortunately this approach means there's no way to say '\r|\n|\r\n' so
> > I'll probably have to change it to use an enum like josh's.
> >
> > > * Source doesn't deal with binary stream where I need block array
> >
> > quickly
> >
> > Look at the definition of Source, emphasis on the square brackets:
> >
> > abstract class Source extends Iterator[Char]
> >
> > For better or worse, it is not at all designed for reading bytes. It
> > was like that when I found it and that assumption has strengthened.
> >
> > > scala.io.File.foreachLine{ line => process(line) }
> >
> > Here is what you can do with this latest checkin:
> >
> > // creates a file
> > scala> io.File("/tmp/newfile.txt") writeAll List("one\n", "two\n")
> >
> > // iterates over the lines of a file
> > scala> io.File("/tmp/newfile.txt").lines foreach println
> > one
> > two
> >
> > On Sat, Aug 15, 2009 at 10:30:22AM -0300, Jonhnny Weslley wrote:
> > > +1. But foreachLine is a better name.
> >
> > When you are tempted to name a method fooBar where foo is a fundamental
> > method and Bar is something very specific, it's improbable you have hit
> > upon the optimal interface.
> >
> > --
> > Paul Phillips | Every election is a sort of advance auction sale
> > Future Perfect | of stolen goods.
> > Empiricist | -- H. L. Mencken
> > up hill, pi pals! |----------*
> > http://www.improving.org/paulp/*----------

On Sun, Aug 16, 2009 at 5:30 PM, Paul Phillips <paulp@improving.org> wrote:

On Sun, Aug 16, 2009 at 04:59:26PM -0400, Zemian Deng wrote:
> Forcing user to provide \n on each string is too much.

What, you never write strings without newlines?

I do. That's why I suggested writeText method. I prefer write them this way:  file.writeText("one\ntwo\nthree")  than your way:  file.writeText.writeAll(List("one\n", "two\n", "three\n"))  

Sorry, you can write
your own methods all the normal ways if you don't like the selection of
convenience methods.

Well, this is a user-list, and I am a user, so I am here to give you feedback that your "convenience" method is not that convenience for me. I know I can always write my own, but then I would see scala.io.File just as scala.io.Source, which I don't use much. 

> Are you not still loading all lines into memory then using foreach on
> them? My suggestion earlier was that you let user pass in a user
> function and your implementation calls as you loop through the line.
> no need to load all text into memory.

I haven't looked that hard but I see no reason it would be loading all
the lines into memory.  What you are describing is what "foreach" does.

Please do. If there is only one method added to your new scala.io.File, I would vote for the foreachLine mehtod. I see people use this all the time! And I miss Scala didn't provide one in api.  

> If you are to add new methods into scala.io.File anyway, I don't see
> why you cannot add foreachBlockBytes method as suggested to handle
> byte array.

Because it doesn't belong there.  The trick with APIs is knowing what
not to add, but this one's not even close since (at present, perhaps now
to be re-evaluted) Source iterates over characters, not bytes.

How is a method dealing with file stream not belong to File. If I need other classes to do this like Java does, then again I don't see a point of using scala.io.File. I can understand you guys might like the method name, but I thought the method usage is valid. Other lang like Groovy provided method like "withInputStream", and even that you still do the same thing over and over again, which is read a block of bytes and then process it.
I think Groovy did a pretty good job on providing "convenient" methods on java.io.File, and Scala will do a even more supper job since it's API will tell me the type of the higher function signature for usage! http://groovy.codehaus.org/groovy-jdk/java/io/File.html
Hope these help.
- Zemian 

--
Paul Phillips      | Every normal man must be tempted at times
Caged Spirit       | to spit on his hands, hoist the black flag,
Empiricist         | and begin to slit throats.
i pull his palp!   |     -- H. L. Mencken

--
A scala web framework http://code.google.com/p/sweetscala
My programming blog http://www.jroller.com/thebugslayer

I noticed the addition of the optional argument since it broke my existing

for (l <- src.getLines) { ... }

code. The compiler now trips over this (complains about missing parameters
and suggests I use getLines_ to refer to the method itself) and effectively
insists I change this into

for (l <- src.getLines()) { ... }

If this is something that is a consequence of default parameter values I
will probably tend to revert to separate methods (possibly using
overloading) instead of using a default value for a parameter.

Is this a temporal imperfection of the current implementation?

Gr. Silvio

On Mon, Aug 17, 2009 at 04:30:40PM -0700, Silvio Bierman wrote:
> If this is something that is a consequence of default parameter values

It is.

> Is this a temporal imperfection of the current implementation?

This was a conscious compromise among several competing concerns. It
was determined that you can't leave off the parameter list entirely if
there are defaults. You can probably find all the discussion in the
scala-internals archive.

Paul Phillips wrote:
>
> On Mon, Aug 17, 2009 at 04:30:40PM -0700, Silvio Bierman wrote:
>> If this is something that is a consequence of default parameter values
>
> It is.
>
>> Is this a temporal imperfection of the current implementation?
>
> This was a conscious compromise among several competing concerns. It
> was determined that you can't leave off the parameter list entirely if
> there are defaults. You can probably find all the discussion in the
> scala-internals archive.
>
>

Ok, I understand. In that case I hope that few parameterless methods from
the current standard lib will be retrofitted with such defaults. In the case
of Source I would have preferred an additional getSeparatedLines(sep :
String) and a getLines that calls it with the platform default.
Default params are great for the use case of a long paramater list where
each or most of them have a sensible default but where it is likely that
callers will want to divert from at least some of the defaults. Because of
this () consequence adding parameters with defaults for relatively rare use
cases to parameterless methods is a step backwards, IMHO.

Best regards,

Silvio

On Tue, Aug 18, 2009 at 5:46 AM, Silvio Bierman <sbierman@jambo-software.com> wrote:

Paul Phillips wrote:
>
> On Mon, Aug 17, 2009 at 04:30:40PM -0700, Silvio Bierman wrote:
>> If this is something that is a consequence of default parameter values
>
> It is.
>
>> Is this a temporal imperfection of the current implementation?
>
> This was a conscious compromise among several competing concerns.  It
> was determined that you can't leave off the parameter list entirely if
> there are defaults.  You can probably find all the discussion in the
> scala-internals archive.
>
>

Ok, I understand. In that case I hope that few parameterless methods from
the current standard lib will be retrofitted with such defaults. In the case
of Source I would have preferred an additional getSeparatedLines(sep :
String) and a getLines that calls it with the platform default.
Default params are great for the use case of a long paramater list where
each or most of them have a sensible default but where it is likely that
callers will want to divert from at least some of the defaults. Because of
this () consequence adding parameters with defaults for relatively rare use
cases to parameterless methods is a step backwards, IMHO.

Having getLines use the platform default would have broken lots of code that uses scala.io.    Having the compiler warn you to add () is a nicety.  Not having indication that your once-working code is now broken is not a nicety.

I honestly don't see what's wrong with adding () to methods.
 

Best regards,

Silvio

--
View this message in context: http://www.nabble.com/feedback-on-scala.io.Source-tp24940271p25021605.html
Sent from the Scala - User mailing list archive at Nabble.com.

- Josh

Josh Suereth wrote:
>
> On Tue, Aug 18, 2009 at 5:46 AM, Silvio Bierman
> > wrote:
>
>>
> Having getLines use the platform default would have broken lots of code
> that
> uses scala.io. Having the compiler warn you to add () is a nicety. Not
> having indication that your once-working code is now broken is not a
> nicety.
>
>

I meant that the getLines method should have done what it has done uptil
now. That is indeed not using the platfrom default as separator. I just
meant that the more generic new method could be called by the parameterless
getLines if that woud make sense.

Josh Suereth wrote:
>
> I honestly don't see what's wrong with adding () to methods.
>
> - Josh
>
>

What I think is wrong with it is that it clutters up code. I like being
allowed to leave out the () for methods without parameters and would
therefore like to use that in all cases where I call a method without
providing parameters. Having to use () for methods that actually have
parameters for which defaults have been specified makes the code harder to
read especially because of the distinction.

The argument that the () are nice because they make the distinction between
a call to a method without parameters and one with parameters which all have
defaults explicit could just as easily be used to justify insisting on ()
for all methods to make the distinction with a plain field access explicit
(yes, I know that under the hood these are actually methods).

I did not follow the discussion around this topic but I can understand that
ambiguities might be introduced when allowing the absence of () in this
case. That is not a reason to not have default parameter values since they
are very usefull with long parameter lists. I do however find the forced use
of () enough of a drawback to keep it out of the standard libs as much as
possible. I have no problem with using the construct in user code.

I agree it is just a matter of taste and am just expressing mine.

Regards,

Silvio

> Default params are great for the use case of a long paramater
> list where each or most of them have a sensible default but
> where it is likely that callers will want to divert from at
> least some of the defaults. Because of this () consequence
> adding parameters with defaults for relatively rare use cases
> to parameterless methods is a step backwards, IMHO.

+1

KR
Det

I agree with this sentiment. But I want to make clear the problem is taking *parameterless* methods and adding default parameters to them.   Since Source is, frankly, screwed, I make an exception for breakage there, as long as the resulting API is decent enough that we won't have to do it again.

On Wed, Aug 19, 2009 at 6:36 AM, Detering Dirk <Dirk.Detering@bitmarck.de> wrote:

> Default params are great for the use case of a long paramater
> list where each or most of them have a sensible default but
> where it is likely that callers will want to divert from at
> least some of the defaults. Because of this () consequence
> adding parameters with defaults for relatively rare use cases
> to parameterless methods is a step backwards, IMHO.

+1

KR
Det

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[moving to scala-debate, was Re: [scala-user] feedback on
scala.io.Source]
Hi all,

This email is inspired by the Source.getLines situation.

I feel a little awkward with the Source API. I admit I have rarely
used Source and also I have not followed the implementation details.
So, think of my comments as being of a general design-oriented nature.

I believe we MAY need to decide the target audience of the Source API
and I see two distinctions here. The reason I say this is because
Source appears to be a low-level API, with all of its support for
defining line endings etc AT THE CALL SITE, but it seems its main uses
are more higher-level ones.

First, we have the regular, application user, who just wants to
iterate over the lines of a text file in the most straightforward way.
Such an application user does not even have to specify the line
ending, simply because he is lazy and does not care. This is common
grounds. Note that "does not have to specify" should mean "he doesn't
get a chance to specify" and the API must enforce the latter.
==> I feel this higher-level API is what Source should be.

Second, we have some library user, strictly lower-level than the
application user, who truly wants some better control of what is going
underneath. This kind of user may need to use a few more parameters to
define the exact behavior of Source's implementation. Thus, we give
him the ability to control line endings etc at the call site.
==> I feel that currently the Source API fails to support such a lower-
level goal, although it pretends to. For instance, a lower-level user
would want the line endings to appear at the returned lines.

But now look what is happening. I have a getLines method (actually
"getLines()" with the parentheses for the casual, application-level
user) which is broken. The line-ending defaults to the platform one,
instead of defaulting to WHATEVER LINE ENDING THE FILE OF INTEREST
has. So, If I am on Unix and read a file generated with the \r\n
Windows endings, then I get the \r in my lines. This violates the
principle of least surprise.

I think the heart of the problem is we have not sat down and WRITE the
requirements of the Source API from a user perspective. We are in such
a hurry in delving into implementation and abstracting away
implementation things, instead of abstracting away at a higher level.
Of course, you should understand by now (via my general involvement in
the Scala community) that I have deepest respect for all an each one
contributor, so the above statement is not a (poisonous) arrow. Or, to
put it otherwise, if your brain insists on considering it as such,
please count me in the target group.

I have the impression that another possible cause may be the fact that
Source is burdened with too many concerns (is it true?). It could be
better to separate those concerns into different implementation units.
But then again, we need to write down those concerns first.

I am making all these comments because I do not want to see another
lame (as the community has decided, AFAIK) Source creep into Scala
3.0, which is a major release. I generally feel we do not have to
stuff as many new features as possible into 3.0.

Of course, I may be off... But then again, this is debate and I beg
for your...

...Thoughts?

Christos

P.S.1 As a side-note, the default line-endings approach could be more
meaningful when we WRITE stuff, than when we READ.
P.S.2 The funny subject is inspired by Julian Schwinger's book title http://bit.ly/uUJSH

On Aug 18, 2009, at 2:36 AM, Paul Phillips wrote:

> On Mon, Aug 17, 2009 at 04:30:40PM -0700, Silvio Bierman wrote:
>> If this is something that is a consequence of default parameter
>> values
>
> It is.
>
>> Is this a temporal imperfection of the current implementation?
>
> This was a conscious compromise among several competing concerns. It
> was determined that you can't leave off the parameter list entirely if
> there are defaults. You can probably find all the discussion in the
> scala-internals archive.
>

FYI I totally agree that it should just deal with whatever line ending
it finds, and that's the way I originally wrote it, and that's what will
be the default if it's up to me. However most of the annoying work I'm
doing here revolves around the decision to leave the platform default
encoding in place when it's left unspecified. Line ending defaults
aren't the same as charset defaults, but it is all tied together if one
seeks anything resembling consistency.

Even at system level, I'd want to just have a bunch of lines that I can iterate over - without caring for the exact nature of the delimiters.  Including the line terminators in this scenario would be equivalent to putting "," into every other element of a string array.
I'm struggling to imagine a scenario where I could open a file that contained mixed line endings, or where I'd need to handle \r, \n, \r\n, or \n\r differently within a single file, not unless you truly intend for \r to be a carriage return without a line feed so that you can overwrite text to give underlined or bold characters.  I believe this technique died out with punch cards, although I fully appreciate the need to process and import old data sets...
Agreed, once a file is opened then any subsequent appended lines should match whatever encoding is already in place.  This is an important requirement and should be clearly specified.  It should also be transparent to the user of the library.

Anything deeper than this I would argue that you aren't actually working with lines at all, and should really be looking at a character-level API.


On Wed, Aug 19, 2009 at 4:57 PM, Christos KK Loverdos <loverdos@gmail.com> wrote:

[moving to scala-debate, was Re: [scala-user] feedback on scala.io.Source]
Hi all,

This email is inspired by the Source.getLines situation.

I feel a little awkward with the Source API. I admit I have rarely used Source and also I have not followed the implementation details. So, think of my comments as being of a general design-oriented nature.

I believe we MAY need to decide the target audience of the Source API and I see two distinctions here. The reason I say this is because Source appears to be a low-level API, with all of its support for defining line endings etc AT THE CALL SITE, but it seems its main uses are more higher-level ones.

First, we have the regular, application user, who just wants to iterate over the lines of a text file in the most straightforward way. Such an application user does not even have to specify the line ending, simply because he is lazy and does not care. This is common grounds. Note that "does not have to specify" should mean "he doesn't get a chance to specify" and the API must enforce the latter.
==> I feel this higher-level API is what Source should be.

Second, we have some library user, strictly lower-level than the application user, who truly wants some better control of what is going underneath. This kind of user may need to use a few more parameters to define the exact behavior of Source's implementation. Thus, we give him the ability to control line endings etc at the call site.
==> I feel that currently the Source API fails to support such a lower-level goal, although it pretends to. For instance, a lower-level user would want the line endings to appear at the returned lines.

But now look what is happening. I have a getLines method (actually "getLines()" with the parentheses for the casual, application-level user) which is broken. The line-ending defaults to the platform one, instead of defaulting to WHATEVER LINE ENDING THE FILE OF INTEREST has. So, If I am on Unix and read a file generated with the \r\n Windows endings, then I get the \r in my lines. This violates the principle of least surprise.

I think the heart of the problem is we have not sat down and WRITE the requirements of the Source API from a user perspective. We are in such a hurry in delving into implementation and abstracting away implementation things, instead of abstracting away at a higher level. Of course, you should understand by now (via my general involvement in the Scala community) that I have deepest respect for all an each one contributor, so the above statement is not a (poisonous) arrow. Or, to put it otherwise, if your brain insists on considering it as such, please count me in the target group.

I have the impression that another possible cause may be the fact that Source is burdened with too many concerns (is it true?). It could be better to separate those concerns into different implementation units. But then again, we need to write down those concerns first.

I am making all these comments because I do not want to see another lame (as the community has decided, AFAIK) Source creep into Scala 3.0, which is a major release. I generally feel we do not have to stuff as many new features as possible into 3.0.

Of course, I may be off... But then again, this is debate and I beg for your...

...Thoughts?


Christos

P.S.1 As a side-note, the default line-endings approach could be more meaningful when we WRITE stuff, than when we READ.
P.S.2 The funny subject is inspired by Julian Schwinger's book title http://bit.ly/uUJSH

On Aug 18, 2009, at 2:36 AM, Paul Phillips wrote:

On Mon, Aug 17, 2009 at 04:30:40PM -0700, Silvio Bierman wrote:

If this is something that is a consequence of default parameter values

It is.

Is this a temporal imperfection of the current implementation?

This was a conscious compromise among several competing concerns.  It
was determined that you can't leave off the parameter list entirely if
there are defaults.  You can probably find all the discussion in the
scala-internals archive.

> I'm struggling to imagine a scenario where I could open a file that
> contained mixed line endings

Most of our code and related activity uses whatever Linux uses for
line endings. One pom.xml contains Windows line endings. I edited
this with vim in Linux, adding in Linux-style line endings. This file
now has mixed line endings. I am quite glad that this does not
perturb anything I use to open files.

On Wed, Aug 19, 2009 at 6:36 PM, Ricky Clarkson <ricky.clarkson@gmail.com> wrote:

> I'm struggling to imagine a scenario where I could open a file that
> contained mixed line endings

Most of our code and related activity uses whatever Linux uses for
line endings.  One pom.xml contains Windows line endings.  I edited
this with vim in Linux, adding in Linux-style line endings.  This file
now has mixed line endings.  I am quite glad that this does not
perturb anything I use to open files.

Wow, now that's fast feedback with an example.  So close to home as well... 

--
Ricky Clarkson
Java Programmer, AD Holdings
+44 1565 770804
Skype: ricky_clarkson
Google Talk: ricky.clarkson@gmail.com

On Wednesday August 19 2009, Ricky Clarkson wrote:
> > I'm struggling to imagine a scenario where I could open a file that
> > contained mixed line endings
>
> Most of our code and related activity uses whatever Linux uses for
> line endings. One pom.xml contains Windows line endings. I edited
> this with vim in Linux, adding in Linux-style line endings. This
> file now has mixed line endings. I am quite glad that this does not
> perturb anything I use to open files.

I guess Vim has lots of modes, 'cause in my experience (on Linux) it
adapts to the line-endings of the file being edited and preserves it
when you save it unless you explicitly do something like:

:set ff=unix

or

:set ff=dos

And if issuing one of these commands actually changes the files's bytes,
you'll have to save it (or explicitly discard the changes) to quit,
just as if you'd used a Vim editing command.

Randall Schulz

On Wed, Aug 19, 2009 at 6:36 PM, Ricky Clarkson <ricky.clarkson@gmail.com> wrote:

> I'm struggling to imagine a scenario where I could open a file that
> contained mixed line endings

Most of our code and related activity uses whatever Linux uses for
line endings.  One pom.xml contains Windows line endings.  I edited
this with vim in Linux, adding in Linux-style line endings.  This file
now has mixed line endings.  I am quite glad that this does not
perturb anything I use to open files.

 I just thought... Doesn't SVN reformat line endings as appropriate for the local environment?