- About Scala
- Documentation
- Code Examples
- Software
- Scala Developers
RE: file output
Wed, 2009-03-18, 12:51
Slightly off topic:
When I came to Java from other languages, I was
irritated by the unintuitive way Java treats the
file system.
It started with all this different OutputStream
things, went over to the awkward file rename
semantics (*) and ended with the detection that
there is no OO difference between the concepts
'file' and 'directory'(**)
For Scala to downscale well into scripting scenarios,
it would be good to have more support on the most
common cases.
While Java now does not encapsulate the open and
close, languages with first-class function support
should provide that immediately, like what happened
with the collections.
For example:
File.write { out =>
out.write("Hello")
out.write("World")
}
and the opening and closing is done in the
write method without the user's attention.
----
Footnotes use pseudocode for my convenience
(*)
x = new File("hello")
x.renameTo("world") <-- doesn't exist ...
...you have to:
y = new File("world")
x.renameTo(y)
now the entry on disk(!) is renamed to world.
x as file instance has not changed in any way
other that it now has lost its status as
"refers to an existing disk entry".
(**)
In Java, you have to do:
if (myFile.isDirectory() ) {// !
File y = myFile.listFiles() // <-- returns all entries, not only
files !
for( z : y) {
if (z.isDirectory()) // !
//do something on dir
// or alternatively
if (z.isFile()) // !
//do something on z,
// you would not do on dirs, e.g. open and write into
}
}
Instead of
Dir[] y = myDir.listDirs()
for (z : y) {
//do something on z you would only do on dirs
}
// or alternatively
File[] y = myDir.listFiles()
for(z : y) {
// do something on z you would never do on dirs
}
// or perhaps
DirEntry[] y = myDir.list()
for (z : y) {
z match {
case Dir(n) =>
case File(n) =>
}
}
KR
Det
Wed, 2009-03-18, 19:07
#2
Re: file output
Java 7 is supposed to have a new filesystem library.
On Wed, Mar 18, 2009 at 8:21 AM, Tako Schotanus <scala@codejive.org> wrote:
On Wed, Mar 18, 2009 at 8:21 AM, Tako Schotanus <scala@codejive.org> wrote:
Although I loved the flexibility that came with the entire java.io.*
library when _I_ was introduced to Java I must agree that a standard
IO library that makes it really easy to use the basic stuff that we
use most often would be great. Some of it already exists but having
even more of it so Scala is immediately useful for scripting-like
purposes would be really useful.
On Wed, Mar 18, 2009 at 12:51, Detering Dirk <Dirk.Detering@bitmarck.de> wrote:
> Slightly off topic:
>
> When I came to Java from other languages, I was
> irritated by the unintuitive way Java treats the
> file system.
>
> It started with all this different OutputStream
> things, went over to the awkward file rename
> semantics (*) and ended with the detection that
> there is no OO difference between the concepts
> 'file' and 'directory'(**)
>
> For Scala to downscale well into scripting scenarios,
> it would be good to have more support on the most
> common cases.
>
> While Java now does not encapsulate the open and
> close, languages with first-class function support
> should provide that immediately, like what happened
> with the collections.
>
> For example:
>
> File.write { out =>
> out.write("Hello")
> out.write("World")
> }
>
> and the opening and closing is done in the
> write method without the user's attention.
>
>
> ----
> Footnotes use pseudocode for my convenience
>
> (*)
> x = new File("hello")
> x.renameTo("world") <-- doesn't exist ...
> ...you have to:
> y = new File("world")
> x.renameTo(y)
>
> now the entry on disk(!) is renamed to world.
> x as file instance has not changed in any way
> other that it now has lost its status as
> "refers to an existing disk entry".
>
>
> (**)
> In Java, you have to do:
>
> if (myFile.isDirectory() ) {// !
> File y = myFile.listFiles() // <-- returns all entries, not only
> files !
> for( z : y) {
> if (z.isDirectory()) // !
> //do something on dir
> // or alternatively
> if (z.isFile()) // !
> //do something on z,
> // you would not do on dirs, e.g. open and write into
> }
> }
>
> Instead of
>
> Dir[] y = myDir.listDirs()
>
> for (z : y) {
> //do something on z you would only do on dirs
> }
>
> // or alternatively
>
> File[] y = myDir.listFiles()
>
> for(z : y) {
> // do something on z you would never do on dirs
> }
>
> // or perhaps
>
> DirEntry[] y = myDir.list()
>
> for (z : y) {
> z match {
> case Dir(n) =>
> case File(n) =>
> }
> }
>
> KR
> Det
>
>
--
-Tako
Wed, 2009-03-18, 20:37
#3
Re: file output
I was surprised to learn recently that output files are apparently not automatically flushed and closed when a Scala script or program exits normally. Requiring the programmer to explicitly close files is not only an inconvenience, it is a potential source of subtle errors that could go undetected.
Russ P.
On Wed, Mar 18, 2009 at 5:21 AM, Tako Schotanus <scala@codejive.org> wrote:
--
http://RussP.us
Russ P.
On Wed, Mar 18, 2009 at 5:21 AM, Tako Schotanus <scala@codejive.org> wrote:
Although I loved the flexibility that came with the entire java.io.*
library when _I_ was introduced to Java I must agree that a standard
IO library that makes it really easy to use the basic stuff that we
use most often would be great. Some of it already exists but having
even more of it so Scala is immediately useful for scripting-like
purposes would be really useful.
On Wed, Mar 18, 2009 at 12:51, Detering Dirk <Dirk.Detering@bitmarck.de> wrote:
> Slightly off topic:
>
> When I came to Java from other languages, I was
> irritated by the unintuitive way Java treats the
> file system.
>
> It started with all this different OutputStream
> things, went over to the awkward file rename
> semantics (*) and ended with the detection that
> there is no OO difference between the concepts
> 'file' and 'directory'(**)
>
> For Scala to downscale well into scripting scenarios,
> it would be good to have more support on the most
> common cases.
>
> While Java now does not encapsulate the open and
> close, languages with first-class function support
> should provide that immediately, like what happened
> with the collections.
>
> For example:
>
> File.write { out =>
> out.write("Hello")
> out.write("World")
> }
>
> and the opening and closing is done in the
> write method without the user's attention.
>
>
> ----
> Footnotes use pseudocode for my convenience
>
> (*)
> x = new File("hello")
> x.renameTo("world") <-- doesn't exist ...
> ...you have to:
> y = new File("world")
> x.renameTo(y)
>
> now the entry on disk(!) is renamed to world.
> x as file instance has not changed in any way
> other that it now has lost its status as
> "refers to an existing disk entry".
>
>
> (**)
> In Java, you have to do:
>
> if (myFile.isDirectory() ) {// !
> File y = myFile.listFiles() // <-- returns all entries, not only
> files !
> for( z : y) {
> if (z.isDirectory()) // !
> //do something on dir
> // or alternatively
> if (z.isFile()) // !
> //do something on z,
> // you would not do on dirs, e.g. open and write into
> }
> }
>
> Instead of
>
> Dir[] y = myDir.listDirs()
>
> for (z : y) {
> //do something on z you would only do on dirs
> }
>
> // or alternatively
>
> File[] y = myDir.listFiles()
>
> for(z : y) {
> // do something on z you would never do on dirs
> }
>
> // or perhaps
>
> DirEntry[] y = myDir.list()
>
> for (z : y) {
> z match {
> case Dir(n) =>
> case File(n) =>
> }
> }
>
> KR
> Det
>
>
--
-Tako
--
http://RussP.us
Wed, 2009-03-18, 21:37
#4
Re: file output
On Wednesday March 18 2009, Russ Paielli wrote:
> I was surprised to learn recently that output files are apparently
> not automatically flushed and closed when a Scala script or program
> exits normally. Requiring the programmer to explicitly close files is
> not only an inconvenience, it is a potential source of subtle errors
> that could go undetected.
Flushing is subsumed by closing the stream or writer, so the usual
patterns or library functions used to ensure that a stream or writer
(or reader) gets closed will necessarily ensure that any pending,
buffered output is flushed.
> Russ P.
Randall Schulz
Thu, 2009-03-19, 06:17
#5
Re: file output
On Wed, Mar 18, 2009 at 1:25 PM, Randall R Schulz <rschulz@sonic.net> wrote:
I realize that explicitly closing a file also flushes it. But I recently found that a short Scala script I wrote produced no output until I explictly closed the output file. Unless I'm missing something, that means that Scala apparently does not automatically close (and flush) files when a script terminates (and I assume the same is true of a main program). Python automatically closes files at program termination, and I suggest that Scala should too.
Russ P.
--
http://RussP.us
On Wednesday March 18 2009, Russ Paielli wrote:
> I was surprised to learn recently that output files are apparently
> not automatically flushed and closed when a Scala script or program
> exits normally. Requiring the programmer to explicitly close files is
> not only an inconvenience, it is a potential source of subtle errors
> that could go undetected.
Flushing is subsumed by closing the stream or writer, so the usual
patterns or library functions used to ensure that a stream or writer
(or reader) gets closed will necessarily ensure that any pending,
buffered output is flushed.
I realize that explicitly closing a file also flushes it. But I recently found that a short Scala script I wrote produced no output until I explictly closed the output file. Unless I'm missing something, that means that Scala apparently does not automatically close (and flush) files when a script terminates (and I assume the same is true of a main program). Python automatically closes files at program termination, and I suggest that Scala should too.
Russ P.
--
http://RussP.us
Thu, 2009-03-19, 09:07
#6
Re: file output
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
Alternatively, write a class that encapsulates the functionality in those classes and also write a JVM shutdown hook that somehow keeps references to all the files left open and explicitly close them on shutdown.
Or, a bit more elegant: you could use an Automate Resource Management Block -
http://polyglot-window.blogspot.com/2008/02/arm-blocks-in-scala.html
However, if you just want to use the plain old Java classes then simply close your files explicitly like everyone else writing Java programs. In fact, to be rigorous you should close them in the finally block following a try block.
Ishaaq
2009/3/19 Russ Paielli <russ.paielli@gmail.com>
Alternatively, write a class that encapsulates the functionality in those classes and also write a JVM shutdown hook that somehow keeps references to all the files left open and explicitly close them on shutdown.
Or, a bit more elegant: you could use an Automate Resource Management Block -
http://polyglot-window.blogspot.com/2008/02/arm-blocks-in-scala.html
However, if you just want to use the plain old Java classes then simply close your files explicitly like everyone else writing Java programs. In fact, to be rigorous you should close them in the finally block following a try block.
Ishaaq
2009/3/19 Russ Paielli <russ.paielli@gmail.com>
On Wed, Mar 18, 2009 at 1:25 PM, Randall R Schulz <rschulz@sonic.net> wrote:
On Wednesday March 18 2009, Russ Paielli wrote:
> I was surprised to learn recently that output files are apparently
> not automatically flushed and closed when a Scala script or program
> exits normally. Requiring the programmer to explicitly close files is
> not only an inconvenience, it is a potential source of subtle errors
> that could go undetected.
Flushing is subsumed by closing the stream or writer, so the usual
patterns or library functions used to ensure that a stream or writer
(or reader) gets closed will necessarily ensure that any pending,
buffered output is flushed.
I realize that explicitly closing a file also flushes it. But I recently found that a short Scala script I wrote produced no output until I explictly closed the output file. Unless I'm missing something, that means that Scala apparently does not automatically close (and flush) files when a script terminates (and I assume the same is true of a main program). Python automatically closes files at program termination, and I suggest that Scala should too.
Russ P.
--
http://RussP.us
Thu, 2009-03-19, 18:57
#7
Re: file output
On Thu, Mar 19, 2009 at 12:57 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help. That's all I'm saying.
Russ P.
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help. That's all I'm saying.
Russ P.
Thu, 2009-03-19, 19:47
#8
Re: file output
Russ Paielli wrote:
There's nothing though to stop anyone from implementing a standard extensions library like what Groovy has. I'd imagine you could then import scala.pimpmystandards._ or some such and let anyone who wants that sort scripty experience to have it. Personally I used Groovy a lot and liked it for a while since I got more done for the same effort than with Java. After a number of issues came up I really wanted native Java speed and assurances of static typing. I'll readily admit I'm still on the scala learning curve but I don't really missed Groovy and its sugar at all.
d03afaa20903191046k43c8868od666c9aa47dd604b [at] mail [dot] gmail [dot] com" type="cite">On Thu, Mar 19, 2009 at 12:57 AM, Ishaaq Chandy <ishaaq [at] gmail [dot] com" rel="nofollow">ishaaq@gmail.com> wrote:I don't really think Scala wants to be considered a good scripting language nor do I think Scala has a standard for file output. If you are using a java.io.FileWriter then you get its behavior. Nothing extra added -- I believe that's a good thing.
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help. That's all I'm saying.
Russ P.
There's nothing though to stop anyone from implementing a standard extensions library like what Groovy has. I'd imagine you could then import scala.pimpmystandards._ or some such and let anyone who wants that sort scripty experience to have it. Personally I used Groovy a lot and liked it for a while since I got more done for the same effort than with Java. After a number of issues came up I really wanted native Java speed and assurances of static typing. I'll readily admit I'm still on the scala learning curve but I don't really missed Groovy and its sugar at all.
-- Brett Knights 250-338-5309 (o) 250-792-3530 (c)
Thu, 2009-03-19, 20:07
#9
Re: file output
On Thu, Mar 19, 2009 at 11:45 AM, Brett Knights <brett@knightsofthenet.com> wrote:
I was under the impression that Scala was designed to be useful for a wide range of uses, from "scripting" to large-scale programming. If so, I think that's a very worthwhile objective.
In Python, you almost never need to close a file explicitly. With the amount of memory on most computers today, that's just not necessary unless perhaps you are opening thousands of files, which I never do.
Russ P.
I don't really think Scala wants to be considered a good scripting language nor do I think Scala has a standard for file output. If you are using a java.io.FileWriter then you get its behavior. Nothing extra added -- I believe that's a good thing.
I was under the impression that Scala was designed to be useful for a wide range of uses, from "scripting" to large-scale programming. If so, I think that's a very worthwhile objective.
In Python, you almost never need to close a file explicitly. With the amount of memory on most computers today, that's just not necessary unless perhaps you are opening thousands of files, which I never do.
Russ P.
Thu, 2009-03-19, 20:37
#10
Re: file output
Russ Paielli wrote:
> On Thu, Mar 19, 2009 at 11:45 AM, Brett Knights
> > wrote:
>
>
> I don't really think Scala wants to be considered a good scripting
> language nor do I think Scala has a standard for file output. If
> you are using a java.io.FileWriter then you get its behavior.
> Nothing extra added -- I believe that's a good thing.
>
>
> I was under the impression that Scala was designed to be useful for a
> wide range of uses, from "scripting" to large-scale programming. If
> so, I think that's a very worthwhile objective.
>
> In Python, you almost never need to close a file explicitly. With the
> amount of memory on most computers today, that's just not necessary
> unless perhaps you are opening thousands of files, which I never do.
>
> Russ P.
Sure, wide range of uses is a great objective. I think scala has that
right now by not being tuned too tightly to any particular usage. If I
write a servlet in scala and want to do file I/O I'll always close the
file explicitly. The app may not exit for months. In that case any hooks
that make sure the file gets closed when the app exits is just baggage.
Thu, 2009-03-19, 22:47
#11
Re: file output
Russ,
If you don't mind adding a dependency, scalax (http://scalax.scalaforge.org/) has the concept of ManagedResource (http://scalax.scalaforge.org/api/scalax/control/ManagedResource.html) that will automatically close resources (such as file streams) when used in a for comprehension.
It also has some implicits that extend File and String that make scripting tasks easier. e.g. The following copies to contents of 'in.txt' to 'out.txt' line by line, automatically opening and closing streams as required.
import scalax.io.Implicits._
import scalax.data.Implicits._
object Test extends Application {
for (writer <- "out.txt".toFile.writer; line <- "in.txt".toFile.lines) {
writer.write(line + "\n")
}
}
Cheers,
Andrew
If you don't mind adding a dependency, scalax (http://scalax.scalaforge.org/) has the concept of ManagedResource (http://scalax.scalaforge.org/api/scalax/control/ManagedResource.html) that will automatically close resources (such as file streams) when used in a for comprehension.
It also has some implicits that extend File and String that make scripting tasks easier. e.g. The following copies to contents of 'in.txt' to 'out.txt' line by line, automatically opening and closing streams as required.
import scalax.io.Implicits._
import scalax.data.Implicits._
object Test extends Application {
for (writer <- "out.txt".toFile.writer; line <- "in.txt".toFile.lines) {
writer.write(line + "\n")
}
}
Cheers,
Andrew
Fri, 2009-03-20, 00:57
#12
Re: file output
2009/3/20 Russ Paielli <russ.paielli@gmail.com>
Well, I did say it and I will continue to say it - FileWriter is a Java class, not part of any Scala library, you can hardly blame Scala for the behaviour of standard Java library classes. As I suggested in my reply, and others in theirs, you can always extend this behaviour in a third-party library (or use something someone else has written) if it does not seem right to you. Alternatively, if enough people think this is a significant issue (I personally don't) then perhaps such a thing can be rolled back into the standard library, or perhaps (much more usefully IMO) something like the ARM blocks I pointed out to you to in my previous message (didn't realise scalax had an implementation - great!!) could be rolled back into the standard library.
In any case, simply using a plain old Java IO class like FileWriter and not closing it will result in the behaviour it has always adhered to - it will not flush. Whether you use it in Scala or Java or any other JVM language is immaterial.
Because doing so encourages sloppy programming that could lead to data loss. Send a SIGKILL to a Pyton program with unflushed open files - this will result in data loss. On the other hand if programmers learnt to always close (or flush at the right transaction boundaries) resources that would minimise data loss even if the process is killed (as long as the close/flush took place early on in the process' lifecycle).
Out of curiosity I had a look at the source for Jython, the JVM implementation of Python, to see what it does. Apparently it does exactly what I suggested in my earlier post - it keeps a reference to every open file and closes it using a JVM shutdown hook. I originally suggested it to you thinking of it as a kludge that I would personally never resort to, but there you go - Jython uses it! For the record, I still think it is a kludge. ARM blocks are a much more useful and reliable mechanism, relying on a JVM hook to do the job is unreliable in cases of sudden termination.
Had a look at the CPython code as well - they implement it using a destructor function, which gets called on deallocation. Suffers from the same problem sudden termination problems as Jython.
On the other hand, if you explicitly closed the file right after using it (or used an ARM block that did that for you), then the program can go on running for any length of time later and you can be assured that the file got written in its entirety when you closed it - regardless of whether the process gets terminated abruptly on or not.
Note that I have tested this - sending a SIGKILL to both a Jython and a CPython process that had opened unflushed files caused data loss.
Or not, depending on your viewpoint/background. Whether Scala wants to be a scripting language is debateable, but I won't go there as I suspect the varied opinions therein are even more highly subjective.
Regards,
Ishaaq
On Thu, Mar 19, 2009 at 12:57 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
Well, I did say it and I will continue to say it - FileWriter is a Java class, not part of any Scala library, you can hardly blame Scala for the behaviour of standard Java library classes. As I suggested in my reply, and others in theirs, you can always extend this behaviour in a third-party library (or use something someone else has written) if it does not seem right to you. Alternatively, if enough people think this is a significant issue (I personally don't) then perhaps such a thing can be rolled back into the standard library, or perhaps (much more usefully IMO) something like the ARM blocks I pointed out to you to in my previous message (didn't realise scalax had an implementation - great!!) could be rolled back into the standard library.
In any case, simply using a plain old Java IO class like FileWriter and not closing it will result in the behaviour it has always adhered to - it will not flush. Whether you use it in Scala or Java or any other JVM language is immaterial.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
Because doing so encourages sloppy programming that could lead to data loss. Send a SIGKILL to a Pyton program with unflushed open files - this will result in data loss. On the other hand if programmers learnt to always close (or flush at the right transaction boundaries) resources that would minimise data loss even if the process is killed (as long as the close/flush took place early on in the process' lifecycle).
Out of curiosity I had a look at the source for Jython, the JVM implementation of Python, to see what it does. Apparently it does exactly what I suggested in my earlier post - it keeps a reference to every open file and closes it using a JVM shutdown hook. I originally suggested it to you thinking of it as a kludge that I would personally never resort to, but there you go - Jython uses it! For the record, I still think it is a kludge. ARM blocks are a much more useful and reliable mechanism, relying on a JVM hook to do the job is unreliable in cases of sudden termination.
Had a look at the CPython code as well - they implement it using a destructor function, which gets called on deallocation. Suffers from the same problem sudden termination problems as Jython.
On the other hand, if you explicitly closed the file right after using it (or used an ARM block that did that for you), then the program can go on running for any length of time later and you can be assured that the file got written in its entirety when you closed it - regardless of whether the process gets terminated abruptly on or not.
Note that I have tested this - sending a SIGKILL to both a Jython and a CPython process that had opened unflushed files caused data loss.
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help.
Or not, depending on your viewpoint/background. Whether Scala wants to be a scripting language is debateable, but I won't go there as I suspect the varied opinions therein are even more highly subjective.
Regards,
Ishaaq
Fri, 2009-03-20, 11:47
#13
RE: file output
> I was under the impression that Scala was designed to be
> useful for a wide range of uses, from "scripting" to
> large-scale programming.
Well, I got the impression that Scala is designed to be
useful for a wide range, from large-scale programming
even down to scripting.
Please note the subtile but important difference!
The scripting scale is provided mostly due to the
expressiveness of general language constructs and
tools like the scala shell.
KR
Det
Fri, 2009-03-20, 11:57
#14
Re: file output
>In Python, you almost never need to close a file explicitly. With the amount of memory on most computers today, that's just not >necessary unless perhaps you are opening thousands of files, which I never do.
Operating systems still only allow a limited number of open files. For example on Solaris the number of file descriptors defaults to something around 1000. With an application that opens a large number of files or sockets not closing them explicitly can be a serious bug.
Python gets around this because it uses reference counting for garbage collection. That means that cleanup of objects (finalization) is deterministic, so you are guaranteed that a file will be closed pretty soon after you lose any references to it. Because the JVM uses proper garbage collection it can't provide this guarantee, and so use of finalization is discouraged.
Resource management blocks is the best that it is possible to do in Scala, and hopefully at some point it will get a nice IO library using them.
Cormac
Operating systems still only allow a limited number of open files. For example on Solaris the number of file descriptors defaults to something around 1000. With an application that opens a large number of files or sockets not closing them explicitly can be a serious bug.
Python gets around this because it uses reference counting for garbage collection. That means that cleanup of objects (finalization) is deterministic, so you are guaranteed that a file will be closed pretty soon after you lose any references to it. Because the JVM uses proper garbage collection it can't provide this guarantee, and so use of finalization is discouraged.
Resource management blocks is the best that it is possible to do in Scala, and hopefully at some point it will get a nice IO library using them.
Cormac
Fri, 2009-03-20, 13:37
#15
Re: file output
Well, I got the impression that Scala is designed to be
useful for a wide range, from large-scale programming
even down to scripting.
Please note the subtile but important difference!
The scripting scale is provided mostly due to the
expressiveness of general language constructs and
tools like the scala shell.
While the expressiveness of the language constructs does remind me of that usually given you by scripting languages, the scala shell itself seems to me more like a sandbox in which to test out ideas rather than a true scripting environment - the lack of the ability to import and run external script files is a big shortcoming if that is its true use (though if it can, or if there are plans to support this in the future, then that would be a pleasent surprise to me).
Ishaaq
Fri, 2009-03-20, 13:47
#16
Re: file output
2009/3/20 Cormac Blackwell <cormac.blackwell@gmail.com>
That is not necessarily the only solution, as I pointed out, you could easily write an IO library that uses JVM shutdown hooks (just as Jython does) to emulate Python's behaviour (in essence keeping a reference to clean up on shutdown), but I'd vote strongly against including such a traversity in the standard library - for the reasons I stated in my earlier post.
Resource management blocks is the best that it is possible to do in Scala, and hopefully at some point it will get a nice IO library using them.
It would indeed be nice if the standard library supported ARM blocks though.
Ishaaq
Fri, 2009-03-20, 14:47
#17
Re: file output
~/test> cat test.scala
def foo = println("this is foo")
println("Hello Ishaaq")
~/test> scala
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.6.0_10).
Type in expressions to have them evaluated.
Type :help for more information.
scala> :help
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.6.0_10).
Type in expressions to have them evaluated.
Type :help for more information.
Type :load followed by a filename to load a Scala file.
Type :replay to reset execution and replay all previous commands.
Type :quit to exit the interpreter.
scala> :load test.scala
Loading test.scala...
foo: Unit
Hello Ishaaq
scala> foo
this is foo
scala> :quit
~/test>
On Fri, Mar 20, 2009 at 5:21 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
def foo = println("this is foo")
println("Hello Ishaaq")
~/test> scala
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.6.0_10).
Type in expressions to have them evaluated.
Type :help for more information.
scala> :help
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.6.0_10).
Type in expressions to have them evaluated.
Type :help for more information.
Type :load followed by a filename to load a Scala file.
Type :replay to reset execution and replay all previous commands.
Type :quit to exit the interpreter.
scala> :load test.scala
Loading test.scala...
foo: Unit
Hello Ishaaq
scala> foo
this is foo
scala> :quit
~/test>
On Fri, Mar 20, 2009 at 5:21 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
While the expressiveness of the language constructs does remind me of that usually given you by scripting languages, the scala shell itself seems to me more like a sandbox in which to test out ideas rather than a true scripting environment - the lack of the ability to import and run external script files is a big shortcoming if that is its true use (though if it can, or if there are plans to support this in the future, then that would be a pleasent surprise to me).
Fri, 2009-03-20, 18:07
#18
Re: file output
Yes, thanks James, I do know that works :)
No, I wasn't clear - what I meant was the inability of one script to load another. The :load command seems to work only in interactive mode - at least it does in v2.7.3.
Which implies that the interpreter can only support monolithic single file scripts. I'm not particularly complaining about this - I was just making a point that the interpreter is really a sandbox for testing out constructs/ideas on the fly - a very useful thing indeed that I appreciate immensely for scala development, however it is nowhere near powerful enough to be considered a proper scripting platform. Hence my misgivings about Scala being used as a scripting language.
However, I would love to be proved wrong - so if there is a way to load a script file non-interactively from within another script file please do let me know.
Ishaaq
2009/3/21 James Iry <jamesiry@gmail.com>
No, I wasn't clear - what I meant was the inability of one script to load another. The :load command seems to work only in interactive mode - at least it does in v2.7.3.
Which implies that the interpreter can only support monolithic single file scripts. I'm not particularly complaining about this - I was just making a point that the interpreter is really a sandbox for testing out constructs/ideas on the fly - a very useful thing indeed that I appreciate immensely for scala development, however it is nowhere near powerful enough to be considered a proper scripting platform. Hence my misgivings about Scala being used as a scripting language.
However, I would love to be proved wrong - so if there is a way to load a script file non-interactively from within another script file please do let me know.
Ishaaq
2009/3/21 James Iry <jamesiry@gmail.com>
~/test> cat test.scala
def foo = println("this is foo")
println("Hello Ishaaq")
~/test> scala
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.6.0_10).
Type in expressions to have them evaluated.
Type :help for more information.
scala> :help
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.6.0_10).
Type in expressions to have them evaluated.
Type :help for more information.
Type :load followed by a filename to load a Scala file.
Type :replay to reset execution and replay all previous commands.
Type :quit to exit the interpreter.
scala> :load test.scala
Loading test.scala...
foo: Unit
Hello Ishaaq
scala> foo
this is foo
scala> :quit
~/test>
On Fri, Mar 20, 2009 at 5:21 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
While the expressiveness of the language constructs does remind me of that usually given you by scripting languages, the scala shell itself seems to me more like a sandbox in which to test out ideas rather than a true scripting environment - the lack of the ability to import and run external script files is a big shortcoming if that is its true use (though if it can, or if there are plans to support this in the future, then that would be a pleasent surprise to me).
Fri, 2009-03-20, 20:37
#19
Re: file output
On Fri, Mar 20, 2009 at 9:56 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
Yes, thanks James, I do know that works :)
No, I wasn't clear - what I meant was the inability of one script to load another. The :load command seems to work only in interactive mode - at least it does in v2.7.3.
Ah, I misunderstood. You're right, scripts can't import other scripts.
Thu, 2009-04-02, 22:47
#20
Re: file output
The FileWriter uses FileOutputStream which flushes the buffer in its finalizer. Now I wasn't aware that the JVM doesn't finalize objects left on the heap by default when the application exits.
The article below says you can change the default by calling System.runFinalizersOnExit(true).
http://www.javaworld.com/javaworld/jw-06-1998/jw-06-techniques.html
Seems it is no news but I thought I'd share it :)
Sebastien
2009/3/19 Russ Paielli <russ.paielli@gmail.com>
The article below says you can change the default by calling System.runFinalizersOnExit(true).
http://www.javaworld.com/javaworld/jw-06-1998/jw-06-techniques.html
Seems it is no news but I thought I'd share it :)
Sebastien
2009/3/19 Russ Paielli <russ.paielli@gmail.com>
On Thu, Mar 19, 2009 at 12:57 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help. That's all I'm saying.
Russ P.
Fri, 2009-04-03, 08:17
#21
Re: file output
On Thu, Apr 2, 2009 at 11:41 PM, Sébastien Bocq <sebastien.bocq@gmail.com> wrote:
The FileWriter uses FileOutputStream which flushes the buffer in its finalizer. Now I wasn't aware that the JVM doesn't finalize objects left on the heap by default when the application exits.
The article below says you can change the default by calling System.runFinalizersOnExit(true).
http://www.javaworld.com/javaworld/jw-06-1998/jw-06-techniques.html
Seems it is no news but I thought I'd share it :)
finalizers was never a good idea, was it?
Sebastien
2009/3/19 Russ Paielli <russ.paielli@gmail.com>On Thu, Mar 19, 2009 at 12:57 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help. That's all I'm saying.
Russ P.
--
Viktor Klang
Senior Systems Analyst
Sat, 2009-04-04, 16:47
#22
Re: file output
I found the idea appealing initially, if the GC frees automatically objects from memory why can it not manage other resources as well? The show stopper is that you cannot make any guarantee on when finalizers will be called which makes your resources management non deterministic. A server may be left with many old files open for example. It impacts performance as well. The allocation of finalizable objects is slower because they must be registered as finalizable by the JVM and the GC will require at least two passes to reclaim them. So if your finalizable objects reference other unreachable objects, it will take more time to garbage collect them as well.
This is explained in more details here:
http://www.devx.com/Java/Article/30192/1954
In conclusion, it is probably better to stay away from finalizers if you write server code. Now if you design a DSL for short scripts, turning on runFinializersOnExit can save some time.
Sebastien
2009/4/3 Viktor Klang <viktor.klang@gmail.com>
This is explained in more details here:
http://www.devx.com/Java/Article/30192/1954
In conclusion, it is probably better to stay away from finalizers if you write server code. Now if you design a DSL for short scripts, turning on runFinializersOnExit can save some time.
Sebastien
2009/4/3 Viktor Klang <viktor.klang@gmail.com>
On Thu, Apr 2, 2009 at 11:41 PM, Sébastien Bocq <sebastien.bocq@gmail.com> wrote:
The FileWriter uses FileOutputStream which flushes the buffer in its finalizer. Now I wasn't aware that the JVM doesn't finalize objects left on the heap by default when the application exits.
The article below says you can change the default by calling System.runFinalizersOnExit(true).
http://www.javaworld.com/javaworld/jw-06-1998/jw-06-techniques.html
Seems it is no news but I thought I'd share it :)
finalizers was never a good idea, was it?
Sebastien
2009/3/19 Russ Paielli <russ.paielli@gmail.com>On Thu, Mar 19, 2009 at 12:57 AM, Ishaaq Chandy <ishaaq@gmail.com> wrote:
Assuming you are using either java.io.FileWriter or java.io.FileOutputStream - in either case it has nothing to do with Scala - these are standard Java classes whose documentation explicitly states that you have to flush/close. If this is not what you expected and you think it is a bug then the only way to fix it is to raise the bug with Sun and wait for a few years :)
I'm using java.io.FileWriter. I don't think you can say "it has nothing to do with Scala" if it is the standard for file output in Scala.
I'm not trying to make a major issue of this. I'm just pointing out that making output files automatically flush and close when the program terminates would be (a) more convenient for the user and (b) less error prone.
Does anyone ever deliberately want output files left unflushed when a program terminates? I doubt it. So why not just make it automatic?
If Scala wants to be considered a good scripting language that can compete with Python, that's one little thing that would help. That's all I'm saying.
Russ P.
--
Viktor Klang
Senior Systems Analyst
Although I loved the flexibility that came with the entire java.io.*
library when _I_ was introduced to Java I must agree that a standard
IO library that makes it really easy to use the basic stuff that we
use most often would be great. Some of it already exists but having
even more of it so Scala is immediately useful for scripting-like
purposes would be really useful.
On Wed, Mar 18, 2009 at 12:51, Detering Dirk wrote:
> Slightly off topic:
>
> When I came to Java from other languages, I was
> irritated by the unintuitive way Java treats the
> file system.
>
> It started with all this different OutputStream
> things, went over to the awkward file rename
> semantics (*) and ended with the detection that
> there is no OO difference between the concepts
> 'file' and 'directory'(**)
>
> For Scala to downscale well into scripting scenarios,
> it would be good to have more support on the most
> common cases.
>
> While Java now does not encapsulate the open and
> close, languages with first-class function support
> should provide that immediately, like what happened
> with the collections.
>
> For example:
>
> File.write { out =>
> out.write("Hello")
> out.write("World")
> }
>
> and the opening and closing is done in the
> write method without the user's attention.
>
>
> ----
> Footnotes use pseudocode for my convenience
>
> (*)
> x = new File("hello")
> x.renameTo("world") <-- doesn't exist ...
> ...you have to:
> y = new File("world")
> x.renameTo(y)
>
> now the entry on disk(!) is renamed to world.
> x as file instance has not changed in any way
> other that it now has lost its status as
> "refers to an existing disk entry".
>
>
> (**)
> In Java, you have to do:
>
> if (myFile.isDirectory() ) {// !
> File y = myFile.listFiles() // <-- returns all entries, not only
> files !
> for( z : y) {
> if (z.isDirectory()) // !
> //do something on dir
> // or alternatively
> if (z.isFile()) // !
> //do something on z,
> // you would not do on dirs, e.g. open and write into
> }
> }
>
> Instead of
>
> Dir[] y = myDir.listDirs()
>
> for (z : y) {
> //do something on z you would only do on dirs
> }
>
> // or alternatively
>
> File[] y = myDir.listFiles()
>
> for(z : y) {
> // do something on z you would never do on dirs
> }
>
> // or perhaps
>
> DirEntry[] y = myDir.list()
>
> for (z : y) {
> z match {
> case Dir(n) =>
> case File(n) =>
> }
> }
>
> KR
> Det
>
>