yields the best of me...

To save you the running, the result is this:

==4
==4
==4
==4
==4
5 threads done
>>>this should be all 4s>>> nullnullnullnullnull

View this message in context: Re: yields the best of me...
Sent from the Scala - User mailing list archive at Nabble.com.

this piece of code got me - i still don't understand why it doesn't work...it may be just because i've been staring at it for too long or there's some underlying screwiness in the lazy evaluation of the yield expressions - does this make sense to you?

package com.razie.pub.learnscala

object CantJoinThreads {
   
   def main (argv:Array[String]) = {
       val result = repeatAndWait (5) {new Integer(4)}
       println (">>>this should be all 4s>>> "+result.mkString)
   }

   /** repeat a number of times on as many threads - but wait for all threads to finish */
    def repeatAndWait[A<:AnyRef] (i:Int) (f: => A) : Iterable[A] = {
            val threads = for (t <- 0 until i) yield new java.lang.Thread() {
                var result : A = null
                override def run() = { 
                    this.result = f 
                    println("=="+this.result)
                }
            }
            
        threads.foreach (_.start)
        threads.foreach (_.join)
        println (threads.length + " threads done")
   
        val res = threads.map (_.result)
        res
  }
}

View this message in context: yields the best of me...
Sent from the Scala - User mailing list archive at Nabble.com.

On Tue, 13 Oct 2009 12:37:17 -0700, Razvan Cojocaru wrote:

> this piece of code got me - i still don't understand why it doesn't
> work...it may be just because i've been staring at it for too long or
> there's some underlying screwiness in the lazy evaluation of the yield
> expressions - does this make sense to you?
>
>
> package com.razie.pub.learnscala
>
> object CantJoinThreads {
>
> def main (argv:Array[String]) = {
> val result = repeatAndWait (5) {new Integer(4)} println (">>>this
> should be all 4s>>> "+result.mkString)
> }
>
> /** repeat a number of times on as many threads - but wait for all
> threads to finish */
> def repeatAndWait[A<:AnyRef] (i:Int) (f: => A) : Iterable[A] = {
> val threads = for (t <- 0 until i) yield new
> java.lang.Thread()
> {
> var result : A = null
> override def run() = {
> this.result = f
> println("=="+this.result)
> }
> }
>
> threads.foreach (_.start)
> threads.foreach (_.join)
> println (threads.length + " threads done")
>
> val res = threads.map (_.result)
> res
> }
> }

(fragment of test.scala):6: error: type mismatch;
found : Null(null)
required: A
var result:A = null

That't can't be right, can it?

On Tue, Oct 13, 2009 at 12:37:44PM -0700, Razvan Cojocaru wrote:
> this piece of code got me - i still don't understand why it doesn't
> work...

I don't even have to read an email which starts this way, I know it'll
be an expression like this:

> val threads = for (t <- 0 until i) yield new java.lang.Thread()

Seriously, I can name that issue in six tunes. That is not a criticism
of you at all, quite the opposite -- it is in my opinion ludicrously
unintuitive that an immutable val is firing off a bunch of new threads
every time you reference it. But so it is. Don't use Range until you
know how it works. OK, I'm sure you didn't know you were using Range...
anyway, (0 until i).toList will make your problem go away.

scala> val threads = for (t <- 0 until 5) yield new java.lang.Thread()
threads: scala.collection.VectorView[java.lang.Thread,Vector[_]] = VectorViewM(Thread[Thread-3,5,main], Thread[Thread-4,5,main], Thread[Thread-5,5,main], Thread[Thread-6,5,main], Thread[Thread-7,5,main])

scala> threads map (_.getId) mkString ", "
res0: String = 20, 21, 22, 23, 24

scala> threads map (_.getId) mkString ", "
res1: String = 25, 26, 27, 28, 29

scala> threads map (_.getId) mkString ", "
res2: String = 30, 31, 32, 33, 34

scala> threads map (_.getId) mkString ", "
res3: String = 35, 36, 37, 38, 39

maybe you're using 2.7 - i'm running the nightly 2.8

Ken Bloom-2 wrote:
>
> On Tue, 13 Oct 2009 12:37:17 -0700, Razvan Cojocaru wrote:
>
> That't can't be right, can it?
>

WOW - Range is out forever. Honest! Given how much time I spent playing with
this...I figured something's off in the VectorView thing but didn't know
who's doing it - I thought it was "yield"...but had gotten tired of
experimenting... :)

So basically, when you read 1 the third time it could be 55 - ...whoever is
responsible owes me a few beers! There must be a reason, but I still want
my beers!

You have one, Paul, btw.

thanks -

Paul Phillips wrote:
>
> On Tue, Oct 13, 2009 at 12:37:44PM -0700, Razvan Cojocaru wrote:
>> this piece of code got me - i still don't understand why it doesn't
>> work...
>
> I don't even have to read an email which starts this way, I know it'll
> be an expression like this:
>
>> val threads = for (t <- 0 until i) yield new
>> java.lang.Thread()
>
> Seriously, I can name that issue in six tunes. That is not a criticism
> of you at all, quite the opposite -- it is in my opinion ludicrously
> unintuitive that an immutable val is firing off a bunch of new threads
> every time you reference it. But so it is. Don't use Range until you
> know how it works. OK, I'm sure you didn't know you were using Range...
> anyway, (0 until i).toList will make your problem go away.
>
> scala> val threads = for (t <- 0 until 5) yield new java.lang.Thread()
> threads: scala.collection.VectorView[java.lang.Thread,Vector[_]] =
> VectorViewM(Thread[Thread-3,5,main], Thread[Thread-4,5,main],
> Thread[Thread-5,5,main], Thread[Thread-6,5,main], Thread[Thread-7,5,main])
>
> scala> threads map (_.getId) mkString ", "
> res0: String = 20, 21, 22, 23, 24
>
> scala> threads map (_.getId) mkString ", "
> res1: String = 25, 26, 27, 28, 29
>
> scala> threads map (_.getId) mkString ", "
> res2: String = 30, 31, 32, 33, 34
>
> scala> threads map (_.getId) mkString ", "
> res3: String = 35, 36, 37, 38, 39
>

If you guys follow http://thereifixedit.com you'll appreciate my kludge at
the bottom.

Now - i didn't try it because I stumbled onto another non-intuitive. When I
used it, the return of my repeatAndWait method became Iterator and
incompatible with Iterable...

It's late and I'm tired and will re-consider tomorow if i'll continue
struggling with scala or switch back to Java, but after a few tens of
thousands of lines of working scala that I have so far, that point is mute.
PLEASE keep it intuitive. Your largest user base will be former Java guys
which may have been C++ superstars years ago, but have had their IQ dropped
in less then half by years of Java...

package scala.razie

/** there, I fixed it! http://thereifixedit.com/ */
object Range {
   def excl (min:Int, max:Int) = new Range (min,max-1)
   def incl (min:Int, max:Int) = new Range (min,max)
}

class Range (min:Int, val max:Int) extends Iterator[Int] {
   var curr = min
   def hasNext:Boolean =  curr < max
   def next:Int = { curr.+=(1); curr}
}

Razvan Cojocaru wrote:
>
> WOW - Range is out forever. Honest! Given how much time I spent playing
> with this...I figured something's off in the

C'est terrible! I couldn't sleep so instead I started to make my kludge work...and it all went horribly wrong. Now I understand that it's the for-yield combination. If I change the generator of the FOR to my kludge, it will create...well, sometimes nothing! The 5 threads will be created and started but after that, the "threads" is empty. To replicate your code:

         val threads = for (t <- scala.razie.Range.excl(0,5)) yield new
java.lang.Thread()
         println ("1 "+threads.mkString)
         println ("2 "+threads.mkString)
         println ("3 "+threads.mkString)

will create this output:

1
Thread[Thread-0,5,main]Thread[Thread-1,5,main]Thread[Thread-2,5,main]Thread[Thread-3,5,main]Thread[Thread-4,5,main]
2 
3 

Which is completely mental. What's different this time is that I get it - my Iterator has state and generates the sequence once... NOW I'm in real trouble....you see, regardless of the language I used/use, I don't normally really care to know what kind of collection/iterable/generator is returned by any one method of any class I deal with - however, now I'm in trouble. I simply cannot use FOR anymore, since the cost of predicting its outcome outweighs any benefits. I mean it's conceptually random if the thing runs once, many times or none at all! I have to seriously reconsider recommending that we as a company, pursue scala on a large scala scale. I don't feel that I can deal with mentoring tens of Java developers that run into this or other similar non-intuitives. Did I bet on the wrong horse? Unless this is some kind of bug that can be fixed and I'm running berserk when I shouldn't be... Razvan Cojocaru wrote: > > If you guys follow http://thereifixedit.com you'll appreciate my kludge at > the bottom. >

> Now - i didn't try it because I stumbled onto another non-intuitive. When > I used it, the return of my repeatAndWait method became Iterator and > incompatible with Iterable... >

> It's late and I'm tired and will re-consider tomorow if i'll continue > struggling with scala or switch back to Java, but after a few tens of > thousands of lines of working scala that I have so far, that point is > mute. PLEASE keep it intuitive. Your largest user base will be former Java > guys which may have been C++ superstars years ago, but have had their IQ > dropped in less then half by years of Java... > >

> package scala.razie
> 
> /** there, I fixed it! http://thereifixedit.com/ */
> object Range {
>    def excl (min:Int, max:Int) = new Range (min,max-1)
>    def incl (min:Int, max:Int) = new Range (min,max)
> }
> 
> class Range (min:Int, val max:Int) extends Iterator[Int] {
>    var curr = min
>    def hasNext:Boolean =  curr <= max
>    def next:Int = { val oldcurr=curr; curr.+=(1); oldcurr}
> }
> 

> > > Razvan Cojocaru wrote: >> >> WOW - Range is out forever. Honest! Given how much time I spent playing >> with this...I figured something's off in the > >

I think you're overreacting.

Yes, "for" does different things depending on whether you use it on an Option, a Stream, a List, a Range, or an Iterator. This is a feature, not a bug. Yes, it means that something you believe to be a Seq[T] can behave very differently depending on what that Seq[T] actually is, but that's true of virtual method dispatch in general and even people whose brains have rotted from years of Java have managed to survive it.

As a matter of library design, I agree that this is a pretty sharp corner. IMO, whatever gets returned by "x to/until y" should -only- have a foreach method, and not a map or flatMap methods. The "for (i <- 0 until n) { .. }" idiom was intended to replace it's Java counterpart, but that limits it to very particular semantics in order to not cause OutOfMemoryErrors for large n. These semantics are familiar to Java programmers when they use "foreach", but entirely surprising when used with "map" or "flatMap" (which is what happens when you use for-yield instead of imperative-for). In short, I think for-yield should be disallowed with "x to/until y". It isn't an idiom present in Java, so the semantics are surprising to Java programmers.

--j

On Tue, Oct 13, 2009 at 9:49 PM, Razvan Cojocaru <razvanc99@yahoo.com> wrote:

C'est terrible!

I couldn't sleep so instead I started to make my kludge work...and it all
went horribly wrong. Now I understand that it's the for-yield combination.
If I change the generator of the FOR to my kludge, it will create...well,
sometimes nothing! The 5 threads will be created and started but after that,
the "threads" is empty. To replicate your code:

<pre>
        val threads = for (t <- scala.razie.Range.excl(0,5)) yield new
java.lang.Thread()
        println ("1 "+threads.mkString)
        println ("2 "+threads.mkString)
        println ("3 "+threads.mkString)
</pre>
will create this output:
<pre>
1
Thread[Thread-0,5,main]Thread[Thread-1,5,main]Thread[Thread-2,5,main]Thread[Thread-3,5,main]Thread[Thread-4,5,main]
2
3
</pre>

Which is completely mental. What's different this time is that I get it - my
Iterator has state and generates the sequence once...

NOW I'm in real trouble....you see, regardless of the language I used/use, I
don't normally really care to know what kind of
collection/iterable/generator is returned by any one method of any class I
deal with - however, now I'm in trouble. I simply cannot use FOR anymore,
since the cost of predicting its outcome outweighs any benefits. I mean it's
conceptually random if the thing runs once, many times or none at all!

I have to seriously reconsider recommending that we as a company, pursue
scala on a large scala scale. I don't feel that I can deal with mentoring
tens of Java developers that run into this or other similar non-intuitives.
Did I bet on the wrong horse?

Unless this is some kind of bug that can be fixed and I'm running berserk
when I shouldn't be...


Razvan Cojocaru wrote:
>
> If you guys follow http://thereifixedit.com you'll appreciate my kludge at
> the bottom.
> <p>
> Now - i didn't try it because I stumbled onto another non-intuitive. When
> I used it, the return of my repeatAndWait method became Iterator and
> incompatible with Iterable...
> <p>
> It's late and I'm tired and will re-consider tomorow if i'll continue
> struggling with scala or switch back to Java, but after a few tens of
> thousands of lines of working scala that I have so far, that point is
> mute. PLEASE keep it intuitive. Your largest user base will be former Java
> guys which may have been C++ superstars years ago, but have had their IQ
> dropped in less then half by years of Java...
>
> <pre>
> package scala.razie
>
> /** there, I fixed it! http://thereifixedit.com/ */
> object Range {
>    def excl (min:Int, max:Int) = new Range (min,max-1)
>    def incl (min:Int, max:Int) = new Range (min,max)
> }
>
> class Range (min:Int, val max:Int) extends Iterator[Int] {
>    var curr = min
>    def hasNext:Boolean =  curr <= max
>    def next:Int = { val oldcurr=curr; curr.+=(1); oldcurr}
> }
> </pre>
>
>
> Razvan Cojocaru wrote:
>>
>> WOW - Range is out forever. Honest! Given how much time I spent playing
>> with this...I figured something's off in the
>
>

--
View this message in context: http://www.nabble.com/yields-the-best-of-me...-tp25879382p25885281.html
Sent from the Scala - User mailing list archive at Nabble.com.

1.) Your for-expression is equivalent to 'scala.razie.Range.excl(0,5).map(_ => new Thread())' So, if you are unsure about what it does look the particular map method up in the documentation. As you might have noticed, implemented in this way the for-expression becomes much more powerful and general than the Java one and warrants some extra study about how it works. 2.) An iterator is a one-shot collection which, in the end, runs dry. So everything is as expected. Replace it with an Iterable and it works like expected. Or even better, use Paul's advice and use (1 to 5).toList or use (for ...).toList . Lazy collections like Range have their place. You just have to make sure not to use them several times if you rely on the identity of the objects or on the side-effects. On Wed, Oct 14, 2009 at 6:49 AM, Razvan Cojocaru wrote: > > C'est terrible! > > I couldn't sleep so instead I started to make my kludge work...and it all > went horribly wrong. Now I understand that it's the for-yield combination. > If I change the generator of the FOR to my kludge, it will create...well, > sometimes nothing! The 5 threads will be created and started but after that, > the "threads" is empty. To replicate your code: > >

>         val threads = for (t <- scala.razie.Range.excl(0,5)) yield new
> java.lang.Thread()
>         println ("1 "+threads.mkString)
>         println ("2 "+threads.mkString)
>         println ("3 "+threads.mkString)
> 

> will create this output: >

> 1
> Thread[Thread-0,5,main]Thread[Thread-1,5,main]Thread[Thread-2,5,main]Thread[Thread-3,5,main]Thread[Thread-4,5,main]
> 2
> 3
> 

> > Which is completely mental. What's different this time is that I get it - my > Iterator has state and generates the sequence once... > > NOW I'm in real trouble....you see, regardless of the language I used/use, I > don't normally really care to know what kind of > collection/iterable/generator is returned by any one method of any class I > deal with - however, now I'm in trouble. I simply cannot use FOR anymore, > since the cost of predicting its outcome outweighs any benefits. I mean it's > conceptually random if the thing runs once, many times or none at all! > > I have to seriously reconsider recommending that we as a company, pursue > scala on a large scala scale. I don't feel that I can deal with mentoring > tens of Java developers that run into this or other similar non-intuitives. > Did I bet on the wrong horse? > > Unless this is some kind of bug that can be fixed and I'm running berserk > when I shouldn't be... > > > Razvan Cojocaru wrote: >> >> If you guys follow http://thereifixedit.com you'll appreciate my kludge at >> the bottom. >>

>> Now - i didn't try it because I stumbled onto another non-intuitive. When >> I used it, the return of my repeatAndWait method became Iterator and >> incompatible with Iterable... >>

>> It's late and I'm tired and will re-consider tomorow if i'll continue >> struggling with scala or switch back to Java, but after a few tens of >> thousands of lines of working scala that I have so far, that point is >> mute. PLEASE keep it intuitive. Your largest user base will be former Java >> guys which may have been C++ superstars years ago, but have had their IQ >> dropped in less then half by years of Java... >> >>

>> package scala.razie
>>
>> /** there, I fixed it! http://thereifixedit.com/ */
>> object Range {
>>    def excl (min:Int, max:Int) = new Range (min,max-1)
>>    def incl (min:Int, max:Int) = new Range (min,max)
>> }
>>
>> class Range (min:Int, val max:Int) extends Iterator[Int] {
>>    var curr = min
>>    def hasNext:Boolean =  curr <= max
>>    def next:Int = { val oldcurr=curr; curr.+=(1); oldcurr}
>> }
>> 

>> >> >> Razvan Cojocaru wrote: >>> >>> WOW - Range is out forever. Honest! Given how much time I spent playing >>> with this...I figured something's off in the >> >> > > -- > View this message in context: http://www.nabble.com/yields-the-best-of-me...-tp25879382p25885281.html > Sent from the Scala - User mailing list archive at Nabble.com. > >

On Tue, 2009-10-13 at 22:58 -0700, Jorge Ortiz wrote:
> In short, I think for-yield should be disallowed with "x to/until y".
> It isn't an idiom present in Java, so the semantics are surprising to
> Java programmers.

Making it strict (which won't happen according to Martin) seems like a
better and less extreme approach than disallowing it altogether while
achieving the same objective (not surprising Java programmers).

Best,
Ismael

Ismael Juma wrote:
> On Tue, 2009-10-13 at 22:58 -0700, Jorge Ortiz wrote:
>> In short, I think for-yield should be disallowed with "x to/until y".
>> It isn't an idiom present in Java, so the semantics are surprising to
>> Java programmers.
>
> Making it strict (which won't happen according to Martin) seems like a
> better and less extreme approach than disallowing it altogether while
> achieving the same objective (not surprising Java programmers).

Yes, there must be a good reason why Range is lazy by default. What is it?

/Jesper Nordenberg

On Wed, 2009-10-14 at 09:32 +0200, Jesper Nordenberg wrote:
> Yes, there must be a good reason why Range is lazy by default. What is it?

Memory usage (particularly for large ranges).

Best,
Ismael

Ismael Juma wrote:
> On Wed, 2009-10-14 at 09:32 +0200, Jesper Nordenberg wrote:
>> Yes, there must be a good reason why Range is lazy by default. What is it?
>
> Memory usage (particularly for large ranges).

A Range object by itself would not take more memory if strict, and most
of the time when you use Range.map you want a strict result. I guess the
cases where laziness helps most is flatMap and filter (although many
range filters can be replaced by a step value), like when nesting ranges:

for (i <- 1 to 10000000; j <- 1 to 10) ...

Which would be quite wasteful in the strict case. I don't see a solution
for that unfortunately.

/Jesper Nordenberg

As a general rule, mixing side effects and laziness (yield isn't
necessarily lazy, but it can be depending on the type your loop is
over) causes you to learn more than you were expecting about how the
things you're using work.

You probably wanted something like Array.fromFunction(x => new
Thread)(i) instead of the for-comprehension.

2009/10/13 Razvan Cojocaru :
> this piece of code got me - i still don't understand why it doesn't
> work...it may be just because i've been staring at it for too long or
> there's some underlying screwiness in the lazy evaluation of the yield
> expressions - does this make sense to you? package com.razie.pub.learnscala
> object CantJoinThreads { def main (argv:Array[String]) = { val result =
> repeatAndWait (5) {new Integer(4)} println (">>>this should be all 4s>>>
> "+result.mkString) } /** repeat a number of times on as many threads - but
> wait for all threads to finish */ def repeatAndWait[A<:AnyRef] (i:Int) (f:
> => A) : Iterable[A] = { val threads = for (t <- 0 until i) yield new
> java.lang.Thread() { var result : A = null override def run() = {
> this.result = f println("=="+this.result) } } threads.foreach (_.start)
> threads.foreach (_.join) println (threads.length + " threads done") val res
> = threads.map (_.result) res } }
> ________________________________
> View this message in context: yields the best of me...
> Sent from the Scala - User mailing list archive at Nabble.com.
>

Thanks, Rick, Paul. You're right. The range and the iterator do what they're
meant to. The for-yield is the problematic bit.

The issue is that what you call lazy is not what I call lazy. A lazy VALUE
is still a value. Nowhere in my code did I ask for a function to be
returned. If I wanted that I would surround the thing in {} or use => or
something explicit like that

val threads = { for (t <- scala.razie.Range.excl(0,5)) yield new
java.lang.Thread() }

In fact what I got back from my simple for is a bound function, assigned to
a VAL with a type Vector-something, which is plain wrong, because it runs my
code above it every time you read the value!

Everyone uses lazy values, but once you got them, they're values. Functions
are those that run every time you call them.

I thought I must've missed something, so I re-read the sections (23&others)
of the scala book (v6) and there is absolutely no warning that there's
side-effects to using the for-yield or map() with anything other than an
idempotent expr2 (or expr3).

I thought about it and the only reasonable solution I could come up with is
that you should not allow me to assign for-yield comprehensions to a val
unless I make them a function, explicitly as above.

Object oriented is generally about modifying state which is a side effect.
You can't expect a Java guy to suddenly mind his side effects when using
something as simple as a for loop, in a language that's supposed to be
friendly to them.

At least you could change the resulting type, from VectorLike-something to
something that contains the idempotent "List ('F', 'u', 'n', 'c').foldRight
("", +)" so I don't have to stare at my (granted nicely white-on-black)
screen for hours trying to figure out how come a Vector's values are null
the third time you inspect it...

I don't think I'm overreacting - I think this particular implementation of a
"lazy" optimisation it's not very well thought out...

Yes, I learned a lesson about the stuff I'm using - not a pleasant one. I
still like other bits of the language and will be very careful when using
for. I will still recommend against it - I believe it will cause more pain
then pleasure to an average Java guy, and will cost the company in man/hours
spent staring instead of saving it moneys, but I will allow it... at some
point.

Now I have to go back and review my own thousands of lines of scala code,
scanning for for/map and spend the rest of my scala days wondering where the
next unexpected bite will come from - it will likely be unexpected and time
consuming...

Ricky Clarkson wrote:
>
> As a general rule, mixing side effects and laziness (yield isn't
> necessarily lazy, but it can be depending on the type your loop is
> over) causes you to learn more than you were expecting about how the
> things you're using work.
>
> You probably wanted something like Array.fromFunction(x => new
> Thread)(i) instead of the for-comprehension.
>

>>>>> "Razvan" == Razvan Cojocaru writes:

Razvan> I thought I must've missed something, so I re-read the sections
Razvan> (23&others) of the scala book (v6) and there is absolutely no
Razvan> warning that there's side-effects to using the for-yield or
Razvan> map() with anything other than an idempotent expr2 (or expr3).

I would agree that the non-strict, non-caching behavior of Range should
have been highlighted in Programming in Scala. (It is covered in
Programming Scala [O'Reilly].)

Once you know about it, I don't think it's that hard to understand or to
work with.

Was just lurking but I just have to reply to this:
"You can't expect a Java guy to suddenly mind his side effects when using something as simple as a for loop"

I'm sorry but in my opinion you cannot say this without batting an eye when Java does not have anything similar as a for-yield. As long as you use the for as you would in Java there is little difference, but the moment you venture into for-yield-country you'd better know the way.
That doesn't mean I don't think there should somehow be a better way to handle this, either by permitting what you were trying to do (although I'm not sure I would agree with that) or by failing in a more obvious way.
Cheers,-Tako


On Wed, Oct 14, 2009 at 15:44, Razvan Cojocaru <razvanc99@yahoo.com> wrote:

Thanks, Rick, Paul. You're right. The range and the iterator do what they're
meant to. The for-yield is the problematic bit.

The issue is that what you call lazy is not what I call lazy. A lazy VALUE
is still a value. Nowhere in my code did I ask for a function to be
returned. If I wanted that I would surround the thing in {} or use => or
something explicit like that

      val threads = { for (t <- scala.razie.Range.excl(0,5)) yield new
java.lang.Thread() }

In fact what I got back from my simple for is a bound function, assigned to
a VAL with a type Vector-something, which is plain wrong, because it runs my
code above it every time you read the value!

Everyone uses lazy values, but once you got them, they're values. Functions
are those that run every time you call them.

I thought I must've missed something, so I re-read the sections (23&others)
of the scala book (v6) and there is absolutely no warning that there's
side-effects to using the for-yield or map() with anything other than an
idempotent expr2 (or expr3).

I thought about it and the only reasonable solution I could come up with is
that you should not allow me to assign for-yield comprehensions to a val
unless I make them a function, explicitly as above.

Object oriented is generally about modifying state which is a side effect.
You can't expect a Java guy to suddenly mind his side effects when using
something as simple as a for loop, in a language that's supposed to be
friendly to them.

At least you could change the resulting type, from VectorLike-something to
something that contains the idempotent "List ('F', 'u', 'n', 'c').foldRight
("", +)" so I don't have to stare at my (granted nicely white-on-black)
screen for hours trying to figure out how come a Vector's values are null
the third time you inspect it...

I don't think I'm overreacting - I think this particular implementation of a
"lazy" optimisation it's not very well thought out...

Yes, I learned a lesson about the stuff I'm using - not a pleasant one. I
still like other bits of the language and will be very careful when using
for. I will still recommend against it - I believe it will cause more pain
then pleasure to an average Java guy, and will cost the company in man/hours
spent staring instead of saving it moneys, but I will allow it... at some
point.

Now I have to go back and review my own thousands of lines of scala code,
scanning for for/map and spend the rest of my scala days wondering where the
next unexpected bite will come from - it will likely be unexpected and time
consuming...


Ricky Clarkson wrote:
>
> As a general rule, mixing side effects and laziness (yield isn't
> necessarily lazy, but it can be depending on the type your loop is
> over) causes you to learn more than you were expecting about how the
> things you're using work.
>
> You probably wanted something like Array.fromFunction(x => new
> Thread)(i) instead of the for-comprehension.
>
> --
> Ricky Clarkson
> Java Programmer, AD Holdings
> +44 1565 770804
> Skype: ricky_clarkson
> Google Talk: ricky.clarkson@gmail.com
>
>

--
View this message in context: http://www.nabble.com/yields-the-best-of-me...-tp25879382p25891359.html
Sent from the Scala - User mailing list archive at Nabble.com.

The rule is simply: you should never perform any side effects in a
function passed to map (which is used by yield), filter or flatMap. If
you must perform side effects, use foreach or while. If you follow that
rule the only potential problem is performance of lazy evaluation, which
often can be fixed quite easily.

Now, it would be great if the Scala compiler could enforce this rule,
but unfortunately it doesn't perform any side effect checking. If you
need this, use another language, for example Haskell or D.

/Jesper Nordenberg

Razvan Cojocaru wrote:
> Thanks, Rick, Paul. You're right. The range and the iterator do what they're
> meant to. The for-yield is the problematic bit.
>
> The issue is that what you call lazy is not what I call lazy. A lazy VALUE
> is still a value. Nowhere in my code did I ask for a function to be
> returned. If I wanted that I would surround the thing in {} or use => or
> something explicit like that
>
> val threads = { for (t <- scala.razie.Range.excl(0,5)) yield new
> java.lang.Thread() }
>
> In fact what I got back from my simple for is a bound function, assigned to
> a VAL with a type Vector-something, which is plain wrong, because it runs my
> code above it every time you read the value!
>
> Everyone uses lazy values, but once you got them, they're values. Functions
> are those that run every time you call them.
>
> I thought I must've missed something, so I re-read the sections (23&others)
> of the scala book (v6) and there is absolutely no warning that there's
> side-effects to using the for-yield or map() with anything other than an
> idempotent expr2 (or expr3).
>
> I thought about it and the only reasonable solution I could come up with is
> that you should not allow me to assign for-yield comprehensions to a val
> unless I make them a function, explicitly as above.
>
> Object oriented is generally about modifying state which is a side effect.
> You can't expect a Java guy to suddenly mind his side effects when using
> something as simple as a for loop, in a language that's supposed to be
> friendly to them.
>
> At least you could change the resulting type, from VectorLike-something to
> something that contains the idempotent "List ('F', 'u', 'n', 'c').foldRight
> ("", +)" so I don't have to stare at my (granted nicely white-on-black)
> screen for hours trying to figure out how come a Vector's values are null
> the third time you inspect it...
>
> I don't think I'm overreacting - I think this particular implementation of a
> "lazy" optimisation it's not very well thought out...
>
> Yes, I learned a lesson about the stuff I'm using - not a pleasant one. I
> still like other bits of the language and will be very careful when using
> for. I will still recommend against it - I believe it will cause more pain
> then pleasure to an average Java guy, and will cost the company in man/hours
> spent staring instead of saving it moneys, but I will allow it... at some
> point.
>
> Now I have to go back and review my own thousands of lines of scala code,
> scanning for for/map and spend the rest of my scala days wondering where the
> next unexpected bite will come from - it will likely be unexpected and time
> consuming...
>
>
> Ricky Clarkson wrote:
>> As a general rule, mixing side effects and laziness (yield isn't
>> necessarily lazy, but it can be depending on the type your loop is
>> over) causes you to learn more than you were expecting about how the
>> things you're using work.
>>
>> You probably wanted something like Array.fromFunction(x => new
>> Thread)(i) instead of the for-comprehension.
>>

> Object oriented is generally about modifying state which is a side effect.
> You can't expect a Java guy to suddenly mind his side effects when using
> something as simple as a for loop, in a language that's supposed to be
> friendly to them.

It wasn't a for loop.

For reference, C#'s LINQ and C#'s yield have the same 'problem', but
are used without serious issues by many many OO-ish programmers.

C#'s yield is always lazy though.

Try not to overreact to features you've only just discovered, let them
swirl around your mouth for a bit first.

Yeah - that pretty much drives the proper nails the coffin... :)

Until just now, I still considered a for (a <- something) yield something(a)
to be a shorthand for a for loop with a collecting feature.

I think the scala books and introductory articles need special mention of
the for-yield and map() behaviour. I've read quite a bit and there was never
a warning...simple behavioural tip - this is called anchoring:you say "for"
and I let my guard down thinking it's just another of those things that
start with a "for"...

thanks

Ricky Clarkson wrote:
>
>> Object oriented is generally about modifying state which is a side
>> effect.
>> You can't expect a Java guy to suddenly mind his side effects when using
>> something as simple as a for loop, in a language that's supposed to be
>> friendly to them.
>
> It wasn't a for loop.
>
> For reference, C#'s LINQ and C#'s yield have the same 'problem', but
> are used without serious issues by many many OO-ish programmers.
>
> C#'s yield is always lazy though.
>
> Try not to overreact to features you've only just discovered, let them
> swirl around your mouth for a bit first.
>

> I have to seriously reconsider recommending that we as a
> company, pursue scala on a large scala scale. I don't feel
> that I can deal with mentoring tens of Java developers that
> run into this or other similar non-intuitives.
> Did I bet on the wrong horse?
[...]
> Yes, I learned a lesson about the stuff I'm using - not a
> pleasant one. I still like other bits of the language and
> will be very careful when using for. I will still recommend
> against it - I believe it will cause more pain then pleasure
> to an average Java guy, and will cost the company in
> man/hours spent staring instead of saving it moneys, but I
> will allow it... at some point.

I think

1.) ... you are tired. :-) Ok.

2.) ... that Scala has enough advantages that weight out the
disadvantages of some pitfalls.

2.) ... you should indeed reconsider recommending scala in your
company, if this kind of problems cause you so much concerns
and trouble.

3.) .. you should indeed reconsider trying out any new technology
or language not already present and known in your company.

The latter, because I think that _every_ change, be it some
new technology or new language somehow foreign to a Java-only
mindset will indeed have one or more painpoints and pitfalls somewhere.
Every.

That is the reason why _every_ change will at first increase
the costs, to achieve the mind transfer to the new thing.
The gain will come afterwards, when the transfer is done.

So if you can't afford the walk through the marshes to reach
a fertile but unknown terrain, stay on the dry land and keep
struggling with the hard but indeed well known earth.
But please don't go out and complain then.

KR
Det

One needs hate in order to understand love - leave the emotional bit at
that...

About complaining - let's agree to disagree. I doubt, therefore I am. If
nobody complained about potholes, would they ever be fixed? Your society
would attract a "visionary" leader which is corrupted by power and slides
into dictatorship. Back to potholes...

After many moths and thousands of lines of code in scala, I don't think
that, overall, it would make an average developer any more productive, so
I'll keep my recommendation simply because of the short term risk and costs.
As scala developers become available off the shelf, we'll obviously hire
them :) yeah, I'm mean - I know that.

I still think some fixes in this area are needed. I'd like to explicitly
recognise the creation of a function...or at least that returned type name
should make its behaviour more obvious.

Detering Dirk-2 wrote:
>
>> I have to seriously reconsider recommending that we as a
>> company, pursue scala on a large scala scale. I don't feel
>> that I can deal with mentoring tens of Java developers that
>> run into this or other similar non-intuitives.
>> Did I bet on the wrong horse?
> [...]
>> Yes, I learned a lesson about the stuff I'm using - not a
>> pleasant one. I still like other bits of the language and
>> will be very careful when using for. I will still recommend
>> against it - I believe it will cause more pain then pleasure
>> to an average Java guy, and will cost the company in
>> man/hours spent staring instead of saving it moneys, but I
>> will allow it... at some point.
>
> I think
>
> 1.) ... you are tired. :-) Ok.
>
> 2.) ... that Scala has enough advantages that weight out the
> disadvantages of some pitfalls.
>
> 2.) ... you should indeed reconsider recommending scala in your
> company, if this kind of problems cause you so much concerns
> and trouble.
>
> 3.) .. you should indeed reconsider trying out any new technology
> or language not already present and known in your company.
>
> The latter, because I think that _every_ change, be it some
> new technology or new language somehow foreign to a Java-only
> mindset will indeed have one or more painpoints and pitfalls somewhere.
> Every.
>
> That is the reason why _every_ change will at first increase
> the costs, to achieve the mind transfer to the new thing.
> The gain will come afterwards, when the transfer is done.
>
> So if you can't afford the walk through the marshes to reach
> a fertile but unknown terrain, stay on the dry land and keep
> struggling with the hard but indeed well known earth.
> But please don't go out and complain then.
>
> KR
> Det
>
>

On Wed, Oct 14, 2009 at 03:18:19PM +0100, Ricky Clarkson wrote:
> Try not to overreact to features you've only just discovered, let them
> swirl around your mouth for a bit first.

While I agree the original poster is overreacting a bit, I also think
it's a big big problem and as you know this isn't my first time at the
dance. The problem is not the behavior of Range per se, but that the
default, encouraged by example, easiest to access mechanism for
iterating over a sequence of integers is what it is.

And in response so seth, the point is also not how hard it is to deal
with it once you know about it, the point is the tax it imposes to find
out about it in the first place.

I hadn't heard ijuma's idea about only having foreach work on ranges
obtained via to/until. I think it's a great idea. It would break some
code, but I will predict once again, this will eventually change anyway
after the Nth monthly iteration of this thread. (Hey, it's like the
thread was obtained via Range.) The sooner we change it the less we
break. Let's just set N=Now.

But making Range strict, as you point out, would make to/until have surprising memory side-effects.

I think to/until should replace the standard Java idiom of:

  for (int i = 0; i < n; i ++) {
    ...
  }

That means using a lazy collection that allows only foreach (and probably filter) and returns only Unit.

For those who understand and relish the behavior of Range, a full Range implementation (with foreach/map/flatMap/etc) should be provided, but it should not be accessible via the to/until idiom encouraged to replace Java behavior. You should have to explicitly construct such a full-featured range. Then, if you use explicitly use a class you don't understand, you deserve what you get.

--j

On Tue, Oct 13, 2009 at 11:43 PM, Ismael Juma <mlists@juma.me.uk> wrote:

On Tue, 2009-10-13 at 22:58 -0700, Jorge Ortiz wrote:
> In short, I think for-yield should be disallowed with "x to/until y".
> It isn't an idiom present in Java, so the semantics are surprising to
> Java programmers.

Making it strict (which won't happen according to Martin) seems like a
better and less extreme approach than disallowing it altogether while
achieving the same objective (not surprising Java programmers).

Best,
Ismael

On Wed, Oct 14, 2009 at 10:19:59AM -0700, Jorge Ortiz wrote:
> That means using a lazy collection that allows only foreach (and
> probably filter) and returns only Unit.

It occurs to me that if we change to/until to return this limited Range
with a couple fewer methods, we "break" code in the sense that we stop
some code from compiling -- but I would bet the farm what we are
actually doing in almost every case is uncovering bugs, either
correctness or performance or both.

So one could look at it as a pure public service, not really a breaking
change. How often does anyone explicitly want the map behavior? I would
be interested to hear in what contexts it is used intentionally.

On Wed, 2009-10-14 at 08:37 -0700, Paul Phillips wrote:
> I hadn't heard ijuma's idea about only having foreach work on ranges
> obtained via to/until. I think it's a great idea.

To be fair, it was Jorge's idea. :)

Best,
Ismael

>>>>> "Paul" == Paul Phillips writes:

Paul> So one could look at it as a pure public service, not really a
Paul> breaking change. How often does anyone explicitly want the map
Paul> behavior?

I certainly don't.

Is there a possibility of not forbidding Range.map, but just having it
return something strict? (or if not strict, at least caching)

I hadn't heard ijuma's idea about only having foreach work on ranges
obtained via to/until.  I think it's a great idea.  It would break some
code,

+1

Odds are, most code affected by this change would either be already broken (or in risk of breaking at the slightest side-effecting change) or the author is fully aware of and expects the lazy behavior, and the fix is as simple as (1 to n).toStream (barring an as-yet-to-be-named method for creating a full lazy range). I would also guess that the number of cases where people are intentionally relying on laziness in for-yield expressions is relatively small (at least, relative to the number of times this comes up on the list).

- Colin

On Wed, 2009-10-14 at 10:19 -0700, Jorge Ortiz wrote:
> But making Range strict, as you point out, would make to/until have
> surprising memory side-effects.

Well, to/until can continue as it is. One could just change methods like
map to be strict. This was also suggested by Jesper elsewhere.

> I think to/until should replace the standard Java idiom of:
>
> for (int i = 0; i < n; i ++) {
> ...
> }
>
> That means using a lazy collection that allows only foreach (and
> probably filter) and returns only Unit.
>
> For those who understand and relish the behavior of Range, a full
> Range implementation (with foreach/map/flatMap/etc) should be
> provided, but it should not be accessible via the to/until idiom
> encouraged to replace Java behavior.

This is probably better than the current situation, but I am not
convinced whether it's better than just making map and co. strict
without affecting memory usage of Range itself.

Best,
Ismael

On Wed, Oct 14, 2009 at 5:37 PM, Paul Phillips wrote:
> On Wed, Oct 14, 2009 at 03:18:19PM +0100, Ricky Clarkson wrote:
>> Try not to overreact to features you've only just discovered, let them
>> swirl around your mouth for a bit first.
>
> While I agree the original poster is overreacting a bit, I also think
> it's a big big problem and as you know this isn't my first time at the
> dance.  The problem is not the behavior of Range per se, but that the
> default, encouraged by example, easiest to access mechanism for
> iterating over a sequence of integers is what it is.
>

http://scala.sygneca.com/common/list-manipulation - last edited 2008/03/26

I'm sure this has come up many more times before that, there was a
large thread back then (the examples sprang forth).

Strict by default seems to be the least shocking (followed by
.projection or .view or .whatever-its-called). I guess that is
particularly difficult to do now though.

> The sooner we change it the less we break. Let's just set N=Now.

I'm personally ok with code breakage now, but I'm sure if I had
productive code I'd be less inclined :-)

Caching just trades one class of bugs (surprising side-effects) for another class of bugs (surprising memory leaks).

--j

On Wed, Oct 14, 2009 at 10:47 AM, Seth Tisue <seth@tisue.net> wrote:

>>>>> "Paul" == Paul Phillips <paulp@improving.org> writes:

 Paul> So one could look at it as a pure public service, not really a
 Paul> breaking change.  How often does anyone explicitly want the map
 Paul> behavior?

I certainly don't.

Is there a possibility of not forbidding Range.map, but just having it
return something strict?  (or if not strict, at least caching)

--
Seth Tisue @ Northwestern University / http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

+1 on the Range.foreach

Solving the Range I think is addressing just part of the issue - maybe the
largest and most common. As much as you are right about me overreacting, I'm
also right to be overreacting to someone overloading the "for" to mean and
do something else...well, sometimes!. How would you like me to override
foldLeft to do foldRightAndBackAgain5Times ...?

Since Java6, the for (x : mycollection/iterable/whathaveyou) is a lifesaver,
missing only the result collection part which I (and probably many others)
thought "yield" was...

Let's keep that on the radar as well, please. That is, to make sure that all
people that see a simple for construct all agree largely on what it means,
can debug it and maintain it...maybe even language aside...

Done overreacting here :)
Razvan

>>>>> "Jorge" == Jorge Ortiz writes:

>> Is there a possibility of not forbidding Range.map, but just having
>> it return something strict? (or if not strict, at least caching)

Jorge> Caching just trades one class of bugs (surprising side-effects)
Jorge> for another class of bugs (surprising memory leaks).

Do you have an example of how that might actually occur in the context
of Range.map...?

Ok, let's explore the design space.

We're all agreed that the following should run in constant space and return Unit.

   for (i <- 0 until 1000000000) ...

The question now is what this next expression should mean:

  val x = for (i <- 0 until 1000000000) yield ...

Under the current semantics, it runs in constant space (and constant time!), but if "..." has side effects (generating random numbers, creating threads, starting actors) then results are surprising.

Under strict semantics, the map expression would OOM. So going from foreach to map could turn correct code into OOMing code.

Under lazy-but-cached semantics, the expression would still run in constant space (and constant time! side-effects wouldn't happen until sometime later), but it would have the possibility of OOMing down the road, after stuff has been evaluated (and good luck tracking that down).

I think all three possible behaviors will be surprising to someone at some point. IMO, the library should not make that choice for the user. The user should explicitly chose strict (List.range(x, y)), lazy-but-cached (Stream.range(x, y)), or lazy (SomeNewCollection.range(x, y)) semantics.

--j

On Wed, Oct 14, 2009 at 10:52 AM, Ismael Juma <mlists@juma.me.uk> wrote:

On Wed, 2009-10-14 at 10:19 -0700, Jorge Ortiz wrote:
> But making Range strict, as you point out, would make to/until have
> surprising memory side-effects.

Well, to/until can continue as it is. One could just change methods like
map to be strict. This was also suggested by Jesper elsewhere.

> I think to/until should replace the standard Java idiom of:
>
>   for (int i = 0; i < n; i ++) {
>     ...
>   }
>
> That means using a lazy collection that allows only foreach (and
> probably filter) and returns only Unit.
>
> For those who understand and relish the behavior of Range, a full
> Range implementation (with foreach/map/flatMap/etc) should be
> provided, but it should not be accessible via the to/until idiom
> encouraged to replace Java behavior.

This is probably better than the current situation, but I am not
convinced whether it's better than just making map and co. strict
without affecting memory usage of Range itself.

Best,
Ismael

How about

1) Override the <- operator, applied to the right, to generate a proper
wrapper to the basic types it can be applied to, so that the
for (x <- something) yield XX
has a finite result?

OR, I like this more:

2) sugarize the syntax even further by introducing a new operator/symbol for
for:
for (x : Iterable/Range/whatever) yield XX
for (x ^ Iterable/Range/whatever) yield XX

to do the deed? In this case, the collection would be finite and you'd do
what? automatically apply a ".toList" to the result of yield? OR have a
yieldaslist or yieldasprojcetion flavors? "collect" ? Yeah, how about
"collect" ?

There's gotta be something in this direction...

Razvan,

The way Scala works is that:

  for (x <- c) f(x)

simply translates to:

  c.foreach(x => f(x))

Likewise:

  for (x <- c) yield f(x)

simply translates to:

  c.map(x => f(x))

Something with a guard:

  for (x <- c if p(x)) yield f(x)

translates to:

  c.filter(x => p(x)).map(x => f(x))

And a nested for:

  for (x1 <- c1; x2 <- c2) yield f(x1, x2)

translates to:

  c1.flatMap(x1 => c2.map(x2 => f(x1, x2)))

If you know these rules, you will ALWAYS know exactly what "for" is doing. It has a very predictable, very mechanical translation. There is never anything surprising about it's behavior.

In this case, "0 until 5" returns an instance of the Range class, and it's the behavior of Range's "map" and "flatMap" methods that make the behavior you observed surprising. It's not worth it to change the spec of "for" (a core, core language feature) when a change to a couple of methods in Range (a small part of the standard library) can have the same effect.

--j

On Wed, Oct 14, 2009 at 11:23 AM, Razvan Cojocaru <razvanc99@yahoo.com> wrote:

How about

1) Override the <- operator, applied to the right, to generate a proper
wrapper to the basic types it can be applied to, so that the
    for (x <- something) yield XX
has a finite result?

OR, I like this more:

2) sugarize the syntax even further by introducing a new operator/symbol for
for:
    for (x : Iterable/Range/whatever) yield XX
    for (x ^ Iterable/Range/whatever) yield XX

to do the deed? In this case, the collection would be finite and you'd do
what? automatically apply a ".toList" to the result of yield? OR have a
yieldaslist or yieldasprojcetion flavors? "collect" ? Yeah, how about
"collect" ?

There's gotta be something in this direction...

--
View this message in context: http://www.nabble.com/yields-the-best-of-me...-tp25879382p25896410.html
Sent from the Scala - User mailing list archive at Nabble.com.

On Wed, 14 Oct 2009 08:37:19 -0700, Paul Phillips wrote:

> On Wed, Oct 14, 2009 at 03:18:19PM +0100, Ricky Clarkson wrote:
>> Try not to overreact to features you've only just discovered, let them
>> swirl around your mouth for a bit first.
>
> While I agree the original poster is overreacting a bit, I also think
> it's a big big problem and as you know this isn't my first time at the
> dance. The problem is not the behavior of Range per se, but that the
> default, encouraged by example, easiest to access mechanism for
> iterating over a sequence of integers is what it is.
>
> And in response so seth, the point is also not how hard it is to deal
> with it once you know about it, the point is the tax it imposes to find
> out about it in the first place.
>
> I hadn't heard ijuma's idea about only having foreach work on ranges
> obtained via to/until. I think it's a great idea. It would break some
> code, but I will predict once again, this will eventually change anyway
> after the Nth monthly iteration of this thread. (Hey, it's like the
> thread was obtained via Range.) The sooner we change it the less we
> break. Let's just set N=Now.

Congratulations -- your suggestion just killed a bunch of functional code
I wrote yesterday. I start with:
val possibleMoves = for (mC <- 0 to 2; cC <- 0 to 2;
if mC+cC==1 || mC+cC==2) yield (mC,cC)
and then create some immutable objects based on transforming another
object, finally take one pass through the resulting sequence of objects,
and place all of those objects on a queue.

Maybe you'd consider making just the *map* method strict, but killing off
perfectly good iteration tools because people can't handle the side
effects is not nice.

I understand what it does...now at least :) :)

However, after Range, I tried with an Iterator - literally, the simplest
version of an Iterator - and had another set of issues. Because the way
map() behaves on Iterator[A] - see my second reply in the thread, with the
thereifixedit kludge...

My point is that the thing behaves completely differently depending on the
collection you pass it. When writing higher-level logic, the actual
collection type is largely as irrelevant as the types of the variables and
functions inferred by the smart scala compiler. That's why they're generally
called Collections and have different features like uniqueness, no nulls,
order LAZYNESS and others you normally safely ignore when iterating...etc.

So...I was trying to address the larger problem.

But....enough folks explained that my point is not valid....so this is the
last mention of it.

Jorge Ortiz-3 wrote:
>
> Razvan,
>
> The way Scala works is that:
>
> for (x <- c) f(x)
>
> simply translates to:
>
> c.foreach(x => f(x))
>
> Likewise:
>
> for (x <- c) yield f(x)
>
> simply translates to:
>
> c.map(x => f(x))
>
> Something with a guard:
>
> for (x <- c if p(x)) yield f(x)
>
> translates to:
>
> c.filter(x => p(x)).map(x => f(x))
>
> And a nested for:
>
> for (x1 <- c1; x2 <- c2) yield f(x1, x2)
>
> translates to:
>
> c1.flatMap(x1 => c2.map(x2 => f(x1, x2)))
>
> If you know these rules, you will ALWAYS know exactly what "for" is doing.
> It has a very predictable, very mechanical translation. There is never
> anything surprising about it's behavior.
>
> In this case, "0 until 5" returns an instance of the Range class, and it's
> the behavior of Range's "map" and "flatMap" methods that make the behavior
> you observed surprising. It's not worth it to change the spec of "for" (a
> core, core language feature) when a change to a couple of methods in Range
> (a small part of the standard library) can have the same effect.
>
> --j
>
> On Wed, Oct 14, 2009 at 11:23 AM, Razvan Cojocaru
> wrote:
>
>>
>>
>> How about
>>
>> 1) Override the <- operator, applied to the right, to generate a proper
>> wrapper to the basic types it can be applied to, so that the
>> for (x <- something) yield XX
>> has a finite result?
>>
>> OR, I like this more:
>>
>> 2) sugarize the syntax even further by introducing a new operator/symbol
>> for
>> for:
>> for (x : Iterable/Range/whatever) yield XX
>> for (x ^ Iterable/Range/whatever) yield XX
>>
>> to do the deed? In this case, the collection would be finite and you'd do
>> what? automatically apply a ".toList" to the result of yield? OR have a
>> yieldaslist or yieldasprojcetion flavors? "collect" ? Yeah, how about
>> "collect" ?
>>
>> There's gotta be something in this direction...
>>
>> --
>> View this message in context:
>> http://www.nabble.com/yields-the-best-of-me...-tp25879382p25896410.html
>> Sent from the Scala - User mailing list archive at Nabble.com.
>>
>>
>
>

> Razvan,
>
> The way Scala works is that:
>
> for (x <- c) f(x)
>
> simply translates to: [...]

I'll print that out and pin it up in my bedroom! Great mail, Jorge, but
I don't think these rules are easy to memorize. They take some time
getting used to.

Yeah, iterators are annoying, as they can only be used once. This is true in Java as well as in Scala. With Scala it's easier to use them inadvertently, as things like "foreach" and "mkString" consume the entire iterator.

Yes, "for" behaves differently depending on which collection you pass it. But that's, in fact, the whole point of Scala's "for" (as opposed to Java's "for"). And here I'm going to blow your mind: you can use "for" even with things that -aren't- collections! Any object which defines foreach/map/flatMap/filter can be used inside with "for", regardless of whether it's a collection or not. You can use "for" for error handling (http://stackoverflow.com/questions/1250957/method-dependencies-and-error-handling), continuations (http://scala.sygneca.com/libs/responder), and even transactions (http://github.com/dpp/liftweb/tree/wip-jta-jonas/lift-jta/).

Scala's "for" is very simple. It's just a straightforward translation to foreach/map/flatMap/filter. However, because of this simplicity, it's also very poweful and useful in all kinds of situations.

Scala could cripple "for", but then it wouldn't be as powerful.

--j

On Wed, Oct 14, 2009 at 12:11 PM, Razvan Cojocaru <razvanc99@yahoo.com> wrote:

I understand what it does...now at least :) :)

However, after Range, I tried with an Iterator - literally, the simplest
version of an Iterator - and had another set of issues. Because the way
map() behaves on Iterator[A] - see my second reply in the thread, with the
thereifixedit kludge...

My point is that the thing behaves completely differently depending on the
collection you pass it. When writing higher-level logic, the actual
collection type is largely as irrelevant as the types of the variables and
functions inferred by the smart scala compiler. That's why they're generally
called Collections and have different features like uniqueness, no nulls,
order LAZYNESS and others you normally safely ignore when iterating...etc.

So...I was trying to address the larger problem.

But....enough folks explained that my point is not valid....so this is the
last mention of it.


Jorge Ortiz-3 wrote:
>
> Razvan,
>
> The way Scala works is that:
>
>   for (x <- c) f(x)
>
> simply translates to:
>
>   c.foreach(x => f(x))
>
> Likewise:
>
>   for (x <- c) yield f(x)
>
> simply translates to:
>
>   c.map(x => f(x))
>
> Something with a guard:
>
>   for (x <- c if p(x)) yield f(x)
>
> translates to:
>
>   c.filter(x => p(x)).map(x => f(x))
>
> And a nested for:
>
>   for (x1 <- c1; x2 <- c2) yield f(x1, x2)
>
> translates to:
>
>   c1.flatMap(x1 => c2.map(x2 => f(x1, x2)))
>
> If you know these rules, you will ALWAYS know exactly what "for" is doing.
> It has a very predictable, very mechanical translation. There is never
> anything surprising about it's behavior.
>
> In this case, "0 until 5" returns an instance of the Range class, and it's
> the behavior of Range's "map" and "flatMap" methods that make the behavior
> you observed surprising. It's not worth it to change the spec of "for" (a
> core, core language feature) when a change to a couple of methods in Range
> (a small part of the standard library) can have the same effect.
>
> --j
>
> On Wed, Oct 14, 2009 at 11:23 AM, Razvan Cojocaru
> <razvanc99@yahoo.com>wrote:
>
>>
>>
>> How about
>>
>> 1) Override the <- operator, applied to the right, to generate a proper
>> wrapper to the basic types it can be applied to, so that the
>>     for (x <- something) yield XX
>> has a finite result?
>>
>> OR, I like this more:
>>
>> 2) sugarize the syntax even further by introducing a new operator/symbol
>> for
>> for:
>>     for (x : Iterable/Range/whatever) yield XX
>>     for (x ^ Iterable/Range/whatever) yield XX
>>
>> to do the deed? In this case, the collection would be finite and you'd do
>> what? automatically apply a ".toList" to the result of yield? OR have a
>> yieldaslist or yieldasprojcetion flavors? "collect" ? Yeah, how about
>> "collect" ?
>>
>> There's gotta be something in this direction...
>>
>> --
>> View this message in context:
>> http://www.nabble.com/yields-the-best-of-me...-tp25879382p25896410.html
>> Sent from the Scala - User mailing list archive at Nabble.com.
>>
>>
>
>

--
View this message in context: http://www.nabble.com/yields-the-best-of-me...-tp25879382p25897175.html
Sent from the Scala - User mailing list archive at Nabble.com.

Hi Jorge,

Your explanation how `for' works is the most concise I have seen so far. Thanks!

On Wed, 2009-10-14 at 11:15 -0700, Jorge Ortiz wrote:
> The question now is what this next expression should mean:
>
> val x = for (i <- 0 until 1000000000) yield ...
>
> Under the current semantics, it runs in constant space (and constant
> time!), but if "..." has side effects (generating random numbers,
> creating threads, starting actors) then results are surprising.
>
> Under strict semantics, the map expression would OOM. So going from
> foreach to map could turn correct code into OOMing code.

It may OOM depending on heap size and the result of yield, but I don't
think it's a big problem to ask people who are dealing with such large
collections and don't intend to force the full collection to use Stream.

> I think all three possible behaviors will be surprising to someone at
> some point. IMO, the library should not make that choice for the user.
> The user should explicitly chose strict (List.range(x, y)),
> lazy-but-cached (Stream.range(x, y)), or lazy
> (SomeNewCollection.range(x, y)) semantics.

I think to/until should handle the common case and the methods you
outline should be used for more specific cases. In any case let's move
this to debate as Martin has done a concrete proposal there (which I
liked - although I just skimmed it).

Best,
Ismael

I think "surprising" is not really the case. It would be surprising for existing code, that is certain. But we do have to consider: is the amount of Scala code written from this point forward quickly overtake the amount of code written so far?   We could easily add a "view" to Range, for those applications where non-strictness is required, so it's not a an either/or question. It's a question of:   1) Is a non-strict Range causing problems? 2) Do we care about these problems enough to change the language? 3) Are we going to break code by changing it?   The answer to 1 is obvious, particularly since for/yield seems to be the preferred choice for creating threads and futures.   In answer to 3, there are the performance/memory problems that were indicated. Also, it will break code that intentionally uses the non-strictness to produce variable results.   And then there's 2, essentially subjective, balancing between 1 and 3.

On Wed, Oct 14, 2009 at 2:19 PM, Jorge Ortiz <jorge.ortiz@gmail.com> wrote:

But making Range strict, as you point out, would make to/until have surprising memory side-effects.

I think to/until should replace the standard Java idiom of:

  for (int i = 0; i < n; i ++) {
    ...
  }

That means using a lazy collection that allows only foreach (and probably filter) and returns only Unit.

For those who understand and relish the behavior of Range, a full Range implementation (with foreach/map/flatMap/etc) should be provided, but it should not be accessible via the to/until idiom encouraged to replace Java behavior. You should have to explicitly construct such a full-featured range. Then, if you use explicitly use a class you don't understand, you deserve what you get.

--j

On Tue, Oct 13, 2009 at 11:43 PM, Ismael Juma <mlists@juma.me.uk> wrote:

On Tue, 2009-10-13 at 22:58 -0700, Jorge Ortiz wrote:
> In short, I think for-yield should be disallowed with "x to/until y".
> It isn't an idiom present in Java, so the semantics are surprising to
> Java programmers.

Making it strict (which won't happen according to Martin) seems like a
better and less extreme approach than disallowing it altogether while
achieving the same objective (not surprising Java programmers).

Best,
Ismael

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

Can you repost the link you mentioned? I couldn't find it.

--j

On Thu, Oct 15, 2009 at 4:11 AM, Ismael Juma <mlists@juma.me.uk> wrote:

On Wed, 2009-10-14 at 11:15 -0700, Jorge Ortiz wrote:
> The question now is what this next expression should mean:
>
>   val x = for (i <- 0 until 1000000000) yield ...
>
> Under the current semantics, it runs in constant space (and constant
> time!), but if "..." has side effects (generating random numbers,
> creating threads, starting actors) then results are surprising.
>
> Under strict semantics, the map expression would OOM. So going from
> foreach to map could turn correct code into OOMing code.

It may OOM depending on heap size and the result of yield, but I don't
think it's a big problem to ask people who are dealing with such large
collections and don't intend to force the full collection to use Stream.

> I think all three possible behaviors will be surprising to someone at
> some point. IMO, the library should not make that choice for the user.
> The user should explicitly chose strict (List.range(x, y)),
> lazy-but-cached (Stream.range(x, y)), or lazy
> (SomeNewCollection.range(x, y)) semantics.

I think to/until should handle the common case and the methods you
outline should be used for more specific cases. In any case let's move
this to debate as Martin has done a concrete proposal there (which I
liked - although I just skimmed it).

Best,
Ismael

On Thu, 2009-10-15 at 12:08 -0700, Jorge Ortiz wrote:
> Can you repost the link you mentioned? I couldn't find it.

http://article.gmane.org/gmane.comp.lang.scala.debate/3474

Best,
Ismael