questioning FP

On Mon, Oct 10, 2011 at 3:11 AM, Ittay Dror <ittay.dror@gmail.com> wrote:

I'd love to see a code example where the introduction of IO[_] makes more errors detectable. In the code example I used in the first message, I couldn't find where it helps.

Consider the following code outline, and assume a purely functional implementation of Scala (even if strict!)
object A {   def foo(String):Int = ...  def bar(String):Int = ...} Then the program:
object B {  def run(f:String,b:String):Int =     A.foo(f)+A.bar(b) }
At the very least, in our purely functional version of Scala we know that the evaluation of A#foo and A#bar cannot possibly affect one-another, we are guaranteed referential transparency.
Now, drop back into *real* Scala.  Without actually knowing what are inside foo and bar we cannot be assured that evaluating foo will not alter the results of bar.  If there is referential transparency between foo and bar, to quote Adrian Moors elsewhere in this thread, it is implicit.
OK, let's switch back to the purely functional version of Scala for a moment and rewrite things like this:
object A {  def foo(String):Int = ...   def bar(String):IO[Int] = ... }
then:
object B {  def run(f:String,b:String):IO[Int] = {    val x = A.foo(f)    A.bar(b)  }}
Since we are back in purely functional land we can observe two things:

1. The evaluation of A.foo(f) cannot possibly effect the results of A.bar(b) and therefore can trivially be eliminated (optimization!) as dead code
2. The evaluation of A.bar(b) cannot possibly alter the result returned bu A.foo(f) otherwise A#foo type signature would indicate that

Unless I'm grossly mistaken we are already in a better world given the above.
Finally, let's write(still in a purely functional Scala universe):
object A {  def foo(String):IO[Int] = ...  def bar(String):IO[Int] = ...   def baz(String):Option[Int] = ...}
then:
object B {  def run(f:String,b:String,z.String):IO[Option[Int]] =     for {       a <- A.foo(f)      b <- A.bar(b)    } yield A.baz.map(_+a+b)}
We know a bunch of information merely from the types alone to help us track down a possible bug in the above program.  We know, from the types that A#baz is a pure function and we can test it's behavior with deterministic outcomes.  On the other hand A#foo and A#bar may interact in strange ways.  In the case of an error in the above program the search space to look for the bug is easily partitioned on types alone.
Extend this to a "real-world" example as necessary.
This doesn't come across as progress?  Using the IO monad can guide us as to where to look for "action at a distance" kinds of bugs.
--
Jim Powers

2011/10/10 martin odersky <martin.odersky@epfl.ch>

On Mon, Oct 10, 2011 at 12:56 PM, Jesper Nordenberg <megagurka@yahoo.com> wrote:
> martin odersky <martin.odersky <at> epfl.ch> writes:
>> So the only thing IO provides is that code *not* using it is certified
>> to be pure, whereas in Scala everything is in the IO monad.
>
> That's quite a huge thing, don't you agree?

Sure. But I think there will at some point be better ways to track
effects than monads. Monads are too
rigid. Adding a single println somewhere means you have to rewrite
your whole program.

On Mon, Oct 10, 2011 at 7:30 AM, martin odersky <martin.odersky@epfl.ch> wrote:

Sure. But I think there will at some point be better ways to track effects than monads. Monads are too
rigid. Adding a single println somewhere means you have to rewrite
your whole program.

Of course, there is the distinct possibility that this desire could be nothing more than wishful thinking.
I'm just a complete moron, but in my moronic search to find that which you desire above, I've basically come to the following conclusions:

• Most of the programming world either cannot cope with the likes of the IO monad and therefore ignore it at their peril. OR
• Some languages embrace models that work based on decades of research that bring useful tools to bear on reasoning about programs with interacting with "the real world" (IO/effects). 

Are there even any hints of research that promises a coherent model of (IO) effects outside of the Monad model?  The current work begin done by Lukas Rytz is, er, less than convincing.
--
Jim Powers

body p { margin-bottom: 0cm; margin-top: 0pt; }

Thank you for this excellent example.

All your points are valid, and very true. The question is not whether we gain something by using the IO monad. It is whether we gain enough. Since using it comes with its own price of trickier code, and mental hoops to jump through, spending brain CPU on how to deal with the monad, rather than the business problem.

Normally, one does not work with functions without knowing what they do. When I know what a function is supposed to do, I usually know what side effects it is doing, or, I don't care because they are an implementation detail. If they were not, then all I know is that something is going on under there, but can't do anything about it.

Ittay

Jim Powers wrote:

CAMjNOCNEC5ndUNSFOhe5BuqqQgJWEsGS41CESxFYLyTcOUTJ+w [at] mail [dot] gmail [dot] com" type="cite"> On Mon, Oct 10, 2011 at 3:11 AM, Ittay Dror <ittay [dot] dror [at] gmail [dot] com" rel="nofollow">ittay.dror@gmail.com> wrote:

I'd love to see a code example where the introduction of IO[_] makes more errors detectable. In the code example I used in the first message, I couldn't find where it helps.

Consider the following code outline, and assume a purely functional implementation of Scala (even if strict!)
object A {   def foo(String):Int = ...   def bar(String):Int = ... }   Then the program:
object B {   def run(f:String,b:String):Int =      A.foo(f)+A.bar(b) }
At the very least, in our purely functional version of Scala we know that the evaluation of A#foo and A#bar cannot possibly affect one-another, we are guaranteed referential transparency.
Now, drop back into *real* Scala.  Without actually knowing what are inside foo and bar we cannot be assured that evaluating foo will not alter the results of bar.  If there is referential transparency between foo and bar, to quote Adrian Moors elsewhere in this thread, it is implicit.
OK, let's switch back to the purely functional version of Scala for a moment and rewrite things like this:
object A {   def foo(String):Int = ...   def bar(String):IO[Int] = ... }
then:
object B {   def run(f:String,b:String):IO[Int] = {     val x = A.foo(f)     A.bar(b)   } }
Since we are back in purely functional land we can observe two things:

1. The evaluation of A.foo(f) cannot possibly effect the results of A.bar(b) and therefore can trivially be eliminated (optimization!) as dead code
2. The evaluation of A.bar(b) cannot possibly alter the result returned bu A.foo(f) otherwise A#foo type signature would indicate that

Unless I'm grossly mistaken we are already in a better world given the above.
Finally, let's write(still in a purely functional Scala universe):
object A {   def foo(String):IO[Int] = ...   def bar(String):IO[Int] = ...   def baz(String):Option[Int] = ... }
then:
object B {   def run(f:String,b:String,z.String):IO[Option[Int]] =      for {       a <- A.foo(f)       b <- A.bar(b)     } yield A.baz.map(_+a+b) }
We know a bunch of information merely from the types alone to help us track down a possible bug in the above program.  We know, from the types that A#baz is a pure function and we can test it's behavior with deterministic outcomes.  On the other hand A#foo and A#bar may interact in strange ways.  In the case of an error in the above program the search space to look for the bug is easily partitioned on types alone.
Extend this to a "real-world" example as necessary.
This doesn't come across as progress?  Using the IO monad can guide us as to where to look for "action at a distance" kinds of bugs.
--
Jim Powers

Thanks for the examples, but...you didn't actually do anything IO-specific.  You made more or less the same point three times, and independently of those points decorated some types with IO.

In particular, you have not given an example that shows why it is a bad idea to inject new state with IO but deal with that state functionally thereafter, or to output existing state, treated functionally prior to that point, with IO.  Just leave off the IO marker and all your points remain; you can do all the same things with unwrapped and/or Option-wrapped types (depending on whether or not the result could fail to exist).

The minimal example where the IO monad is actually better than the _most sensible alternative_ (not a mythically messy imperative situation) seems to me to require an awfully complicated setup, if it even exists at all.  (I suspect based on the logic of the arguments that it does exist, but I have yet to see an example that would not have been simpler without the IO monad specifically.)

  --Rex

On Mon, Oct 10, 2011 at 5:36 PM, Jim Powers <jim@casapowers.com> wrote:

On Mon, Oct 10, 2011 at 3:11 AM, Ittay Dror <ittay.dror@gmail.com> wrote:

I'd love to see a code example where the introduction of IO[_] makes more errors detectable. In the code example I used in the first message, I couldn't find where it helps.

Consider the following code outline, and assume a purely functional implementation of Scala (even if strict!)
object A {   def foo(String):Int = ...  def bar(String):Int = ...} Then the program:
object B {  def run(f:String,b:String):Int =     A.foo(f)+A.bar(b) }
At the very least, in our purely functional version of Scala we know that the evaluation of A#foo and A#bar cannot possibly affect one-another, we are guaranteed referential transparency.
Now, drop back into *real* Scala.  Without actually knowing what are inside foo and bar we cannot be assured that evaluating foo will not alter the results of bar.  If there is referential transparency between foo and bar, to quote Adrian Moors elsewhere in this thread, it is implicit.
OK, let's switch back to the purely functional version of Scala for a moment and rewrite things like this:
object A {  def foo(String):Int = ...   def bar(String):IO[Int] = ... }
then:
object B {  def run(f:String,b:String):IO[Int] = {    val x = A.foo(f)    A.bar(b)  }}
Since we are back in purely functional land we can observe two things:

1. The evaluation of A.foo(f) cannot possibly effect the results of A.bar(b) and therefore can trivially be eliminated (optimization!) as dead code
2. The evaluation of A.bar(b) cannot possibly alter the result returned bu A.foo(f) otherwise A#foo type signature would indicate that

Unless I'm grossly mistaken we are already in a better world given the above.
Finally, let's write(still in a purely functional Scala universe):
object A {  def foo(String):IO[Int] = ...  def bar(String):IO[Int] = ...   def baz(String):Option[Int] = ...}
then:
object B {  def run(f:String,b:String,z.String):IO[Option[Int]] =     for {       a <- A.foo(f)      b <- A.bar(b)    } yield A.baz.map(_+a+b)}
We know a bunch of information merely from the types alone to help us track down a possible bug in the above program.  We know, from the types that A#baz is a pure function and we can test it's behavior with deterministic outcomes.  On the other hand A#foo and A#bar may interact in strange ways.  In the case of an error in the above program the search space to look for the bug is easily partitioned on types alone.
Extend this to a "real-world" example as necessary.
This doesn't come across as progress?  Using the IO monad can guide us as to where to look for "action at a distance" kinds of bugs.
--
Jim Powers

On Tue, Oct 11, 2011 at 12:05 AM, Ittay Dror <ittay.dror@gmail.com> wrote:

Thank you for this excellent example.

All your points are valid, and very true. The question is not whether we gain something by using the IO monad. It is whether we gain enough. Since using it comes with its own price of trickier code, and mental hoops to jump through, spending brain CPU on how to deal with the monad, rather than the business problem.

Generally I would say yes it's worth it.  The issue of "trickier" and "using more CPU cycles" is really mostly a learning issue.  After a while thinking in these terms doesn't prove a challenge. 

Normally, one does not work with functions without knowing what they do. When I know what a function is supposed to do, I usually know what side effects it is doing, or, I don't care because they are an implementation detail. If they were not, then all I know is that something is going on under there, but can't do anything about it.

Again I'll point out the fact that this knowledge you posses is implicit to the computer.  Making more of the knowledge in your head explicit to the computer only furthers its ability to assist you in keeping your thoughts clear.  At the very least this kind of typing is documentation, and at the very best it helps you from running into trouble.  There really is no circumstance where this kind of thinking leads to bad outcomes.  -- Jim Powers

Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"

We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.

In practice, once you have factored out this concern, almost all of your code is pure. I.e. You don't see "IO" in very many places. It's not at all true that most code has to have side-effects.

The argument I'm hearing is that IO is somehow special. A "cross-cutting concern". Bullshit. It's just another DSL.

I'm kind of done with "oh, but don't you want to convince me?!" No, I really don't. The best you can do is try it both ways, know what you're talking about first hand, and then decide for yourself. There's no magic in monads. It's just separation of concerns and combination of components according to a simple interface.

On Tue, Oct 11, 2011 at 12:42 AM, Rex Kerr <ichoran@gmail.com> wrote:

Thanks for the examples, but...you didn't actually do anything IO-specific.

Actually I did ( :-) )!  What particular IO actions were intended by my functions wasn't relevant to the discussion. 

You made more or less the same point three times, and independently of those points decorated some types with IO.

Well, no, not really. Another try - here's a variant of the first example (recall, we have an imaginary purely functional version of Scala):
object A {   def foo(String):Option[Int] = ...  def bar(String):Option[Int] = ... }  Then the program:
object B {  def run(f:String,b:String):Int =      for {      a <- A.foo(f)       b <- A.bar(b)    } yield a+b }
Is the exact same program as:
object B {   def run(f:String,b:String):Int =      for {      b <- A.bar(b)       a <- A.foo(f)    } yield a+b }
Because in the case of a purely functional language I know that I can always commute evaluation order.  Further, the computer knows this too! Substitute in the IO monad for the Option and the guarantees about commutativity go out the window. Granted that using functions in an IO context like above where I didn't take steps to eliminate the possibility of a bug by forcing the correct order (say through the use of a Reader or State monad) is possible.  This is to say that while I can substitute IO for Option above the guarantees of correctness do not come along for the ride. (And, as I've said there are clean ways to enforce a "correct" evaluation order of side-effecting code.  Overall, once these approaches are learned, your "cognitive load" actually goes down not up).

In particular, you have not given an example that shows why it is a bad idea to inject new state with IO but deal with that state functionally thereafter, or to output existing state, treated functionally prior to that point, with IO.  Just leave off the IO marker and all your points remain; you can do all the same things with unwrapped and/or Option-wrapped types (depending on whether or not the result could fail to exist).

No, this claim is false as indicated by the above example: in general, IO cannot be substituted for other Monads and maintain the same correctness guarantees.

The minimal example where the IO monad is actually better than the _most sensible alternative_ (not a mythically messy imperative situation) seems to me to require an awfully complicated setup, if it even exists at all.  (I suspect based on the logic of the arguments that it does exist, but I have yet to see an example that would not have been simpler without the IO monad specifically.)

I'm not aware of a "sensible alternative", I'd be happy to hear of one.  Martin is claiming such a thing exists, it would be nice if is work was fruitful.  I would suspect that if he is he would be ushering in an whole new branch of mathematics :-)  -- Jim Powers

I want to know more about these other, more polymorphic ways. Do you have some papers handy that I could read?

from http://www.cse.unsw.edu.au/~benl/:

• Type Inference and Optimisation for an Impure World, ANU 2010 (PhD thesis) 
  
  
• Witnessing Purity, Constancy and Mutability, (slides), APLAS 2009 

On Tue, Oct 11, 2011 at 2:52 PM, Runar Bjarnason <runarorama@gmail.com> wrote:

I want to know more about these other, more polymorphic ways. Do you have some papers handy that I could read?

On Tue, Oct 11, 2011 at 2:52 PM, Runar Bjarnason wrote:
> I want to know more about these other, more polymorphic ways. Do you have some papers handy that I could read?
>

Start with "The Marriage of effects and monads". Then, add
polymorphism to the effect system. This is still too complicated for
my taste, so we have to find simpler solutions. It's research, which
comes before published papers.

>
> I'm not aware of a "sensible alternative", I'd be happy to hear of one.
>  Martin is claiming such a thing exists, it would be nice if is work was
> fruitful.  I would suspect that if he is he would be ushering in an whole
> new branch of mathematics :-)
>
I don't recall claiminig that it existed. I said I was looking for it.
It's called research.

body p { margin-bottom: 0cm; margin-top: 0pt; }

Runar Bjarnason wrote:

Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"

We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.

Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

In practice, once you have factored out this concern, almost all of your code is pure. I.e. You don't see "IO" in very many places. It's not at all true that most code has to have side-effects.

The argument I'm hearing is that IO is somehow special. A "cross-cutting concern". Bullshit. It's just another DSL.

I'm kind of done with "oh, but don't you want to convince me?!" No, I really don't. The best you can do is try it both ways, know what you're talking about first hand, and then decide for yourself. There's no magic in monads. It's just separation of concerns and combination of components according to a simple interface.

On Tue, Oct 11, 2011 at 9:55 AM, Ittay Dror <ittay.dror@gmail.com> wrote:

Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

Right there.

In practise, you compose effectful things in two ways:

for {
  x <- effectsHere
  y <- effectsHereToo(x)
} yield noEffectsHere(y)

or...

(effectsHere |@| effectsHereToo)((a, b) => noEffectsHere(a, b))

You'll find that most of your code is in "noEffectsHere". It doesn't need to be aware that it can be lifted into IO. This is what separation of concerns is all about.

On Oct 11, 2011 7:56 AM, "Ittay Dror" <ittay.dror@gmail.com> wrote:
>
>
>
> Runar Bjarnason wrote:
>>
>> Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"
>>
>> We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.
>
> Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

As someone who hasn't gotten much further then LYAH, I'll see if I can explain (as much for my benefit as anything else):

Just like the fact that you can use the value stored within an Option using map or flatMap, the value within IO can be passed to pure functions as well.

If you consider the previous example of adding 2 numbers that are retrieved by some IO, you do not need '+' to be aware of the source of these numbers. Ideally you would keep the IO type as close to your 'main' method as possible, using pure functions to actually do something with the results of the IO.

This results in a program where you naturally tend to separate your IO from the functions that process that data, hopefully leaving the majority of your program IO free.

--
Derek

On Oct 11, 2011 7:56 AM, "Ittay Dror" <ittay.dror@gmail.com> wrote:

Runar Bjarnason wrote:

Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"

We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.

Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

In practice, once you have factored out this concern, almost all of your code is pure. I.e. You don't see "IO" in very many places. It's not at all true that most code has to have side-effects.

The argument I'm hearing is that IO is somehow special. A "cross-cutting concern". Bullshit. It's just another DSL.

I'm kind of done with "oh, but don't you want to convince me?!" No, I really don't. The best you can do is try it both ways, know what you're talking about first hand, and then decide for yourself. There's no magic in monads. It's just separation of concerns and combination of components according to a simple interface.

body p { margin-bottom: 0cm; margin-top: 0pt; }

Runar Bjarnason wrote:

CABjJA71pMpfGNXGxthhoyy4N0gsPQioc_030rE5zUjRZ8cSm-Q [at] mail [dot] gmail [dot] com" type="cite">

On Tue, Oct 11, 2011 at 9:55 AM, Ittay Dror <ittay [dot] dror [at] gmail [dot] com" rel="nofollow">ittay.dror@gmail.com> wrote:

Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

Right there.

In practise, you compose effectful things in two ways:

for {
  x <- effectsHere
  y <- effectsHereToo(x)
} yield noEffectsHere(y)

Lets take a more concrete example. I have a function fetchUser: String => IO[User]. What you're suggesting is that this method needs to be used at 'main', to get the User instance and pass it to the rest of my application ('noEffectsHere'), right? So I do all my IO upfront, and then call the rest of my code? It would be hard for me to arrange my code to work like this.

CABjJA71pMpfGNXGxthhoyy4N0gsPQioc_030rE5zUjRZ8cSm-Q [at] mail [dot] gmail [dot] com" type="cite">
or...

(effectsHere |@| effectsHereToo)((a, b) => noEffectsHere(a, b))

You'll find that most of your code is in "noEffectsHere". It doesn't need to be aware that it can be lifted into IO. This is what separation of concerns is all about.

On Tuesday, October 11, 2011 10:20:42 AM UTC-4, Ittay Dror wrote:

Lets take a more concrete example. I have a function fetchUser: String => IO[User]. What you're suggesting is that this method needs to be used at 'main', to get the User instance and pass it to the rest of my application ('noEffectsHere'), right? So I do all my IO upfront, and then call the rest of my code?

No, that's not being suggested at all, and I'm not sure what gives you that idea. Consider:

fetchUser(s) map f

Both s and f can be pure and neither one of them needs to know anything at all about IO. The vast majority of code in real applications falls in this category. In fact, I'll bet that almost none of the code invoked by "fetchUser" has anything at all to do with IO either.

i don't think there is a clean solution.
you'll pollute the hierarchy because of that single println deep down there or you have to make a dirty cut at some point.

doing all sideeffect-stuff before or after all pure stuff is not always possible, and most likely very ugly for complex operations. i tend to just ignore side effects that "shouldn't matter much"

> -------- Original-Nachricht --------
> Datum: Tue, 11 Oct 2011 16:20:42 +0200
> Von: Ittay Dror
> An: Runar Bjarnason
> CC: scala-debate@googlegroups.com
> Betreff: Re: [scala-debate] Re: questioning FP
> 
> Runar Bjarnason wrote:
> >
> >
> > On Tue, Oct 11, 2011 at 9:55 AM, Ittay Dror
>
> > wrote:
> >
> > > Interesting. Maybe I missed something. I thought
> > > that when you use IO you need to propagate it up to 'main'
> > > (or 'run'). So one function that does IO (a println)
> > > pollutes your whole program. Where am I wrong?
> > >
> >
> > Right there.
> >
> > In practise, you compose effectful things in two ways:
> >
> > for {
> >   x <- effectsHere
> >   y <- effectsHereToo(x)
> > } yield noEffectsHere(y)
> >
>
> Lets take a more concrete example. I have a function fetchUser:
> String => IO[User]. What you're suggesting is that this method needs to
> be used at 'main', to get the User instance and pass it to the rest of
> my application ('noEffectsHere'), right? So I do all my IO upfront, and
> then call the rest of my code? It would be hard for me to arrange my
> code to work like this.
>
>
> >
> > or...
> >
> > (effectsHere |@| effectsHereToo)((a, b) => noEffectsHere(a, b))
> >
> > You'll find that most of your code is in "noEffectsHere". It
> > doesn't need to be aware that it can be lifted into IO. This is
> what
> > separation of concerns is all about.
> >
>

body p { margin-bottom: 0cm; margin-top: 0pt; }

Runar Bjarnason wrote:

On Tuesday, October 11, 2011 10:20:42 AM UTC-4, Ittay Dror wrote:

Lets take a more concrete example. I have a function fetchUser: String => IO[User]. What you're suggesting is that this method needs to be used at 'main', to get the User instance and pass it to the rest of my application ('noEffectsHere'), right? So I do all my IO upfront, and then call the rest of my code?

No, that's not being suggested at all, and I'm not sure what gives you that idea. Consider:

fetchUser(s) map f

Both s and f can be pure and neither one of them needs to know anything at all about IO. The vast majority of code in real applications falls in this category. In fact, I'll bet that almost none of the code invoked by "fetchUser" has anything at all to do with IO either.

Sure, then fetchUser is mostly non-IO, but needs to return IO[User]. Similarly, the functions that use fetchUser will also need to return IO[Something]. So a small implementation detail in a function can change its signature and forces it to be used differently (inside a for comprehension).

body p { margin-bottom: 0cm; margin-top: 0pt; }

Derek Williams wrote:

CAOBz8JaUix2d7eDapSGSnVG4Y-faTdddq8E8Xt5bcUAbSkBRQw [at] mail [dot] gmail [dot] com" type="cite">

On Oct 11, 2011 7:56 AM, "Ittay Dror" <ittay [dot] dror [at] gmail [dot] com" rel="nofollow">ittay.dror@gmail.com> wrote:
>
>
>
> Runar Bjarnason wrote:
>>
>> Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"
>>
>> We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.
>
> Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

As someone who hasn't gotten much further then LYAH, I'll see if I can explain (as much for my benefit as anything else):

Just like the fact that you can use the value stored within an Option using map or flatMap, the value within IO can be passed to pure functions as well.

The difference is that for Option I have a catamorphism. So I can always "get out" of using Option. With IO, once I use it, for no matter how small an issue, I need to use it in all the call hierarchy and change everything to work through for comprehension.

CAOBz8JaUix2d7eDapSGSnVG4Y-faTdddq8E8Xt5bcUAbSkBRQw [at] mail [dot] gmail [dot] com" type="cite">

If you consider the previous example of adding 2 numbers that are retrieved by some IO, you do not need '+' to be aware of the source of these numbers. Ideally you would keep the IO type as close to your 'main' method as possible, using pure functions to actually do something with the results of the IO.

This results in a program where you naturally tend to separate your IO from the functions that process that data, hopefully leaving the majority of your program IO free.

A sane developer will do that anyway (a bad developer will muck any codebase).

Returning to my original question, using Options for example (and monads in general) is easy to explain: you have an alternative, more semantically true, to returning null, so your code becomes safer (a function that returns nothing is clearly documented with the type), and remains clean since the use of Option is usually local (at some point you have a default value, and use getOrElse to reduce the Option to a value). With IO, it is trickier to explain:

 First, because IO is not necessarily dangerous (unlike trying to invoke a method on null).

 Second, because the IO monad does not prevent me from doing a dangerous action, just wraps it (if I try to read a non existent file, my app will fail, with or without IO[_]).

Third, it is pervasive (no catamorphism)

So the question is, what clear gain can I show?

CAOBz8JaUix2d7eDapSGSnVG4Y-faTdddq8E8Xt5bcUAbSkBRQw [at] mail [dot] gmail [dot] com" type="cite">

--
Derek

On Oct 11, 2011 7:56 AM, "Ittay Dror" <ittay [dot] dror [at] gmail [dot] com" rel="nofollow">ittay.dror@gmail.com> wrote:

Runar Bjarnason wrote:

Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"

We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.

Interesting. Maybe I missed something. I thought that when you use IO you need to propagate it up to 'main' (or 'run'). So one function that does IO (a println) pollutes your whole program. Where am I wrong?

In practice, once you have factored out this concern, almost all of your code is pure. I.e. You don't see "IO" in very many places. It's not at all true that most code has to have side-effects.

The argument I'm hearing is that IO is somehow special. A "cross-cutting concern". Bullshit. It's just another DSL.

I'm kind of done with "oh, but don't you want to convince me?!" No, I really don't. The best you can do is try it both ways, know what you're talking about first hand, and then decide for yourself. There's no magic in monads. It's just separation of concerns and combination of components according to a simple interface.

On Monday, October 10, 2011 3:25:11 PM UTC-4, martin odersky wrote:

You can just as well
compose closures in a language without the IO monad. What's specific
about IO is that it marks the underlying action as having a particular
kind of effect.

Note that closure composition is monadic too. It's perfectly reasonable to say:

type IO[A] = Function0[A]

I'm on board with better typing though. Uniqueness types or "effect tagging" provide more granular types for effects. But don't seek comfort in that these are not monads, because they are. They can all be described as monadic region calculi, or delimited continuations, i.e. indexed monads. And it would be bad if we had these without the ability to exploit their monadicity in code.

On Tuesday, October 11, 2011 11:26:43 AM UTC-4, Ittay Dror wrote:

Sure, then fetchUser is mostly non-IO, but needs to return IO[User]. Similarly, the functions that use fetchUser will also need to return IO[Something]. So a small implementation detail in a function can change its signature and forces it to be used differently (inside a for comprehension).

It sounds like your apprehension is entirely syntactic. And the fact that a function requires IO is no small implementation detail. It's a very real dependency, a cause of nondeterminism, and a barrier to composition. But go ahead and use null instead of Option, exceptions instead of Either, mutable iterators instead of lists, and identity instead of IO. It's super dynamic and pragmatic.

On Tue, Oct 11, 2011 at 6:02 PM, Runar Bjarnason wrote:
>
>
> On Tuesday, October 11, 2011 11:26:43 AM UTC-4, Ittay Dror wrote:
>>
>> Sure, then fetchUser is mostly non-IO, but needs to return IO[User].
>> Similarly, the functions that use fetchUser will also need to return
>> IO[Something]. So a small implementation detail in a function can change its
>> signature and forces it to be used differently (inside a for comprehension).
>>
>
> It sounds like your apprehension is entirely syntactic. And the fact that a
> function requires IO is no small implementation detail. It's a very real
> dependency, a cause of nondeterminism, and a barrier to composition. But go
> ahead and use null instead of Option, exceptions instead of Either, mutable
> iterators instead of lists, and identity instead of IO. It's super dynamic
> and pragmatic.
>
Let's try to keep the sarcasm down on that list please.

body p { margin-bottom: 0cm; margin-top: 0pt; }

martin odersky wrote:

W6Vrp7rSRD8+CYP5w_XvuDALPA [at] mail [dot] gmail [dot] com" type="cite">

On Tue, Oct 11, 2011 at 6:02 PM, Runar Bjarnason  wrote:

On Tuesday, October 11, 2011 11:26:43 AM UTC-4, Ittay Dror wrote:

Sure, then fetchUser is mostly non-IO, but needs to return IO[User].
Similarly, the functions that use fetchUser will also need to return
IO[Something]. So a small implementation detail in a function can change its
signature and forces it to be used differently (inside a for comprehension).

It sounds like your apprehension is entirely syntactic. And the fact that a
function requires IO is no small implementation detail. It's a very real
dependency, a cause of nondeterminism, and a barrier to composition. But go
ahead and use null instead of Option, exceptions instead of Either, mutable
iterators instead of lists, and identity instead of IO. It's super dynamic
and pragmatic.

Let's try to keep the sarcasm down on that list please.

For the record, I wasn't offended by Runar's reply. On the contrary, I'm glad he keeps on replying to my answers.

W6Vrp7rSRD8+CYP5w_XvuDALPA [at] mail [dot] gmail [dot] com" type="cite">

body p { margin-bottom: 0cm; margin-top: 0pt; }

Runar Bjarnason wrote:

On Tuesday, October 11, 2011 11:26:43 AM UTC-4, Ittay Dror wrote:

Sure, then fetchUser is mostly non-IO, but needs to return IO[User]. Similarly, the functions that use fetchUser will also need to return IO[Something]. So a small implementation detail in a function can change its signature and forces it to be used differently (inside a for comprehension).

It sounds like your apprehension is entirely syntactic. And the fact that a function requires IO is no small implementation detail. It's a very real dependency, a cause of nondeterminism, and a barrier to composition. But go ahead and use null instead of Option, exceptions instead of Either, mutable iterators instead of lists, and identity instead of IO. It's super dynamic and pragmatic.

My points:
1. With Option and Either I have catamorphism, so using them in a function does not necessarily escape the function. And they avoid the alternative (meaning, if I use Option, i will never get an NPE)
2. Using the IO monad does not prevent non-determinism. The actions will run just the same (meaning if I use IO[_] i will still get FileNotFoundException). It only adds a tagging
3. The tagging is cumbersome to use.

On Tue, Oct 11, 2011 at 11:36 AM, Ittay Dror <ittay.dror@gmail.com> wrote:

 Second, because the IO monad does not prevent me from doing a dangerous action, just wraps it (if I try to read a non existent file, my app will fail, with or without IO[_]).

And if I take the head of a List, my app will fail too. If I divide an Int by zero, my app will fail. This doesn't mean List and Int are useless types.

That having been said, I am working on getting regional IO into Scalaz's IO monad. This will guarantee, in the type system, that you can never read, write, or close a file that isn't open. Having kind polymorphism in Scala would make this much easier than it is (Martin, Adriaan, et al, take note).

On Tue, Oct 11, 2011 at 8:36 AM, Jim Powers <jim@casapowers.com> wrote:

object B {  def run(f:String,b:String):Int =      for {      a <- A.foo(f)       b <- A.bar(b)    } yield a+b }
Is the exact same program as:
object B {   def run(f:String,b:String):Int =      for {      b <- A.bar(b)       a <- A.foo(f)    } yield a+b }
Because in the case of a purely functional language I know that I can always commute evaluation order.  Further, the computer knows this too! Substitute in the IO monad for the Option and the guarantees about commutativity go out the window.

Granted, but your earlier code didn't do anything to guarantee order, so it wasn't of any use except as a tag.  And it's not a very useful tag because the tag is at the _declaration site_ not the _use site_.  At least if you had to call something like Console.readLine it would be clear where the action was.  If you mark inputs as being IO[String] and then unwrap them somewhere buried in the code, you don't have to know which sort of thing you're unwrapping (maybe it was an Option or some other sort of wrapper).
 

Granted that using functions in an IO context like above where I didn't take steps to eliminate the possibility of a bug by forcing the correct order (say through the use of a Reader or State monad) is possible.

Indeed.  But if one does things this way, it is a huge hassle (of the major-rewrite-to-add-a-println variety).  For specialized cases, encoding a finite state machine into your types (for instance) can be exactly what you need to keep things straight.  But from what I've seen, this is a niche case when it comes to IO.
 

The minimal example where the IO monad is actually better than the _most sensible alternative_ (not a mythically messy imperative situation) seems to me to require an awfully complicated setup, if it even exists at all.  (I suspect based on the logic of the arguments that it does exist, but I have yet to see an example that would not have been simpler without the IO monad specifically.)

I'm not aware of a "sensible alternative", I'd be happy to hear of one.

The sensible alternative is untagged side-effectful code that immediately moves data into an appropriate typesafe and perhaps immutable form.

  --Rex

body p { margin-bottom: 0cm; margin-top: 0pt; }

Runar Bjarnason wrote:

MujFQyy-dpqChNv-pXw4sAiAzA [at] mail [dot] gmail [dot] com" type="cite">

On Tue, Oct 11, 2011 at 11:36 AM, Ittay Dror <ittay [dot] dror [at] gmail [dot] com" rel="nofollow">ittay.dror@gmail.com> wrote:

 Second, because the IO monad does not prevent me from doing a dangerous action, just wraps it (if I try to read a non existent file, my app will fail, with or without IO[_]).

And if I take the head of a List, my app will fail too. If I divide an Int by zero, my app will fail. This doesn't mean List and Int are useless types.

I thought that similar to how Option guards us from nulls, IO guards us from side-effects (Int is not supposed to guard us from division.). If it doesn't, then again, what is the point? Just to tag a function as doing IO? And then tag all its callers? And make their code a bit awkward (for comprehension)? Isn't this like checked exceptions?

MujFQyy-dpqChNv-pXw4sAiAzA [at] mail [dot] gmail [dot] com" type="cite">
That having been said, I am working on getting regional IO into Scalaz's IO monad. This will guarantee, in the type system, that you can never read, write, or close a file that isn't open. Having kind polymorphism in Scala would make this much easier than it is (Martin, Adriaan, et al, take note).

cool! This is something which has tangible benefits.

On Tue, Oct 11, 2011 at 12:23 PM, Ittay Dror <ittay.dror@gmail.com> wrote:

I thought that similar to how Option guards us from nulls, IO guards us from side-effects (Int is not supposed to guard us from division.). If it doesn't, then again, what is the point? Just to tag a function as doing IO? And then tag all its callers? And make their code a bit awkward (for comprehension)? Isn't this like checked exceptions?

I don't think it's useful to think of it as a tag. It is a data type. Again, a value of type IO[A] is a program, written (supposedly) in a DSL that talks about reading and writing some opaque IO subsystem. In practise, the instruction set of this DSL is very small indeed (open file, close file, read file, write file, and maybe a handful more). But you're right in that a value of this type can generally not be converted to anything else. This is why IO should form only the outermost shell of your program, a very thin layer that talks to the outside world. I think you'll want to do that whether or not you choose to take advantage of monadic composition.

All I can say is "try it". I don't think you will run into all the problems you think you might, and it's not as awkward as all that (certainly not as awkward as, say, trying to retrofit to pure code a function that assumes the database is configured and open and tries to read from it). You don't ever need to "get out" of IO. You can always take a function of type A => B and turn it into a function of type IO[A] => IO[B].

Sorry, this was meant for the list

On Tue, Oct 11, 2011 at 11:36 AM, Ittay Dror <ittay.dror@gmail.com> wrote:

As someone who hasn't gotten much further then LYAH, I'll see if I can explain (as much for my benefit as anything else):

Just like the fact that you can use the value stored within an Option using map or flatMap, the value within IO can be passed to pure functions as well.

The difference is that for Option I have a catamorphism. So I can always "get out" of using Option. With IO, once I use it, for no matter how small an issue, I need to use it in all the call hierarchy and change everything to work through for comprehension.

You can get out, it's called running the program and it yields () - your program is just one big catamorphism over "actions" (how these actions are evaluated is entirely up to the implementation of IO that you use). :-D
You don't need for comprehensions you can map/flatMap directly (you can use >>= [and friends] from Scalaz ;-) ).

If you consider the previous example of adding 2 numbers that are retrieved by some IO, you do not need '+' to be aware of the source of these numbers. Ideally you would keep the IO type as close to your 'main' method as possible, using pure functions to actually do something with the results of the IO.

This results in a program where you naturally tend to separate your IO from the functions that process that data, hopefully leaving the majority of your program IO free.

A sane developer will do that anyway (a bad developer will muck any codebase).

Having something like IO (or monads in general) helps you to stay sane in an insane world where "bad" developers will muckity-muck your code.  Again, it's really just being clear to the compiler what your intentions are; no harm can come from that.  In fact, it helps your bad developers learn how to better organize their code so they become better developers.  This is not merely discipline, it's computer-assisted proof development.
In the end your (our) profession is really only going to become more productive and, er, effective if we work hand-in-hand with software that can better reason about our intentions.  Today that means FP for clarity and concision and types to help with correctness and documentation.  Tomorrow this will mean (just guessing here) dependent types and theorem-proving systems (take a look at ATS: http://www.ats-lang.org/, or Agda: http://wiki.portal.chalmers.se/agda/pmwiki.php for some examples).  Either way there are really only upsides to better (machine enforced) clarity.  This will come at a cost of having to be explicit more and implicit less.

Returning to my original question, using Options for example (and monads in general) is easy to explain: you have an alternative, more semantically true, to returning null, so your code becomes safer (a function that returns nothing is clearly documented with the type), and remains clean since the use of Option is usually local (at some point you have a default value, and use getOrElse to reduce the Option to a value). With IO, it is trickier to explain:

 First, because IO is not necessarily dangerous (unlike trying to invoke a method on null).

Well even "pure" functions can be dangerous: division by zero, non-termination, etc.  With an improved ability to express intent via the type (and proof) system (including monadic types like IO) we gain confidence in the correctness of our programs.  At the very least enforced partitions are created to help isolate defects.

 Second, because the IO monad does not prevent me from doing a dangerous action, just wraps it (if I try to read a non existent file, my app will fail, with or without IO[_]).

And purity alone can't guarantee termination, avoid of resource exhaustion (stack/heap), not stepping outside of array bounds, eliminating division by zero, treating partial functions as total, etc. One of the hallmarks of FP is a tendency towards concision, and often that concision helps to identify/avoid some of the aforementioned problems.  Types like IO may not prevent mistakes associated with side-effecting code, but it does isolate and label such side-effecting code.

Third, it is pervasive (no catamorphism)

So the question is, what clear gain can I show?

That clarity of intent is beneficial. That isolating components into composable parts, including side-effecting code, encourages better comprehension and reuse of code.    -- 
Jim Powers

Cool - It is the first time someone points out that this is like using null or exceptions… BUT, isn’t the point that this kind of IO[] wrapping creates the same kind of headaches that a checked exception does?

 

From: scala-debate@googlegroups.com [mailto:scala-debate@googlegroups.com] On Behalf Of Ittay Dror
Sent: October-11-11 12:10 PM
To: martin odersky
Cc: scala-debate@googlegroups.com; Runar Bjarnason
Subject: Re: [scala-debate] Re: questioning FP

 

 

martin odersky wrote:

On Tue, Oct 11, 2011 at 6:02 PM, Runar Bjarnason  wrote:

On Tuesday, October 11, 2011 11:26:43 AM UTC-4, Ittay Dror wrote:

Sure, then fetchUser is mostly non-IO, but needs to return IO[User].

Similarly, the functions that use fetchUser will also need to return

IO[Something]. So a small implementation detail in a function can change its

signature and forces it to be used differently (inside a for comprehension).

It sounds like your apprehension is entirely syntactic. And the fact that a

function requires IO is no small implementation detail. It's a very real

dependency, a cause of nondeterminism, and a barrier to composition. But go

ahead and use null instead of Option, exceptions instead of Either, mutable

iterators instead of lists, and identity instead of IO. It's super dynamic

and pragmatic.

Let's try to keep the sarcasm down on that list please.

For the record, I wasn't offended by Runar's reply. On the contrary, I'm glad he keeps on replying to my answers.

 -- Martin

On Tue, Oct 11, 2011 at 9:44 PM, Razvan Cojocaru wrote:
> Cool - It is the first time someone points out that this is like using null
> or exceptions… BUT, isn’t the point that this kind of IO[] wrapping creates
> the same kind of headaches that a checked exception does?

Very good observation. That's why you want polymorphism.

You just took the magic out of monads for me.  What have I left to look forward to?  The co-magic of co-monads?

On Tue, Oct 11, 2011 at 8:33 AM, Runar Bjarnason <runarorama@gmail.com> wrote:

Well, that's true with anything, that you have to decide for yourself if it's worth the trouble to understand. Before you can say "this is beneficial" you have to ask "beneficial for whom and for what purpose?"

We are using an IO-like monad to isolate database effects where I work. We're also using a separate State-like monad to compose computations that have state (instead of using mutable state). We find this to be a huge benefit. Especially when we can re-use the same code for both things that seemingly have nothing to do with each other. The code that is IO-specific is very short and confined to one or two files.

In practice, once you have factored out this concern, almost all of your code is pure. I.e. You don't see "IO" in very many places. It's not at all true that most code has to have side-effects.

The argument I'm hearing is that IO is somehow special. A "cross-cutting concern". Bullshit. It's just another DSL.

I'm kind of done with "oh, but don't you want to convince me?!" No, I really don't. The best you can do is try it both ways, know what you're talking about first hand, and then decide for yourself. There's no magic in monads. It's just separation of concerns and combination of components according to a simple interface.

martin odersky skrev 2011-10-11 22:42:
> Very good observation. That's why you want polymorphism.

Can you give an example of the polymorphism you want that the IO monad
cannot provide?

/Jesper Nordenberg

On Tue, Oct 11, 2011 at 11:32 PM, Jesper Nordenberg wrote:
> martin odersky skrev 2011-10-11 22:42:
>>
>> Very good observation. That's why you want polymorphism.
>
> Can you give an example of the polymorphism you want that the IO monad
> cannot provide?
>
xs map f

should be effectful if f is.

On Tue, Oct 11, 2011 at 5:38 PM, martin odersky <martin.odersky@epfl.ch> wrote:

  xs map f

should be effectful if f is.

Maybe "map" is not the abstraction you're looking for, but "traverse":

http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.2/doc.sxr/scalaz/Traverse.scala.html

martin odersky skrev 2011-10-11 23:38:
> On Tue, Oct 11, 2011 at 11:32 PM, Jesper Nordenberg wrote:
>> martin odersky skrev 2011-10-11 22:42:
>>>
>>> Very good observation. That's why you want polymorphism.
>>
>> Can you give an example of the polymorphism you want that the IO monad
>> cannot provide?
>>
> xs map f
>
> should be effectful if f is.

I'm not sure I agree with that. If f doesn't have side effects the map
operation can for example be performed in parallel. If f has side
effects the map operation must be performed sequentially.

So, the lack of side effects gives you less implementation restrictions.

/Jesper Nordenberg

On Tue, Oct 11, 2011 at 11:59 PM, Jesper Nordenberg wrote:
> martin odersky skrev 2011-10-11 23:38:
>>
>> On Tue, Oct 11, 2011 at 11:32 PM, Jesper Nordenberg
>>  wrote:
>>>
>>> martin odersky skrev 2011-10-11 22:42:
>>>>
>>>> Very good observation. That's why you want polymorphism.
>>>
>>> Can you give an example of the polymorphism you want that the IO monad
>>> cannot provide?
>>>
>>    xs map f
>>
>> should be effectful if f is.
>
> I'm not sure I agree with that. If f doesn't have side effects the map
> operation can for example be performed in parallel. If f has side effects
> the map operation must be performed sequentially.
>
> So, the lack of side effects gives you less implementation restrictions.
>

I believe on parallel collections map might require its argument to be
side-effect free. But the only practical solution I see for the
foreseeable future would be to do that as part of an optional type
system. On sequential collections map should certainly be polymorphic.

Cheers

 

   xs map f

should be effectful if f is.

I'm not sure I agree with that.

if `f` prints to the console (or throws an exception, or formats your drive, or all of the above), so will `map`
thus, `map f` should be considered as having the same effect as `f` (assuming a "traditional" implementation of `map`, which actually executes `f`)
it seems to me it would be unsound not to have this
you seem to be saying map should be ad-hoc polymorphic in terms of the purity of f (in the sense of having overloads for different effects). I think that's reasonable, but it does not make the above untrue.

On Tue, Oct 11, 2011 at 11:52 PM, Runar Bjarnason wrote:
>
>
> On Tue, Oct 11, 2011 at 5:38 PM, martin odersky
> wrote:
>>
>>   xs map f
>>
>> should be effectful if f is.
>>
>
> Maybe "map" is not the abstraction you're looking for, but "traverse":

I definitively want a one-parameter function like map. Not sure what
it would take to make traverse into that. What I see so far looks
complicated, but I can see behind that if one presents the use cases
well.

Cheers

On Tue, Oct 11, 2011 at 6:13 PM, Adriaan Moors <adriaan.moors@epfl.ch> wrote:

 

   xs map f

should be effectful if f is.

I'm not sure I agree with that.

if `f` prints to the console (or throws an exception, or formats your drive, or all of the above), so will `map`
thus, `map f` should be considered as having the same effect as `f` (assuming a "traditional" implementation of `map`, which actually executes `f`)

Interesting, consider this:
object ReverseEffects {  val xs = List((d:String) => { println("Format hard drive: %s".format(d)); d.length },                  (_:String) => { println("throws an exception"); 0},                 (_:String) => { println("all of the above"); 0 })  val f:(String => Int) => Int = (a:String => Int) => a("sda1")   def doMap { xs map f }}
How would something like this be thought of?  I would presume that the polymorphic effect characteristic of 'xs map f' would not be limited to the 'same side effects as `f`' as `f`, by itself, doesn't do anything with regard to side-effects.  Instead the effect would be derived from the entire expression, yes?

it seems to me it would be unsound not to have this
you seem to be saying map should be ad-hoc polymorphic in terms of the purity of f (in the sense of having overloads for different effects). I think that's reasonable, but it does not make the above untrue.

--
Jim Powers

Of course it's more pragmatic (until Tony tells me I'm using "pragmatic" incorrectly)...
If the compiler catches the bug, I have to fix it before I show the customer and get paid...If the bug only manifests itself in some low probability runtime condition, I may have already been paid and it's either someone else's problem or an opportunity to be paid again...
Clearly dynamic is very pragmatic. :-)
On Tuesday, October 11, 2011 12:02:59 PM UTC-4, Runar Bjarnason wrote:

On Tuesday, October 11, 2011 11:26:43 AM UTC-4, Ittay Dror wrote:

Sure, then fetchUser is mostly non-IO, but needs to return IO[User]. Similarly, the functions that use fetchUser will also need to return IO[Something]. So a small implementation detail in a function can change its signature and forces it to be used differently (inside a for comprehension).

It sounds like your apprehension is entirely syntactic. And the fact that a function requires IO is no small implementation detail. It's a very real dependency, a cause of nondeterminism, and a barrier to composition. But go ahead and use null instead of Option, exceptions instead of Either, mutable iterators instead of lists, and identity instead of IO. It's super dynamic and pragmatic.

 
On Tuesday, October 11, 2011 6:18:55 PM UTC-4, martin odersky wrote:

> Maybe "map" is not the abstraction you're looking for, but "traverse":

I definitively want a one-parameter function like map. Not sure what
it would take to make traverse into that. What I see so far looks
complicated, but I can see behind that if one presents the use cases
well.

 Not complicated at all. The pattern is presented in some detail in the paper "The Essence of the Iterator Pattern" by Gibbons and Oliveira:
http://www.comlab.ox.ac.uk/jeremy.gibbons/publications/iterator.pdf

The basic idea is that data of some type F[A] can be traversed with a function of type A => M[B] to produce M[F[B]], where M represents some effect, and supports certain operations.

We have implemented this in Scalaz. For example, here is a use case where the "effect" is none at all. This is just "map":

scala> List(1,2,3).traverse[Id, Int](_ * 2)
res0: List[Int] = List(2, 4, 6)

The "Id" type here is just "type Id[A] = A".

Here is one where the effect is printing to the console:

scala> Stream("a", "b", "c").traverse(putStrLn)
res1: scalaz.effects.IO[scala.collection.immutable.Stream[Unit]] = scalaz.effects.IO$$anon$2@32064883

scala> res1.unsafePerformIO
a
b
c
res2: scala.collection.immutable.Stream[Unit] = Stream((), ?)

There is also a version where the result is discarded, if we only care about the effect:

scala> Stream("a", "b", "c").traverse_(putStrLn)
res3: scalaz.effects.IO[Unit] = scalaz.effects.IO$$anon$2@13641904

scala> res3.unsafePerformIO
a
b
c

Here is an example where the "effect" is that the computation is executed concurrently by a background thread:

scala> List(1,2,3).traverse(x => promise { x * 2 })
res4: scalaz.concurrent.Promise[List[Int]] = <promise>

scala> res4.get
res5: List[Int] = List(2, 4, 6)

A much more sophisticated example is where the "effect" is that computation succeeds or fails in an Either-like data structure, with failures accumulated on the left in a non-empty list:

scala> type MyValidation[A] = ValidationNEL[NumberFormatException, A]
defined type alias MyValidation

If everything succeeds, we get a success with a list in it:

scala> List("1", "2").traverse[MyValidation, Int](_.parseInt.liftFailNel)
res150: MyValidation[List[Int]] = Success(List(1, 2))

If anything fails, we get a failure with a nonempty list of errors in it:

scala> List("1", "a", "b").traverse[MyValidation, Int](_.parseInt.liftFailNel)
res151: MyValidation[List[Int]] = Failure(NonEmptyList(java.lang.NumberFormatException: For input string: "a", java.lang.NumberFormatException: For input string: "b"))

The "traverse" method is defined on any type F[_] for which there exists an implicit Traverse[F]. The method accepts a function of type A => M[B], and results in M[F[B]]. An implicit Applicative[M] instance is required. So M must be some applicative functor. These are exceedingly common. Any monad is an applicative functor. Every monoid yields an applicative functor (so traverse works as you would expect if M[B] happens to be a monoid, therefore it also generalizes foldLeft and foldRight), and of course if F[_] is applicative and G[_] is applicative, then the composite functor F[G[_]] is applicative as well.

Here is the what the Applicative interface might look like. A minimal definition requires overriding at least either "apply" or "zipWith" in addition to implementing "pure".

trait Applicative[Z[_]] {
  def map[A, B](fa: Z[A], f: A => B): Z[B] = apply(pure(f), fa)
  def apply[A, B](f: Z[A => B], a: Z[A]): Z[B] = zipWith(f, a, (_:A => B)(_: A))
  def zipWith[A, B, C](a: Z[A], b: Z[B], f: (A, B) => C): Z[C] = apply(map(a, f.curried), b)
  def pure[A](a: => A): Z[A] 
}

Traverse with Applicative is extremely versatile. Here it is generalizing "unzip":

scala> type Pair[A] = (A, A)
defined type alias Pair

scala> implicit val pairApplicative = new Applicative[Pair] {
     |   override def apply[A, B](f: (A => B, A => B), a: (A, A)) = (f._1(a._1), f._2(a._2))
     |   def pure[A](a: => A) = (a, a)
     | }
pairApplicative: java.lang.Object with scalaz.Applicative[Pair] = $anon$1@63721cce

We now have a kind of "unzip" operation on anything that can be traversed.

scala> List((1, 10),(2, 20),(3, 30)).traverse[Pair, Int](x => x)
res191: (List[Int], List[Int]) = (List(1, 2, 3),List(10, 20, 30))

Traversing with the identity function has a shorthand called "sequence"

scala> some((1, 2)).sequence[Pair, Int]
res193: (Option[Int], Option[Int]) = (Some(1),Some(2))

On Wed, Oct 12, 2011 at 7:52 AM, Runar Bjarnason <runarorama@gmail.com> wrote:

 
On Tuesday, October 11, 2011 6:18:55 PM UTC-4, martin odersky wrote:

> Maybe "map" is not the abstraction you're looking for, but "traverse":

I definitively want a one-parameter function like map. Not sure what
it would take to make traverse into that. What I see so far looks
complicated, but I can see behind that if one presents the use cases
well.

 Not complicated at all. The pattern is presented in some detail in the paper "The Essence of the Iterator Pattern" by Gibbons and Oliveira:
http://www.comlab.ox.ac.uk/jeremy.gibbons/publications/iterator.pdf

The basic idea is that data of some type F[A] can be traversed with a function of type A => M[B] to produce M[F[B]], where M represents some effect, and supports certain operations.

We have implemented this in Scalaz. For example, here is a use case where the "effect" is none at all. This is just "map":

scala> List(1,2,3).traverse[Id, Int](_ * 2)
res0: List[Int] = List(2, 4, 6)

The "Id" type here is just "type Id[A] = A".

BTW, we're working on a new type class organization so that Applicative[ID] can override `traverse`, `sequence` et al for efficiency.
https://github.com/retronym/scalaz7-experimental/blob/master/example/src/main/scala/scalaz/example/TraverseUsage.scala
https://github.com/retronym/scalaz7-experimental/blob/master/core/src/main/scala/scalaz/Ident.scala
https://github.com/retronym/scalaz7-experimental/blob/master/core/src/main/scala/scalaz/Applicative.scala
-jason

On Wed, Oct 12, 2011 at 7:52 AM, Runar Bjarnason wrote:
>
> On Tuesday, October 11, 2011 6:18:55 PM UTC-4, martin odersky wrote:
>>
>> > Maybe "map" is not the abstraction you're looking for, but "traverse":
>>
>> I definitively want a one-parameter function like map. Not sure what
>> it would take to make traverse into that. What I see so far looks
>> complicated, but I can see behind that if one presents the use cases
>> well.
>
>
> Not complicated at all. The pattern is presented in some detail in the paper
> "The Essence of the Iterator Pattern" by Gibbons and Oliveira:
> http://www.comlab.ox.ac.uk/jeremy.gibbons/publications/iterator.pdf
>
> The basic idea is that data of some type F[A] can be traversed with a
> function of type A => M[B] to produce M[F[B]], where M represents some
> effect, and supports certain operations.
>
> We have implemented this in Scalaz. For example, here is a use case where
> the "effect" is none at all. This is just "map":
>
> scala> List(1,2,3).traverse[Id, Int](_ * 2)
> res0: List[Int] = List(2, 4, 6)
>
> The "Id" type here is just "type Id[A] = A".
>
> Here is one where the effect is printing to the console:
>
> scala> Stream("a", "b", "c").traverse(putStrLn)
> res1: scalaz.effects.IO[scala.collection.immutable.Stream[Unit]] =
> scalaz.effects.IO$$anon$2@32064883
>
> scala> res1.unsafePerformIO
> a
> b
> c
> res2: scala.collection.immutable.Stream[Unit] = Stream((), ?)
>
> There is also a version where the result is discarded, if we only care about
> the effect:
>
> scala> Stream("a", "b", "c").traverse_(putStrLn)
> res3: scalaz.effects.IO[Unit] = scalaz.effects.IO$$anon$2@13641904
>
> scala> res3.unsafePerformIO
> a
> b
> c
>
> Here is an example where the "effect" is that the computation is executed
> concurrently by a background thread:
>
> scala> List(1,2,3).traverse(x => promise { x * 2 })
> res4: scalaz.concurrent.Promise[List[Int]] =
>
> scala> res4.get
> res5: List[Int] = List(2, 4, 6)
>
> A much more sophisticated example is where the "effect" is that computation
> succeeds or fails in an Either-like data structure, with failures
> accumulated on the left in a non-empty list:
>
> scala> type MyValidation[A] = ValidationNEL[NumberFormatException, A]
> defined type alias MyValidation
>
> If everything succeeds, we get a success with a list in it:
>
> scala> List("1", "2").traverse[MyValidation, Int](_.parseInt.liftFailNel)
> res150: MyValidation[List[Int]] = Success(List(1, 2))
>
> If anything fails, we get a failure with a nonempty list of errors in it:
>
> scala> List("1", "a", "b").traverse[MyValidation,
> Int](_.parseInt.liftFailNel)
> res151: MyValidation[List[Int]] =
> Failure(NonEmptyList(java.lang.NumberFormatException: For input string: "a",
> java.lang.NumberFormatException: For input string: "b"))
>
> The "traverse" method is defined on any type F[_] for which there exists an
> implicit Traverse[F]. The method accepts a function of type A => M[B], and
> results in M[F[B]]. An implicit Applicative[M] instance is required. So M
> must be some applicative functor. These are exceedingly common. Any monad is
> an applicative functor. Every monoid yields an applicative functor (so
> traverse works as you would expect if M[B] happens to be a monoid, therefore
> it also generalizes foldLeft and foldRight), and of course if F[_] is
> applicative and G[_] is applicative, then the composite functor F[G[_]] is
> applicative as well.
>
> Here is the what the Applicative interface might look like. A minimal
> definition requires overriding at least either "apply" or "zipWith" in
> addition to implementing "pure".
>
> trait Applicative[Z[_]] {
> def map[A, B](fa: Z[A], f: A => B): Z[B] = apply(pure(f), fa)
> def apply[A, B](f: Z[A => B], a: Z[A]): Z[B] = zipWith(f, a, (_:A => B)(_:
> A))
> def zipWith[A, B, C](a: Z[A], b: Z[B], f: (A, B) => C): Z[C] =
> apply(map(a, f.curried), b)
> def pure[A](a: => A): Z[A]
> }
>
> Traverse with Applicative is extremely versatile. Here it is generalizing
> "unzip":
>
> scala> type Pair[A] = (A, A)
> defined type alias Pair
>
> scala> implicit val pairApplicative = new Applicative[Pair] {
>      |   override def apply[A, B](f: (A => B, A => B), a: (A, A)) =
> (f._1(a._1), f._2(a._2))
>      |   def pure[A](a: => A) = (a, a)
>      | }
> pairApplicative: java.lang.Object with scalaz.Applicative[Pair] =
> $anon$1@63721cce
>
> We now have a kind of "unzip" operation on anything that can be traversed.
>
> scala> List((1, 10),(2, 20),(3, 30)).traverse[Pair, Int](x => x)
> res191: (List[Int], List[Int]) = (List(1, 2, 3),List(10, 20, 30))
>
> Traversing with the identity function has a shorthand called "sequence"
>
> scala> some((1, 2)).sequence[Pair, Int]
> res193: (Option[Int], Option[Int]) = (Some(1),Some(2))
>
>
That looks indeed very nice. I am still a bit confused because the
scalaz sources you showed had a two-argument traverse. What is the
relationship between the two?

In essence, you are saying you can simulate effect polymorphism
through higher-kinded types, which is very neat.

Btw I was on the thesis committee of Bruno Oliveira. Pity the "essence
of the iterator pattern" paper did not form part of the thesis so it
was not discussed in the defense. His thesis was all about visitors,
which, in the presence of pattern matching is a less obvious win.

On Wed, Oct 12, 2011 at 9:48 AM, martin odersky <martin.odersky@epfl.ch> wrote: [snip]

> We now have a kind of "unzip" operation on anything that can be traversed. >
> scala> List((1, 10),(2, 20),(3, 30)).traverse[Pair, Int](x => x)
> res191: (List[Int], List[Int]) = (List(1, 2, 3),List(10, 20, 30))
>
> Traversing with the identity function has a shorthand called "sequence"
>
> scala> some((1, 2)).sequence[Pair, Int]
> res193: (Option[Int], Option[Int]) = (Some(1),Some(2))
>
>
That looks indeed very nice. I am still a bit confused because the
scalaz sources you showed had a two-argument traverse. What is the
relationship between the two?

In essence, you are saying you can simulate effect polymorphism
through higher-kinded types, which is very neat.

Btw I was on the thesis committee of Bruno Oliveira. Pity the "essence
of the iterator pattern" paper did not form part of the thesis so it
was not discussed in the defense. His thesis was all about visitors,
which, in the presence of pattern matching is a less obvious win.

val ls: List[A]val f: A => Effect[B]ls traverse f  
// expands to: implicitly[Traverse[List]].traverse(ls, f)(implicitly[Applicative[Effect]]
-jason

On Wed, Oct 12, 2011 at 9:54 AM, Jason Zaugg wrote:
> On Wed, Oct 12, 2011 at 9:48 AM, martin odersky
> wrote:
>  [snip]
>>
>> > We now have a kind of "unzip" operation on anything that can be
>> > traversed.
>> >
>> > scala> List((1, 10),(2, 20),(3, 30)).traverse[Pair, Int](x => x)
>> > res191: (List[Int], List[Int]) = (List(1, 2, 3),List(10, 20, 30))
>> >
>> > Traversing with the identity function has a shorthand called "sequence"
>> >
>> > scala> some((1, 2)).sequence[Pair, Int]
>> > res193: (Option[Int], Option[Int]) = (Some(1),Some(2))
>> >
>> >
>> That looks indeed very nice. I am still a bit confused because the
>> scalaz sources you showed had a two-argument traverse. What is the
>> relationship between the two?
>>
>> In essence, you are saying you can simulate effect polymorphism
>> through higher-kinded types, which is very neat.
>>
>> Btw I was on the thesis committee of Bruno Oliveira. Pity the "essence
>> of the iterator pattern" paper did not form part of the thesis so it
>> was not discussed in the defense. His thesis was all about visitors,
>> which, in the presence of pattern matching is a less obvious win.
>
> val ls: List[A]
> val f: A => Effect[B]
> ls traverse f
> // expands to:
> implicitly[Traverse[List]].traverse(ls, f)(implicitly[Applicative[Effect]]
> -jason
>
I understand. Thanks for the explanation, Jason!

Adriaan Moors epfl.ch> writes:
> if `f` prints to the console (or throws an exception, or formats your
> drive, or all of the above), so will `map`
>
> thus, `map f` should be considered as having the same effect as `f`
> (assuming a "traditional" implementation of `map`, which actually
> executes `f`)
>
> it seems to me it would be unsound not to have this
>
> you seem to be saying map should be ad-hoc polymorphic in terms of the
> purity of f (in the sense of having overloads for different effects). 
> I think that's reasonable, but it does not make the above untrue.

I'm saying that a map operation with side effects is not the same as a map
operation without side effects. Consider the Haskell definitions:

mapM :: Monad m => (a -> m b) -> [a] -> m [b]
map :: (a -> b) -> [a] -> [b]

For mapM the reasonable choice is to perform the map operations in sequence,
but for map you can for example perform the operations in parallel (not very
useful for lists but consider the generic collection case) and there will be
no observable difference. The contracts of map and mapM are different so
they are not the same function.

/Jesper Nordenberg

On Wed, Oct 12, 2011 at 10:31 AM, Jesper Nordenberg <megagurka@yahoo.com> wrote:

Adriaan Moors <adriaan.moors <at> epfl.ch> writes:
> if `f` prints to the console (or throws an exception, or formats your
> drive, or all of the above), so will `map`
>
> thus, `map f` should be considered as having the same effect as `f`
> (assuming a "traditional" implementation of `map`, which actually
> executes `f`)
>
> it seems to me it would be unsound not to have this
>
> you seem to be saying map should be ad-hoc polymorphic in terms of the
> purity of f (in the sense of having overloads for different effects). 
> I think that's reasonable, but it does not make the above untrue.

I'm saying that a map operation with side effects is not the same as a map
operation without side effects.

and I'm saying it doesn't have to be that way -- you can have different implementations if you want to (using ad-hoc polymorphism)but it shouldn't be imposed on you by the type system, so that you can have one parametrically polymorphic map (which abstracts over the effect, just like it abstracts over the element type)
as the scalaz folks have shown with traverse, this restriction is not inherent to monads/applicative functors; it stems from a restriction in Haskell's type system (you can't abstract over class contexts, so that you can't write one map to traverse them all -- i.e., this signature is not expressible in Haskell:  map :: M m => (a -> m b) -> [a] -> m [b] -- where M is abstract, but this is exactly what traverse achieves)

Consider the Haskell definitions:

mapM :: Monad m => (a -> m b) -> [a] -> m [b]
map :: (a -> b) -> [a] -> [b] 

For mapM the reasonable choice is to perform the map operations in sequence,
but for map you can for example perform the operations in parallel (not very
useful for lists but consider the generic collection case) and there will be
no observable difference. The contracts of map and mapM are different so
they are not the same function.

I'm not sure -- is the following really that big a leap:from... abstracting over the builder in order to map Lists to Lists and Sets to Sets without implementing map all over for each combination to...    using the same kind of abstractions to be able to plug in different evaluation strategies (much like we are doing with the parallel collections, in fact)
abstraction is about factoring out commonalities, and varying only what is truly different

On Wednesday, October 12, 2011 4:31:16 AM UTC-4, Jesper Nordenberg wrote:

For mapM the reasonable choice is to perform the map operations in sequence,

With the constraint that m is a Monad, yes. But the implication of Applicative is that combination of effects is associative.

From Data.Traversable in the Haskell base libraries:

traverse :: Applicative f => (a -> f b) -> t a -> f (t b)