questioning FP

2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

I know where the problem  comes from. But unless it is solved, the abstraction (or at least the specific implementation in #scalaz) has limited value

You could also question the value of maintaining a large, strict and flat collection in main memory. I you really need that a finger tree would be more efficient and I guess it could be traversed without a SO.

--
Sébastien

Since when was 10,000 elements a "large [..] collection"? On a daily basis I deal with collections several orders of magnitude greater than this.

Date: Sat, 15 Oct 2011 19:03:54 +0200
Subject: Re: [scala-debate] Re: questioning FP
From: sebastien.bocq@gmail.com
To: oxbow_lakes@hotmail.com
CC: scala-debate@googlegroups.com



2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

I know where the problem  comes from. But unless it is solved, the abstraction (or at least the specific implementation in #scalaz) has limited value

You could also question the value of maintaining a large, strict and flat collection in main memory. I you really need that a finger tree would be more efficient and I guess it could be traversed without a SO.

--
Sébastien

2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

Since when was 10,000 elements a "large [..] collection"? On a daily basis I deal with collections several orders of magnitude greater than this.

You're taking what I said out of context. The [..] was not optional. I wouldn't keep so many values in a flat structure like a linked list, or an array. Operations would be way too expensive.
 
--
Sébastien

Doesn't this depend very much on the operations you need?  Lists and arrays make for very efficient stacks and give fast traversal compared to e.g. finger trees.  And flat structures which need little manipulation are particularly common when dealing with IO.

On Sat, Oct 15, 2011 at 1:18 PM, Sébastien Bocq <sebastien.bocq@gmail.com> wrote:

2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

Since when was 10,000 elements a "large [..] collection"? On a daily basis I deal with collections several orders of magnitude greater than this.

You're taking what I said out of context. The [..] was not optional. I wouldn't keep so many values in a flat structure like a linked list, or an array. Operations would be way too expensive.
 
--
Sébastien

On Sat, Oct 15, 2011 at 12:26 PM, Ittay Dror <ittay.dror@gmail.com> wrote:

State monad is S => (A, S), so the state here is Connection, which doesn't change (or at least, not in a way where you can distinguish I think), so it looks like A, the result, is produced  without referential transparency (Runar, correct me please). 

Well, actually, it is a state monad and it depends on the version of Scalaz which order the result or the state are in:
Scalaz6sealed trait State[S, +A] {  def apply(s: S): (S, A)
Scalaz7:sealed trait StateT[S, F[_], A] {  def apply(s: S): F[(A, S)] = ...trait StateTs {  type State[S, A] = StateT[S, Identity, A]  The order of the values returned in the Tuple has no significance, you only need to be consistent.  In Runar's example he puts the state in the first position instead of the second.  No biggie.
--
Jim Powers

body p { margin-bottom: 0cm; margin-top: 0pt; }

Jim Powers wrote:

h5iPtkDy628nMNJQx1BLwa4W2kU-Q-iykM-aho-cgQ [at] mail [dot] gmail [dot] com" type="cite">On Sat, Oct 15, 2011 at 12:26 PM, Ittay Dror <ittay [dot] dror [at] gmail [dot] com" rel="nofollow">ittay.dror@gmail.com> wrote:

State monad is S => (A, S), so the state here is Connection, which doesn't change (or at least, not in a way where you can distinguish I think), so it looks like A, the result, is produced  without referential transparency (Runar, correct me please). 

Well, actually, it is a state monad and it depends on the version of Scalaz which order the result or the state are in:
Scalaz6 sealed trait State[S, +A] {   def apply(s: S): (S, A)
Scalaz7: sealed trait StateT[S, F[_], A] {   def apply(s: S): F[(A, S)] = ... trait StateTs {   type State[S, A] = StateT[S, Identity, A]   The order of the values returned in the Tuple has no significance, you only need to be consistent.  In Runar's example he puts the state in the first position instead of the second.  No biggie.

I wasn't referring to order. The state is the argument to the function (Connection in this case). And you're supposed to return a new state instance (e.g., pop an element from a stack state). Otherwise, the function pulls a value from thin air ==> not referentially transparent. In Runar's example, I suspect that the function will return the same value when invoked again on the returned connection since its state is opaque.

Ittay

h5iPtkDy628nMNJQx1BLwa4W2kU-Q-iykM-aho-cgQ [at] mail [dot] gmail [dot] com" type="cite">
--
Jim Powers

True, but is there a good practical use case for maintaining extremely large stacks that must be traversed one element at a time in one sweep?

You can build ropes on top finger trees, which present a good trade-off between traversing and other manipulation functions. But this is another discussion.

Could you give an example of the flat structures common for IO? I don't see what you mean.

2011/10/15 Rex Kerr <ichoran@gmail.com>

Doesn't this depend very much on the operations you need?  Lists and arrays make for very efficient stacks and give fast traversal compared to e.g. finger trees.  And flat structures which need little manipulation are particularly common when dealing with IO.

On Sat, Oct 15, 2011 at 1:18 PM, Sébastien Bocq <sebastien.bocq@gmail.com> wrote:

2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

Since when was 10,000 elements a "large [..] collection"? On a daily basis I deal with collections several orders of magnitude greater than this.

You're taking what I said out of context. The [..] was not optional. I wouldn't keep so many values in a flat structure like a linked list, or an array. Operations would be way too expensive.
 
--
Sébastien

--
Sébastien

On Sat, Oct 15, 2011 at 12:07 PM, Rex Kerr <ichoran@gmail.com> wrote:

On Sat, Oct 15, 2011 at 11:41 AM, Jim Powers <jim@casapowers.com> wrote:

On Sat, Oct 15, 2011 at 3:27 AM, Rex Kerr <ichoran@gmail.com> wrote:

The solution was of course to refactor so that opening a database connection was disallowed. Instead, a command would assume an open database connection, do something with it, and pass it on. The type we used was something like this:

type IO[A] = java.sql.Connection => (A, java.sql.Connection)

Now, instead of having outer commands call inner commands, we simply chain them with Kleisli composition: outer.compose(inner), and they share the same connection and act as "one command". This resulted in us refactoring to a bunch of primitive commands that we could chain, instead of monolithic ones.

(3) refactoring as a bunch of primitive commands that you chain by sequential appearance in a method (or in a list over which you fold or map the connection) rather than Kleisli composition.  (I.e. do the same thing, but write methods that implement only java.sql.Connection => A; if you're not enforcing via types only certain combinations, there isn't much point in returning the connection.)

Runar's example is using a state monad where the A is the type of the state.

Ah, okay.  Nothing wrong with using a state monad; it's a perfectly sensible way to build a finite state machine.  I'm not that familiar with it, so I didn't recognize it from the signature.

State in the State Monad does not refer to a finite state machine, it is arbitrary state.  For example:
import scalaz._ import Scalaz._
case class Foo(x:Int)
def plusOne:Foo => Foo =   (f => f.copy(x = f.x + 1))
val plusTwo:State[Foo,Unit] = for {  _ <- modify[Foo](plusOne)   _ <- modify[Foo](plusOne)} yield ()
plusTwo ~> Foo(0)
==> Foo(2)
You can now compose plusTwos together:
val plusFour:State[Foo,Unit] = for {  _ <- plusTwo  _ <- plusTwo } yield ()
plusFour ~> Foo(0)
==> Foo(4)
--
Jim Powers

On Oct 15, 2011, at 3:27, Rex Kerr wrote:

>
> That sort of works (although the type you've specified works only for input?

A combinator for writing to the database just has type IO[Unit].

> Or you're using A to store state in the type system as well as wrap input?). What's not clear, however, is how this is superior to:

I'm sure I could have solved it with an ad hoc solution like an AbstractBeanFactoryProxy or something simple like that, but the way I see it is that I needed nested dependencies with shared context, and that's what a monad is.

> (or in a list over which you fold or map the connection) rather than Kleisli composition.

That is Kleisli composition.

> (I.e. do the same thing, but write methods that implement only java.sql.Connection => A;

That's perfectly reasonable, and that's just the Reader monad. I wanted to use something more State-like to discourage parallelism.

I must be doing something very, very wrong as I regularly use strict structures for large numbers of items. Matching engines for example. Or anything which uses Sets.

Date: Sat, 15 Oct 2011 19:18:19 +0200
Subject: Re: [scala-debate] Re: questioning FP
From: sebastien.bocq@gmail.com
To: oxbow_lakes@hotmail.com
CC: scala-debate@googlegroups.com



2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

Since when was 10,000 elements a "large [..] collection"? On a daily basis I deal with collections several orders of magnitude greater than this.

You're taking what I said out of context. The [..] was not optional. I wouldn't keep so many values in a flat structure like a linked list, or an array. Operations would be way too expensive.
 
--
Sébastien

On Sat, Oct 15, 2011 at 10:38 AM, Chris Marshall <oxbow_lakes@hotmail.com> wrote:

Personally I think this whole thread has been hugely interesting and I love this particular email from Runar. Unfortunately, for those keen to use it:
scala> (1 to 10000 toList).traverse[Option, Int](x => Some(x * 2))java.lang.StackOverflowError at scalaz.Traverse$$anon$3$$anonfun$2$$anonfun$apply$3.apply(Traverse.scala:28) at scalaz.Applys$$anon$2$$anonfun$apply$1$$anonfun$apply$2.apply(Apply.scala:12) at scala.Option.map(Option.scala:133)         ... etc
I'm sure you know that I by no means intend to denigrate the efforts being put into scalaz and I have the highest regard to all involved, but fantastic abstractions that are unusable in practice are, um, unusable in practice.

A bit off topic, but I think this is fixed in scalaz master, at least for List. Using traverse with any other Traversable involves Stream.foldr which is not stack friendly. If you modify TraversableTraverse to use List instead of Stream you should avoid the stack overflow.
I keep kicking myself for submitting a broken fix for this problem (tried to optimize it too much, so was rightfully reverted), if I had just kept it simple this wouldn't be a problem.
--
Derek Williams

On Sat, Oct 15, 2011 at 2:26 PM, Runar Oli <runarorama@gmail.com> wrote:
>
>
> On Oct 15, 2011, at 3:27, Rex Kerr <ichoran@gmail.com> wrote:
>
>>
>> That sort of works (although the type you've specified works only for input?
>
> A combinator for writing to the database just has type IO[Unit].

Okay, but that's just a marker.


>> Or you're using A to store state in the type system as well as wrap input?).  What's not clear, however, is how this is superior to:
>
> I'm sure I could have solved it with an ad hoc solution like an AbstractBeanFactoryProxy or something simple like that, but the way I see it is that I needed nested dependencies with shared context, and that's what a monad is.

I did agree that it solved the problem.

>>> (or in a list over which you fold or map the connection) rather than Kleisli composition.
>
> That is Kleisli composition.

It's a special case, no?  I thought that type transformations along the chain were part of the point of Kleisli composition.  (Then again, given your type signature, it doesn't look like you were doing this.)

>>   (I.e. do the same thing, but write methods that implement only java.sql.Connection => A;
>
> That's perfectly reasonable, and that's just the Reader monad. I wanted to use something more State-like to discourage parallelism.

Seems like it's just as easy to use or not use parallelism in each.  If you're not updating the types, you can always grab something out of order and try to use it twice in parallel.  java.sql.Connection isn't going to know any better.

  --Rex

2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

I must be doing something very, very wrong as I regularly use strict structures for large numbers of items. Matching engines for example. Or anything which uses Sets.

You're probably doing it well. Efficient implementations of Sets are not flat. That was also in the [...]. My take is that it is possible to recurse over these structures and implement traverse without blowing up the stack.
 
--
Sébastien

Well, an IndexedSeq will be a Vector, which isn't flat either (as I understand it):
scala> (1 to 10000 toIndexedSeq).traverse[Option, Int](x => Some(x * 2))java.lang.StackOverflowError at scalaz.Traverse$$anon$3$$anonfun$2$$anonfun$apply$3.apply(Traverse.scala:28) at scalaz.Applys$$anon$2$$anonfun$apply$1$$anonfun$apply$2.apply(Apply.scala:12) at scala.Option.map(Option.scala:133)

Date: Sat, 15 Oct 2011 20:45:59 +0200
Subject: Re: [scala-debate] Re: questioning FP
From: sebastien.bocq@gmail.com
To: oxbow_lakes@hotmail.com
CC: scala-debate@googlegroups.com



2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

I must be doing something very, very wrong as I regularly use strict structures for large numbers of items. Matching engines for example. Or anything which uses Sets.

You're probably doing it well. Efficient implementations of Sets are not flat. That was also in the [...]. My take is that it is possible to recurse over these structures and implement traverse without blowing up the stack.
 
--
Sébastien

Yes, but scalaz has no way to access and exploit the underlying structure.

Seems relies on a stream as intermediate collection:
https://github.com/scalaz/scalaz/blob/master/core/src/main/scala/scalaz/Traverse.scala

2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

Well, an IndexedSeq will be a Vector, which isn't flat either (as I understand it):
scala> (1 to 10000 toIndexedSeq).traverse[Option, Int](x => Some(x * 2))java.lang.StackOverflowError at scalaz.Traverse$$anon$3$$anonfun$2$$anonfun$apply$3.apply(Traverse.scala:28) at scalaz.Applys$$anon$2$$anonfun$apply$1$$anonfun$apply$2.apply(Apply.scala:12) at scala.Option.map(Option.scala:133)

Date: Sat, 15 Oct 2011 20:45:59 +0200
Subject: Re: [scala-debate] Re: questioning FP
From: sebastien.bocq@gmail.com
To: oxbow_lakes@hotmail.com
CC: scala-debate@googlegroups.com



2011/10/15 Chris Marshall <oxbow_lakes@hotmail.com>

I must be doing something very, very wrong as I regularly use strict structures for large numbers of items. Matching engines for example. Or anything which uses Sets.

You're probably doing it well. Efficient implementations of Sets are not flat. That was also in the [...]. My take is that it is possible to recurse over these structures and implement traverse without blowing up the stack.
 
--
Sébastien

--
Sébastien

The bug here is ultimately a result of an attempt to abstract on the core scala libraries, which cannot be done in a practical way (there is a subtle illusion that it can -- we all fell for it).

It's fixed in scalaz7 by abandoning the effort to abstract at that point.

On 16/10/2011 2:38 AM, "Chris Marshall" <oxbow_lakes@hotmail.com> wrote:

Personally I think this whole thread has been hugely interesting and I love this particular email from Runar. Unfortunately, for those keen to use it:
scala> (1 to 10000 toList).traverse[Option, Int](x => Some(x * 2))java.lang.StackOverflowError at scalaz.Traverse$$anon$3$$anonfun$2$$anonfun$apply$3.apply(Traverse.scala:28) at scalaz.Applys$$anon$2$$anonfun$apply$1$$anonfun$apply$2.apply(Apply.scala:12) at scala.Option.map(Option.scala:133)         ... etc
I'm sure you know that I by no means intend to denigrate the efforts being put into scalaz and I have the highest regard to all involved, but fantastic abstractions that are unusable in practice are, um, unusable in practice.
Chris
Date: Tue, 11 Oct 2011 22:52:38 -0700
From: runarorama@gmail.com
To: scala-debate@googlegroups.com
CC: runarorama@gmail.com; megagurka@yahoo.com; pub@razie.com; ittay.dror@gmail.com
Subject: [scala-debate] Re: questioning FP

  Not complicated at all. The pattern is presented in some detail in the paper "The Essence of the Iterator Pattern" by Gibbons and Oliveira:
http://www.comlab.ox.ac.uk/jeremy.gibbons/publications/iterator.pdf

The basic idea is that data of some type F[A] can be traversed with a function of type A => M[B] to produce M[F[B]], where M represents some effect, and supports certain operations.

We have implemented this in Scalaz. For example, here is a use case where the "effect" is none at all. This is just "map":

scala> List(1,2,3).traverse[Id, Int](_ * 2)
res0: List[Int] = List(2, 4, 6)

On Oct 15, 2011, at 14:51, Rex Kerr wrote:

>
> > A combinator for writing to the database just has type IO[Unit].
>
> Okay, but that's just a marker.
>

No.

On Saturday, October 15, 2011 2:22:32 PM UTC-4, Jim Powers wrote:

State in the State Monad does not refer to a finite state machine

It does. It's essentially a Mealy machine, which is characterized by the fact that outputs depend on both inputs and the current state. I.e. a state transition edge in a Mealy machine can be modeled with a function of type:
type Mealy[S, A, B] = (A, S) => (B, S)
Of course, if we curry the argument, this is just:
A => S => (B, S)
If we take the result type of this and wrap it:
case class State[S, B](apply: S => (B, S))
then a Mealy machine is just an arrow in the Kleisli category under the State monad:
A => State[S, B]

On Sat, Oct 15, 2011 at 4:30 PM, Runar Oli <runarorama@gmail.com> wrote:

On Oct 15, 2011, at 14:51, Rex Kerr <ichoran@gmail.com> wrote:

>
> > A combinator for writing to the database just has type IO[Unit].
>
> Okay, but that's just a marker.
>

No.

Depends what you call a marker, I guess.

It does help you compose input and output operations into one.

It doesn't help you use input to create output.
It doesn't help you keep your output straight.

In normal use cases it doesn't do anything _different_ than methods that call other methods, but it might be slightly more convenient as you don't have to keep passing the same argument (but it's slightly less convenient because you have to keep dragging Unit along, unless you have an (A => A) => (A => (Unit,A)) handy, which is pretty easy to arrange).

So, okay, it's not _just_ a marker.  It just doesn't seem to have novel properties.  You need one more level of sophistication for that.

  --Rex

OK sure. I guess what I was going for is State can meet all your statefull needs. It's not a thing for "special" State needs :-).

Then again, I don't know of any program that doesn't generate outputs based on inputs and current state.

--
Jim Powers

On Oct 15, 2011 7:28 PM, "Runar Bjarnason" <runarorama@gmail.com> wrote:

On Saturday, October 15, 2011 2:22:32 PM UTC-4, Jim Powers wrote:

State in the State Monad does not refer to a finite state machine

It does. It's essentially a Mealy machine, which is characterized by the fact that outputs depend on both inputs and the current state. I.e. a state transition edge in a Mealy machine can be modeled with a function of type:
type Mealy[S, A, B] = (A, S) => (B, S)
Of course, if we curry the argument, this is just:
A => S => (B, S)
If we take the result type of this and wrap it:
case class State[S, B](apply: S => (B, S))
then a Mealy machine is just an arrow in the Kleisli category under the State monad:
A => State[S, B]

That's what I was thinking of without knowing the terminology.  Thanks!  I'll hopefully remember what to call it next time.
  --Rex

On Sat, Oct 15, 2011 at 7:28 PM, Runar Bjarnason <runarorama@gmail.com> wrote:

On Saturday, October 15, 2011 2:22:32 PM UTC-4, Jim Powers wrote:

State in the State Monad does not refer to a finite state machine

It does. It's essentially a Mealy machine, which is characterized by the fact that outputs depend on both inputs and the current state. I.e. a state transition edge in a Mealy machine can be modeled with a function of type:
type Mealy[S, A, B] = (A, S) => (B, S)
Of course, if we curry the argument, this is just:
A => S => (B, S)
If we take the result type of this and wrap it:
case class State[S, B](apply: S => (B, S))
then a Mealy machine is just an arrow in the Kleisli category under the State monad:
A => State[S, B]

On Sat, Oct 15, 2011 at 2:09 AM, Runar Bjarnason <runarorama@gmail.com> wrote:

I can relate a story where having an IO monad solved a real problem at work. This was not very long ago, for a web-based application talking to a database via JDBC. This was a million-LOC enterprise wossname with all the trimmings. The part that talked to the database was designed with a kind of strategy pattern, where you would inherit from a class named Command and overload a method named "body". Commands were executed using execute, the implementation of which went something like this:
1. Read the inputs and validate them.
2. Open a database connection.
3. If all is well, execute the body, passing the inputs.
4. Close the database connections.

Plus some complicated error-handling etc, but you get the idea.

At some point we started noticing that the database was accumulating row locks while the application was running with about 5000 concurrent users. If left alone, the app would become unresponsive, so we resorted to manually killing database sessions.

The reason this was occurring is that some commands actually depended on other commands, so occasionally a programmer would implement a command so that it would call another Command's execute method directly. The nested session would of course depend on rows that were already being manipulated (and not committed yet) by the outer session.

Any red-blooded functional programmer would at this point be screaming "use a monad!" Wherever you have an inner thing that depends on an outer thing, but needs shared context, you have a monad. The solution was of course to refactor so that opening a database connection was disallowed. Instead, a command would assume an open database connection, do something with it, and pass it on. The type we used was something like this:

type IO[A] = java.sql.Connection => (A, java.sql.Connection)

Now, instead of having outer commands call inner commands, we simply chain them with Kleisli composition: outer.compose(inner), and they share the same connection and act as "one command". This resulted in us refactoring to a bunch of primitive commands that we could chain, instead of monolithic ones.

On a lark I decided to put my rather poor FP skills to the test to see if I could model Runar's example above.  Here's my current version:
https://gist.github.com/1291831
Like I said, I pretty much suck at FP (in particular Scala FP because things are generally less clear than in Haskell, especially when it comes to type type inference), but that may be a good thing in this case: more people may be able to understand my example.  It was a hoot to do and let me play around with Scalaz7, which is simply amazing.
While doing this example I cam across RegionT:https://github.com/scalaz/scalaz/blob/scalaz7/core/src/main/scala/scalaz/effect/RegionT.scala
Which is clearly the "right way" to handle connections and transactions.  Also, the current implementation is rather brittle in the face of error conditions.  Validations clearly could be used to good effect here.

Of course, this was in Java, which as everybody knows, is just a functional academic language, and we should all pat ourselves on the back for not being that academic.

Frankly, I find this amazing given the capability differences between Scala's and Java's type systems.  Must have been fun like a tooth abscess.

On Sun, Oct 16, 2011 at 11:07 PM, Jim Powers <jim@casapowers.com> wrote:

While doing this example I cam across RegionT: https://github.com/scalaz/scalaz/blob/scalaz7/core/src/main/scala/scalaz/effect/RegionT.scala
Which is clearly the "right way" to handle connections and transactions.  Also, the current implementation is rather brittle in the face of error conditions.  Validations clearly could be used to good effect here.

Sorry, I meant to say that the current implementation of my "toy"/"example" is brittle.  I can hardly say that about RegionT. 

--
Jim Powers

Looks awesome.

On Sunday, October 16, 2011 11:07:50 PM UTC-4, Jim Powers wrote:

On Sat, Oct 15, 2011 at 2:09 AM, Runar Bjarnason <runar...@gmail.com> wrote:

I can relate a story where having an IO monad solved a real problem at work. This was not very long ago, for a web-based application talking to a database via JDBC. This was a million-LOC enterprise wossname with all the trimmings. The part that talked to the database was designed with a kind of strategy pattern, where you would inherit from a class named Command and overload a method named "body". Commands were executed using execute, the implementation of which went something like this:
1. Read the inputs and validate them.
2. Open a database connection.
3. If all is well, execute the body, passing the inputs.
4. Close the database connections.

Plus some complicated error-handling etc, but you get the idea.

At some point we started noticing that the database was accumulating row locks while the application was running with about 5000 concurrent users. If left alone, the app would become unresponsive, so we resorted to manually killing database sessions.

The reason this was occurring is that some commands actually depended on other commands, so occasionally a programmer would implement a command so that it would call another Command's execute method directly. The nested session would of course depend on rows that were already being manipulated (and not committed yet) by the outer session.

Any red-blooded functional programmer would at this point be screaming "use a monad!" Wherever you have an inner thing that depends on an outer thing, but needs shared context, you have a monad. The solution was of course to refactor so that opening a database connection was disallowed. Instead, a command would assume an open database connection, do something with it, and pass it on. The type we used was something like this:

type IO[A] = java.sql.Connection => (A, java.sql.Connection)

Now, instead of having outer commands call inner commands, we simply chain them with Kleisli composition: outer.compose(inner), and they share the same connection and act as "one command". This resulted in us refactoring to a bunch of primitive commands that we could chain, instead of monolithic ones.

On a lark I decided to put my rather poor FP skills to the test to see if I could model Runar's example above.  Here's my current version:
https://gist.github.com/1291831
Like I said, I pretty much suck at FP (in particular Scala FP because things are generally less clear than in Haskell, especially when it comes to type type inference), but that may be a good thing in this case: more people may be able to understand my example.  It was a hoot to do and let me play around with Scalaz7, which is simply amazing.
While doing this example I cam across RegionT:https://github.com/scalaz/scalaz/blob/scalaz7/core/src/main/scala/scalaz/effect/RegionT.scala
Which is clearly the "right way" to handle connections and transactions.  Also, the current implementation is rather brittle in the face of error conditions.  Validations clearly could be used to good effect here.

Of course, this was in Java, which as everybody knows, is just a functional academic language, and we should all pat ourselves on the back for not being that academic.

Frankly, I find this amazing given the capability differences between Scala's and Java's type systems.  Must have been fun like a tooth abscess.

We can get something like regions, only specialized to tracking database connections: It's essentially like ST, and it would look something like this (assume java.sql.Connection):

https://gist.github.com/1292001

Note that this guarantees (in the type system) that if executeQuery can access a Connection, then it is open. It also guarantees that the connection is finally closed (because there is no combinator to close a connection inside an IOM, so you must do so via withConnection).



On Sunday, October 16, 2011 11:07:50 PM UTC-4, Jim Powers wrote:

On Sat, Oct 15, 2011 at 2:09 AM, Runar Bjarnason <runar...@gmail.com> wrote:

I can relate a story where having an IO monad solved a real problem at work. This was not very long ago, for a web-based application talking to a database via JDBC. This was a million-LOC enterprise wossname with all the trimmings. The part that talked to the database was designed with a kind of strategy pattern, where you would inherit from a class named Command and overload a method named "body". Commands were executed using execute, the implementation of which went something like this:
1. Read the inputs and validate them.
2. Open a database connection.
3. If all is well, execute the body, passing the inputs.
4. Close the database connections.

Plus some complicated error-handling etc, but you get the idea.

At some point we started noticing that the database was accumulating row locks while the application was running with about 5000 concurrent users. If left alone, the app would become unresponsive, so we resorted to manually killing database sessions.

The reason this was occurring is that some commands actually depended on other commands, so occasionally a programmer would implement a command so that it would call another Command's execute method directly. The nested session would of course depend on rows that were already being manipulated (and not committed yet) by the outer session.

Any red-blooded functional programmer would at this point be screaming "use a monad!" Wherever you have an inner thing that depends on an outer thing, but needs shared context, you have a monad. The solution was of course to refactor so that opening a database connection was disallowed. Instead, a command would assume an open database connection, do something with it, and pass it on. The type we used was something like this:

type IO[A] = java.sql.Connection => (A, java.sql.Connection)

Now, instead of having outer commands call inner commands, we simply chain them with Kleisli composition: outer.compose(inner), and they share the same connection and act as "one command". This resulted in us refactoring to a bunch of primitive commands that we could chain, instead of monolithic ones.

On a lark I decided to put my rather poor FP skills to the test to see if I could model Runar's example above.  Here's my current version:
https://gist.github.com/1291831
Like I said, I pretty much suck at FP (in particular Scala FP because things are generally less clear than in Haskell, especially when it comes to type type inference), but that may be a good thing in this case: more people may be able to understand my example.  It was a hoot to do and let me play around with Scalaz7, which is simply amazing.
While doing this example I cam across RegionT:https://github.com/scalaz/scalaz/blob/scalaz7/core/src/main/scala/scalaz/effect/RegionT.scala
Which is clearly the "right way" to handle connections and transactions.  Also, the current implementation is rather brittle in the face of error conditions.  Validations clearly could be used to good effect here.

Of course, this was in Java, which as everybody knows, is just a functional academic language, and we should all pat ourselves on the back for not being that academic.

Frankly, I find this amazing given the capability differences between Scala's and Java's type systems.  Must have been fun like a tooth abscess.

Sorry, I meant to say that the current implementation of my "toy"/"example" is brittle.  I can hardly say that about RegionT. 

From looking quickly at the source, it seems RegionT does not implement the subtyping of and inferring regions part of the paper.Are there any plans to support this?
(This is not a critic. RegionT is already awesome per se.)
Best,
Nicolas.

Regions are not fully implemented yet. Stay tuned.

Hi Ittay,
First of all, thanks for starting this thread because IO is an inevitable question to anyone trying to do FP.
I've been reading this thread on and off trying to setup my mind about how I should feel concerned about the issue you describe. Then I realized that I am completely concerned with specs2! 
Since the code is public, this might be a good example to support the discussion. Basically in specs2, when you execute a specification, you follow a (simplified) chain of actions like that:
 run(spec) = select |> execute |> report
In the latest version of specs2, I've added a feature doing some IO:
 run(spec) = select(IO[uses previous stats]) |> execute |> storeStats(does IO) |> report
This allows the user to select only examples which previously failed based on results stored on the disk. This also means that the whole chain should be more or less all "tainted" by IO types, whereas now the fact that I'm using IO is buried inside the components doing the selection  and the storage.
I take it, from the current discussion, that a more functional way to architecture things (and this why I was mentioning "architecture" in one of my tweets) would be to be something like this:
 run(spec) = readStats |> select |> execute |> report |> storeStats
This way the IO is left at the periphery of the system and not buried in the middle.
That being said, I think that one of your concerns is that we've propagated to the whole system something which is an "implementation detail": the way stats are stored and retrieved in order to provide the functionality. And you write "why should the top of the application care that I'm creating events as an architectural decision?".
Maybe we would benefit from a change of views here, as Geoff proposes. Writing logs is an important functionality of the application. It serves a user (the ops team, the developers). In that case, it's not a bad idea to make this kind of output explicit in the system. This might even have some influence on how we treat log statements as a first-class feature in a system.
Eric.
PS: new messages coming up to this thread as I write. I can't keep up,...

Just for cross-reference, I want to point out that this problem is now solved in scalaz.
Eric.

Hi Martin, 

 In fact, there is only one lazily
evaluated language in wide usage today and even its creators have said
that laziness was probably a mistake.

Do you have any links relevant to this? I'm interested because back when I was doing my (incomplete) Ph.D., laziness was God's Word--one of many things I disagreed with that led me to quit my degree. I amazed (but pleased) that reality could have forced a rethink of this idea, and am interested in reading up on it.
Thanks,Ken 

I've read through the responses so far, and am going to respond to the most general form of the question, "Questioning FP". I think it's a perfectly valid question applicable not just to monads, but to many other sacred cows of the FP world, up to and including mutable state.
First, let met make the obligatory statement. FP can be very valuable in certain circumstances. In fact, if performance was of no concern _whatsoever_, I imagine all of my code would be FP, and probably not even fancy FP at that.
That said, FP is _one amongst many_ programming tools that we use to confront the fact that manipulating information is the most complex engineering task in the world today. It should not aspire to be the "answer" (but unfortunately many proponents present it that way).
Unfortunately, much of Higher-Order Programming allowed by a language such as Scala suffers from what I think of as the Academia Problem. Here, you have a bunch of very bright people isolated from the rest of the world, who love playing with ideas and don't get the fact that other people don't share their love. In addition, they get tenure by pushing knowledge to the limits of obscurity. ("A Ph.D. is someone who learns more and more about less and less, until they know everything about nothing.") I believe this results in the overselling of many ideas that really can't make it in the real world. Don't get me wrong, some very valuable stuff also comes out of this process. But it's easy to be dazzled by phis and sigmas when in fact there is nothing of value there.
I hasten to add that Martin and his team have, IMHO, strongly avoided these pitfalls and have been steadfast in their pursuit of Scala as a "real" programming language.
W.R.T. the content of the original post (monads in I/O), I think that's just an example of what I've outlined above--academia trying to claim an answer it does not have. Actually, I would apply that to monads in general--there are certainly specific monads that are highly useful, but after going through many, many papers purporting to show the benefits of monads in general, I have been left cold.
Hope this helps,Ken

Sent from my phone

On Dec 5, 2011, at 3:11, Kenneth McDonald wrote:

> I've read through the responses so far, and am going to respond to
> the most general form of the question, "Questioning FP". I think
> it's a perfectly valid question applicable not just to monads, but
> to many other sacred cows of the FP world, up to and including
> mutable state.
>
> First, let met make the obligatory statement. FP can be very
> valuable in certain circumstances. In fact, if performance was of no
> concern _whatsoever_, I imagine all of my code would be FP, and
> probably not even fancy FP at that.
>
> That said, FP is _one amongst many_ programming tools that we use to
> confront the fact that manipulating information is the most complex
> engineering task in the world today. It should not aspire to be the
> "answer" (but unfortunately many proponents present it that way).
>
> Unfortunately, much of Higher-Order Programming allowed by a
> language such as Scala suffers from what I think of as the Academia
> Problem. Here, you have a bunch of very bright people isolated from
> the rest of the world, who love playing with ideas and don't get the
> fact that other people don't share their love. In addition, they get
> tenure by pushing knowledge to the limits of obscurity. ("A Ph.D. is
> someone who learns more and more about less and less, until they
> know everything about nothing.") I believe this results in the
> overselling of many ideas that really can't make it in the real
> world. Don't get me wrong, some very valuable stuff also comes out
> of this process. But it's easy to be dazzled by phis and sigmas when
> in fact there is nothing of value there.
>
> I hasten to add that Martin and his team have, IMHO, strongly
> avoided these pitfalls and have been steadfast in their pursuit of
> Scala as a "real" programming language.

And so has Cay Horstman. And the Stanford guys. And the Netlogo
people. I don't doubt that there's content on these lists that may
look academic to many. But does it come from academics?

Martin

>
> W.R.T. the content of the original post (monads in I/O), I think
> that's just an example of what I've outlined above--academia trying
> to claim an answer it does not have. Actually, I would apply that to
> monads in general--there are certainly specific monads that are
> highly useful, but after going through many, many papers purporting
> to show the benefits of monads in general, I have been left cold.
>
> Hope this helps,
> Ken

Hi Ken, I don't think it's helpful to re-open an old thread with a potentially offensive claim.

Garbage collection (as an analogy to functional programming) has many benefits but arguably slows down software and definitely makes it harder to reason about memory usage.  Many people have accepted garbage collection as a tool that benefits more than it harms in many cases.

If academics are "isolated from the rest of the world", they must have some good ideas.  There's a lot less resistance to functional programming than their used to be.

As for phis and sigmas, calculus is full of symbols and proofs which are confusing when they are not known.  Despite its academic roots, its quite useful in engineering.  At the same time, many people pursue research in fruitless directions.  Doubtless they produce papers that are less than convincing.

What I gathered from the thread was that Ittay was looking for a good example of how the IO monad is useful.  Some examples were presented and people divided into two camps: those who found the IO monad useful and those that thought it was burdensome or didn't solve the problem at hand.

I think most people would agree that monads are useful in Haskell and are an interesting experiment in handling IO.  I think we'd all like to see other ways of handling IO, and we might even decide that we like the new ones better.

Peace.  Michael

Like the Ig Nobel Prize winners? http://improbable.com/ig/winners/

On Sunday, December 4, 2011 11:59:53 PM UTC-6, schmmd wrote:

Hi Ken, I don't think it's helpful to re-open an old thread with a potentially offensive claim.

I guess I don't see it as being terribly offensive. The fact that academics are often isolated from the demands of the "real world", is, as far as I'm concerned, often true. When I worked for the Human Genome Project, we were going to take a decade to map the genome--then Craig Ventner (millionaire private sector biologist) came along, and all of a sudden the time for mapping the genome came down to 2-3 years. As a software engineer on at one of the two major centers for that project (St. Louis), most of the software I had to work with was academic in origin--and it was laughable. No test coverage, not even a basic understanding of the fundamentals of software engineering, in any reasonable environment the people who had originated that software would have been fired immediately. And I have lots of other academic and semi-academic experience to back up my claim.
NOTE, that doesn't preclude me from saying that there are academic centers of excellence. Martin at EPFL is an obvious example, but I can think of others. John Ousterhout (Tcl/Tk) and his team did a fantastic job at engineering a widget kit that was way ahead of anything else, from a base in academia. For a long time, I followed Concurrent Clean as a promising (fully functional but adventurous) language out of academia.
I'm not trying to claim that academia is always bad; I am simply stating that academia is highly insulated from the "real world", and as such, is prone to veering off into meaningless tangents. All of my experience indicates this is a reasonable claim. (Reasonable does not imply correct--I'm not saying that I'm right, simply that people should consider this claim seriously.)

Garbage collection (as an analogy to functional programming) has many benefits but arguably slows down software and definitely makes it harder to reason about memory usage.  Many people have accepted garbage collection as a tool that benefits more than it harms in many cases.

I've seen no arguments that garbage collection either slows down software (significantly), or makes it harder to reason about memory usage. (Lazy evaluation, not _that_ makes it harder to reason about memory usage). Garbage collection, in simplistic forms, does introduce unpredictable pauses into program execution, which is perhaps what you intended to say. In any case, I don't see the parallels. GC is a very low-level construct that really doesn't affect the way that programs are written. 

If academics are "isolated from the rest of the world", they must have some good ideas.  There's a lot less resistance to functional programming than their used to be.

Enough chimps on enough typewriters will produce some useful ideas. And I wasn't trying to argue against the use of functional programming, simply against trying to push it too far--which seems to happen with all new ideas, not just FP 

As for phis and sigmas, calculus is full of symbols and proofs which are confusing when they are not known.  Despite its academic roots, its quite useful in engineering.  At the same time, many people pursue research in fruitless directions.  Doubtless they produce papers that are less than convincing.

Strongly disagree. I took calculus through ODEs and Vector calculus, simply because I found it beautiful. Those areas have had 100+ years to refine what is genuinely valuable vs. what is irrelevant. There is little left in those presentations that is irrelevant. 

What I gathered from the thread was that Ittay was looking for a good example of how the IO monad is useful.  Some examples were presented and people divided into two camps: those who found the IO monad useful and those that thought it was burdensome or didn't solve the problem at hand.

And that reflects the state of the art in FP right now; we haven't separated the wheat from the chaff. Count me on the cynical side, but I could still be proven wrong.
Ken 

I think most people would agree that monads are useful in Haskell and are an interesting experiment in handling IO.  I think we'd all like to see other ways of handling IO, and we might even decide that we like the new ones better.

Peace.  Michael

One inline reply. Also, I infer from your response that universities
are an "unreasonable work environment".

>> As for phis and sigmas, calculus is full of symbols and proofs which are
>> confusing when they are not known.  Despite its academic roots, its quite
>> useful in engineering.  At the same time, many people pursue research in
>> fruitless directions.  Doubtless they produce papers that are less than
>> convincing.

> Strongly disagree. I took calculus through ODEs and Vector calculus, simply
> because I found it beautiful. Those areas have had 100+ years to refine what
> is genuinely valuable vs. what is irrelevant. There is little left in those
> presentations that is irrelevant.

I don't understand what you are strongly disagreeing with. That some
people are confused by calculus because it contains symbols? That
calculus is useful in engineering? Or that some people produce papers
that are less than convincing?

On Sun, Dec 4, 2011 at 6:54 PM, Kenneth McDonald wrote:
> Hi Martin,
>
>>
>>  In fact, there is only one lazily
>> evaluated language in wide usage today and even its creators have said
>> that laziness was probably a mistake.
>
> Do you have any links relevant to this? I'm interested because back when I
> was doing my (incomplete) Ph.D., laziness was God's Word--one of many things
> I disagreed with that led me to quit my degree. I amazed (but pleased) that
> reality could have forced a rethink of this idea, and am interested in
> reading up on it.
> Thanks,
> Ken

Simon Peyton-Jones has oft been misquoted as saying, to paraphrase,
"the next Haskell won't be lazy," or something to that effect. The
thing the people quoting this always seem to miss is that he was being
facetious and, as I've heard reported, said it with a big smile.

I'm curious why, if you disagreed with it so much and had a well
reasoned argument about why laziness is undesirable, you didn't try
harder to over turn this popular opinion.

Rich

If I remember correctly, in one of his videos SJP (it's on the net somewhere) said that he might be tempted to implement a strict Haskell but then he was pretty sure that after a while he would miss the laziness and come back to his previous decision.
Eric.

Am Montag, 5. Dezember 2011, 23:43:08 schrieb Kenneth McDonald:
> On Sunday, December 4, 2011 11:59:53 PM UTC-6, schmmd wrote:
> > Hi Ken, I don't think it's helpful to re-open an old thread with a
> > potentially offensive claim.
>
> I guess I don't see it as being terribly offensive. The fact that academics
> are often isolated from the demands of the "real world", is, as far as I'm
> concerned, often true. When I worked for the Human Genome Project, we were
> going to take a decade to map the genome--then Craig Ventner (millionaire
> private sector biologist) came along, and all of a sudden the time for
> mapping the genome came down to 2-3 years. As a software engineer on at one
> of the two major centers for that project (St. Louis), most of the software
> I had to work with was academic in origin--and it was laughable. No test
> coverage, not even a basic understanding of the fundamentals of software
> engineering, in any reasonable environment the people who had originated
> that software would have been fired immediately. And I have lots of other
> academic and semi-academic experience to back up my claim.

Where those people writing that stuff informaticians? Or originated that stuff
from biologists?
It wouldn't be surprising if the latter was true. Biologists have a tendency
to "evolve" code and test it in production - that's the way our world works
;-) so its perfectly reasonable [I'm a biologist myself - so I dare make that
joke - no offense intended].

But I agree - industrial production settings are a lot different than academic
playgrounds. But I think we have to explore different ideas to separate the
good from the not-so-good ones.

And I am willing to attest from own experience that do-it-yourself imperative
programming with multiple threads an multiple cores (or on a single core by
the way) is a not-so-good one. I'm thankful that I see the dawn of
abstractions (including some FP patterns) that help me to circumvent at least
some of the concurrency pitfalls.

Greetings
Bernd

As I remember it, the main quoted advantage of laziness is that it keeps you honest (http://research.microsoft.com/en-us/um/people/simonpj/papers/haskell-ret...).
There must be other ways to stay true -- using an effect system, say.

I had lunch with SPJ today. Laziness is the correct default.

On Dec 6, 2011 10:22 AM, "etorreborre" <etorreborre@gmail.com> wrote:

If I remember correctly, in one of his videos SJP (it's on the net somewhere) said that he might be tempted to implement a strict Haskell but then he was pretty sure that after a while he would miss the laziness and come back to his previous decision.
Eric.

If I remember correctly, in one of his videos SJP (it's on the net somewhere) said that he might be tempted to implement a strict Haskell but then he was pretty sure that after a while he would miss the laziness and come back to his previous decision.

 This is the talk said something to this effect: http://skillsmatter.com/podcast/scala/talk-by-haskell-expert-simon-peyton-jones 

Exactly what I remembered (at 1:02:42), thanks Maciek!
And just after that he adds: "One thing I would not add is subtyping" :-).
Eric.

 
2011/12/6 Tony Morris <tmorris@tmorris.net>

I had lunch with SPJ today. Laziness is the correct default.

On Dec 6, 2011 10:22 AM, "etorreborre" <etorreborre@gmail.com> wrote:

If I remember correctly, in one of his videos SJP (it's on the net somewhere) said that he might be tempted to implement a strict Haskell but then he was pretty sure that after a while he would miss the laziness and come back to his previous decision.
Eric.

Not for systems programming though.
http://hasp.cs.pdx.edu/overview.html

Haskell would be pretty useless outside academia without C and its FFI. Isn't that because lazy languages lie about their underlying platform using a lot of built-in runtime magic? I heard kernel devs compare Haskell to swan swim: beautiful above the surface but with little legs paddling frenetically under the water. I don't know if it is entirely true but a funny analogy nonetheless.

On Wed, Dec 7, 2011 at 5:52 AM, Sébastien Bocq wrote:
> Haskell would be pretty useless outside academia without C and its FFI.
> Isn't that because lazy languages lie about their underlying platform using
> a lot of built-in runtime magic? I heard kernel devs compare Haskell to swan

jvms are written in c, no?
garbage collectors are written in c, no?
java most often runs on an operating system written in c, probably, no?
yet we use the java ecosystem and don't belittle it that much.

so how far away is haskell really? in some ways not so far. in other
ways, yes, far.

but that's the power of smart people working on good abstractions! :-)
once they refine it over, say, 50 years, then we've got a wonderful
thing.

On Mon, Dec 5, 2011 at 7:22 PM, etorreborre <etorreborre@gmail.com> wrote:

If I remember correctly, in one of his videos SJP (it's on the net somewhere) said that he might be tempted to implement a strict Haskell but then he was pretty sure that after a while he would miss the laziness and come back to his previous decision.

It was at this event: http://www.erlang-factory.com/conference/London2009/speakers/SimonPeytonJones (Sadly the video no longer seems to be available :-( )
After the presentation SPJ was asked if he had to do Haskell over again what would he change.  His response did include some musings about thinking hard about laziness by default, but he quickly added that he would probably miss its usefulness.
-- Jim Powers

On Fri, Dec 9, 2011 at 7:44 PM, Jim Powers <jim@casapowers.com> wrote:

It was at this event: http://www.erlang-factory.com/conference/London2009/speakers/SimonPeytonJones (Sadly the video no longer seems to be available :-( )
After the presentation SPJ was asked if he had to do Haskell over again what would he change.  His response did include some musings about thinking hard about laziness by default, but he quickly added that he would probably miss its usefulness.

Never mind, the Skills Matter link is the correct video.

--
Jim Powers

On Tue, Dec 6, 2011 at 6:52 AM, Tony Morris wrote:
> I had lunch with SPJ today. Laziness is the correct default.
>

The problem with the lazy by default choice is that you don't have a
few interesting types, like Lists...

In Haskell, [a] is a type of a coList. It is not bad per se but it
breaks a bit of modularity.
A function of type f :: a -> [a] can either return a finite or infinite coList.

Meaning that if somebody else wrote that function, I have to trust
documentation, ask him or read the implementation
before writing length(f "foo").

This is because the type system does not allow to restrict [a] to only
contain actual lists....

This link calls that paucity of types and gives interesting points on
the subject:

http://existentialtype.wordpress.com/2011/04/24/the-real-point-of-laziness/

Best regards,

Nicolas.

While laziness is handy is some cases, I just don't think it's a good
default on todays computers where strictness is the only way to get
reasonable performance.

Given unlimited memory and computing power things comes in a new
perspective.

/Jesper Nordenberg

nicolas.oury@gmail.com skrev 2011-12-10 10:16:
> On Tue, Dec 6, 2011 at 6:52 AM, Tony Morris wrote:
>> I had lunch with SPJ today. Laziness is the correct default.
>>
>
> The problem with the lazy by default choice is that you don't have a
> few interesting types, like Lists...
>
> In Haskell, [a] is a type of a coList. It is not bad per se but it
> breaks a bit of modularity.
> A function of type f :: a -> [a] can either return a finite or infinite coList.
>
> Meaning that if somebody else wrote that function, I have to trust
> documentation, ask him or read the implementation
> before writing length(f "foo").
>
> This is because the type system does not allow to restrict [a] to only
> contain actual lists....
>
> This link calls that paucity of types and gives interesting points on
> the subject:
>
> http://existentialtype.wordpress.com/2011/04/24/the-real-point-of-laziness/
>
> Best regards,
>
> Nicolas.
>

On 10/12/11 19:16, nicolas.oury@gmail.com wrote:
> On Tue, Dec 6, 2011 at 6:52 AM, Tony Morris wrote:
>> I had lunch with SPJ today. Laziness is the correct default.
>>
> The problem with the lazy by default choice is that you don't have a
> few interesting types, like Lists...
>
> In Haskell, [a] is a type of a coList. It is not bad per se but it
> breaks a bit of modularity.
> A function of type f :: a -> [a] can either return a finite or infinite coList.

This is not a problem with laziness. It is a problem with lists; in
particular, the fact that [a] is inhabited by a value for which length
does not produce a result. It is perfectly possible to produce a new
type for which this delineation exists.

> Meaning that if somebody else wrote that function, I have to trust
> documentation, ask him or read the implementation
> before writing length(f "foo").
>
> This is because the type system does not allow to restrict [a] to only
> contain actual lists....

Yes it does. Try it.

> This link calls that paucity of types and gives interesting points on
> the subject:
>
> http://existentialtype.wordpress.com/2011/04/24/the-real-point-of-laziness/

I have to admit, this article had me going the longest when it was first
written. It has been satisfactorily debunked by knowledgeable peers in
discussion. I am sympathetic to anyone who may have fallen for it like I
did.

> Best regards,
>
> Nicolas.