2.9.0.RC1 strangeness

It's probably worth mentioning that the code has been compiling fine since 2.8.0.

On 4/7/11 4:21 PM, Alex Cruise wrote:
> Nobody really likes paying for disk space, but are there any other
> places where old non-release builds are archived? Sometimes
> Hudson/Jenkins keeps build artifacts around, but I can't see any from my
> unauthenticated perspective.

I have thousands (every svn rev from 16000ish to 21000ish, and I'll
bring it up to date sooner or later.) Sitting on a disk. Would like to
make it available somehow if it can be easy and not bankrupt me.

On 4/7/11 4:21 PM, Alex Cruise wrote:

Nobody really likes paying for disk space, but are there any other
places where old non-release builds are archived?  Sometimes
Hudson/Jenkins keeps build artifacts around, but I can't see any from my
unauthenticated perspective.

I have thousands (every svn rev from 16000ish to 21000ish, and I'll bring it up to date sooner or later.) Sitting on a disk.  Would like to make it available somehow if it can be easy and not bankrupt me.

On 4/7/11 4:48 PM, Maxime Lévesque wrote:
> Ok, so should I open a ticket ?

Not unless it's going to be simplified from the email. Some of us are
slow and need to save our spare brainpower for setting alarm clocks and
traversing revolving doors.

I have thousands (every svn rev from 16000ish to 21000ish, and I'll bring it up to date sooner or later.) Sitting on a disk.  Would like to make it available somehow if it can be easy and not bankrupt me.

On 4/7/11 4:48 PM, Maxime Lévesque wrote:

Ok, so should I open a ticket ?

Not unless it's going to be simplified from the email.  Some of us are slow and need to save our spare brainpower for setting alarm clocks and traversing revolving doors.

Of course, If I open a ticket, it will include an "as minimal as possible" reproduction.
I was just asking in case someone would respond : "don't bother, something is seriously broken in the implicit convs. and we already know about it"


2011/4/7 Paul Phillips <paulp@improving.org>

On 4/7/11 4:48 PM, Maxime Lévesque wrote:

Ok, so should I open a ticket ?

Not unless it's going to be simplified from the email.  Some of us are slow and need to save our spare brainpower for setting alarm clocks and traversing revolving doors.

On 4/7/11 6:55 PM, Maxime Lévesque wrote:
> def nvlZ[A,B](a: NumericalExpressionZ[Option[A]], b:
> NumericalExpressionZ[B]) = new
> NvlFunctionNumericalZ[A,B](a.asInstanceOf[NumericalExpressionZ[A]],b)
>
> //Strange : commenting this line "fixes" compilation (side effect !?!?!)
> def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b:
> NonNumericalExpressionZ[A]) = null

That's a big clue right there.

You're calling an overloaded method and the overloading resolution
depends on implicit conversions. Then to add a little zazz you're
making it infer the return types of the overloaded methods.

Commenting out that line fixes it because it removes the overload. As
it is, the inferred return type of that method will be "Null" which is
going to get you in trouble because it's more specific.

It compiles like this:

def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b:
NonNumericalExpressionZ[A]): AnyRef = null

Type inference doesn't perform an exhaustive search of the solution
space. I don't know for sure, but I don't think it's a bug, just
something which you used to get away with.

> val q = nvlZ(Some(4F), 0)
> //Strange : this compiles :
> computeZ2(q)
>
> // while this doesn't :
> computeZ2(nvlZ(Some(4F), 0))

Type inference is performed with an "expected type". When you assign to
a value like that it's easy to see a different result because the
expected type of the call to computeZ2 will be more precise. When it's
one big expression with an onion of inferred types and implicit
conversions and overload resolution, sometimes the ?s are going to come
out the other end.

If you give the second one the same advantage as the first one, namely
knowing what the type of q is:

computeZ2(nvlZ(Some(4F), 0): NvlFunctionNumericalZ[Float,Int])

then it will compile.

Also just FYI your code example is a good distance from being minimized.

Ok, I managed to isolate a self containted reproduction, here it is, see the comments
and the compilation error message :

object Zaza {

  trait TypedExpressionNodeZ[T]
  class NumericalExpressionZ[T] extends TypedExpressionNodeZ[T]
  class NonNumericalExpressionZ[T] extends TypedExpressionNodeZ[T]

  implicit def conv1(i: Option[Float]) = new NumericalExpressionZ[Option[Float]]
  implicit def conv2(i: Int) = new NumericalExpressionZ[Int]

  implicit def binaryOpConv1Z(op: BinaryAMSOpZ[Byte,Byte]): TypedExpressionNodeZ[Byte] = null
  implicit def binaryOpConv2Z(op: BinaryAMSOpZ[Byte,Int]): TypedExpressionNodeZ[Int] = null
  implicit def binaryOpConv1Z1(op: BinaryAMSOpZ[Float,Int]): TypedExpressionNodeZ[Float] = null


  class BinaryAMSOpZ[A1,A2](a1: NumericalExpressionZ[A1], a2: NumericalExpressionZ[A2])

  class NvlFunctionNumericalZ[A1,A2](a1: NumericalExpressionZ[A1], a2: NumericalExpressionZ[A2]) extends BinaryAMSOpZ[A1,A2](a1,a2)

  def nvlZ[A,B](a: NumericalExpressionZ[Option[A]], b: NumericalExpressionZ[B]) = new NvlFunctionNumericalZ[A,B](a.asInstanceOf[NumericalExpressionZ[A]],b)

  //Strange : commenting this line "fixes" compilation (side effect !?!?!)
  def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b: NonNumericalExpressionZ[A]) = null

  def computeZ2[T1](e1: TypedExpressionNodeZ[T1]) = {}


  val q = nvlZ(Some(4F), 0)
  //Strange : this compiles :
  computeZ2(q)

  // while this doesn't :
  computeZ2(nvlZ(Some(4F), 0))
}



2011/4/7 Maxime Lévesque <maxime.levesque@gmail.com>

Of course, If I open a ticket, it will include an "as minimal as possible" reproduction.
I was just asking in case someone would respond : "don't bother, something is seriously broken in the implicit convs. and we already know about it"


2011/4/7 Paul Phillips <paulp@improving.org>

On 4/7/11 4:48 PM, Maxime Lévesque wrote:

Ok, so should I open a ticket ?

Not unless it's going to be simplified from the email.  Some of us are slow and need to save our spare brainpower for setting alarm clocks and traversing revolving doors.

  val q = nvlZ(Some(4F), 0)
  //Strange : this compiles :
  computeZ2(q)

  // while this doesn't :
  computeZ2(nvlZ(Some(4F), 0))

On 4/7/11 6:55 PM, Maxime Lévesque wrote:

  def nvlZ[A,B](a: NumericalExpressionZ[Option[A]], b:
NumericalExpressionZ[B]) = new
NvlFunctionNumericalZ[A,B](a.asInstanceOf[NumericalExpressionZ[A]],b)

  //Strange : commenting this line "fixes" compilation (side effect !?!?!)
  def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b:
NonNumericalExpressionZ[A]) = null

That's a big clue right there.

You're calling an overloaded method and the overloading resolution depends on implicit conversions.  Then to add a little zazz you're making it infer the return types of the overloaded methods.

Commenting out that line fixes it because it removes the overload.  As it is, the inferred return type of that method will be "Null" which is going to get you in trouble because it's more specific.

It compiles like this:

 def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b: NonNumericalExpressionZ[A]): AnyRef = null

Type inference doesn't perform an exhaustive search of the solution space.  I don't know for sure, but I don't think it's a bug, just something which you used to get away with.

  val q = nvlZ(Some(4F), 0)
  //Strange : this compiles :
  computeZ2(q)

  // while this doesn't :
  computeZ2(nvlZ(Some(4F), 0))

Type inference is performed with an "expected type".  When you assign to a value like that it's easy to see a different result because the expected type of the call to computeZ2 will be more precise.  When it's one big expression with an onion of inferred types and implicit conversions and overload resolution, sometimes the ?s are going to come out the other end.

If you give the second one the same advantage as the first one, namely knowing what the type of q is:

 computeZ2(nvlZ(Some(4F), 0): NvlFunctionNumericalZ[Float,Int])

then it will compile.

Also just FYI your code example is a good distance from being minimized.

The "real" code doesn't return null (the AnyRef is there for the sake of simplification) but a type that is not the same one as the other signature of nvlZ,
(and not a subtype) I'm trying to recreate the situation in my simplified example without having AnyRef as a return type.

but I don't understand why it is relevant, because at the call site of nlZ in this example, the overload resolution
is not ambiguous, since the params can *not* be (a: NonNumericalExpressionZ[Option[A]], b: NonNumericalExpressionZ[A])
since actual params are not of these types and there is no available implicit conv that will make them NonNumericalExpressionZ
so there is no way that the problematic signature of nvlZ can be considered a candidate for overload resolution.
Why I mean to say here is that the "wrong" nvlZ should get excluded and not cause problems, since it isn't a candidate based on the fact
that it's input args.

Now for this part :

  val q = nvlZ(Some(4F), 0)
  //Strange : this compiles :
  computeZ2(q)

  // while this doesn't :
  computeZ2(nvlZ(Some(4F), 0))

I fail to see how the 2 cases are different at the type level, I can totally see how the can differ in the implemetation of the type system
but maybe It's because I'm not knowledgeable enough of the language spec.

2011/4/8 Paul Phillips <paulp@improving.org>

On 4/7/11 6:55 PM, Maxime Lévesque wrote:

  def nvlZ[A,B](a: NumericalExpressionZ[Option[A]], b:
NumericalExpressionZ[B]) = new
NvlFunctionNumericalZ[A,B](a.asInstanceOf[NumericalExpressionZ[A]],b)

  //Strange : commenting this line "fixes" compilation (side effect !?!?!)
  def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b:
NonNumericalExpressionZ[A]) = null

That's a big clue right there.

You're calling an overloaded method and the overloading resolution depends on implicit conversions.  Then to add a little zazz you're making it infer the return types of the overloaded methods.

Commenting out that line fixes it because it removes the overload.  As it is, the inferred return type of that method will be "Null" which is going to get you in trouble because it's more specific.

It compiles like this:

 def nvlZ[A](a: NonNumericalExpressionZ[Option[A]], b: NonNumericalExpressionZ[A]): AnyRef = null

Type inference doesn't perform an exhaustive search of the solution space.  I don't know for sure, but I don't think it's a bug, just something which you used to get away with.

  val q = nvlZ(Some(4F), 0)
  //Strange : this compiles :
  computeZ2(q)

  // while this doesn't :
  computeZ2(nvlZ(Some(4F), 0))

Type inference is performed with an "expected type".  When you assign to a value like that it's easy to see a different result because the expected type of the call to computeZ2 will be more precise.  When it's one big expression with an onion of inferred types and implicit conversions and overload resolution, sometimes the ?s are going to come out the other end.

If you give the second one the same advantage as the first one, namely knowing what the type of q is:

 computeZ2(nvlZ(Some(4F), 0): NvlFunctionNumericalZ[Float,Int])

then it will compile.

Also just FYI your code example is a good distance from being minimized.

On 4/8/11 8:30 AM, Maxime Lévesque wrote:
> I further simplified the thing, please bear with me Paul, I'll
> continue to simplify it, but please let me know if you see
> something.

One aspect of minimizing is not having really long identifiers which are
meaningless in the context of the alleged bug. Having to continually
use my vision parsing component to distinguish "NonNumericalExpressionZ"
vs "NumericalExpressionZ" and "NvlFunctionNumericalZ" and
"NvlFunctionNonNumericalZ" is taxing and makes me a lot less inclined to
get to the interesting part. (I am happy for those among us who see
right through these things.)

On 4/8/11 8:30 AM, Maxime Lévesque wrote:

I further simplified the thing, please bear with me Paul, I'll
continue to simplify it, but please let me know if you see
something.

One aspect of minimizing is not having really long identifiers which are meaningless in the context of the alleged bug.  Having to continually use my vision parsing component to distinguish "NonNumericalExpressionZ" vs "NumericalExpressionZ" and "NvlFunctionNumericalZ" and "NvlFunctionNonNumericalZ" is taxing and makes me a lot less inclined to get to the interesting part.  (I am happy for those among us who see right through these things.)

On 4/8/11 9:40 AM, Maxime Lévesque wrote:
> Paul, I totally understand your point, I changed the naming sheme.
> Whatever I can do to make your life simpler I'll do it, I don't want
> to waste your time, it is far too precious for the Scala community.

Always nice to see such a stellar attitude! I spent much of today trying
to make the voluminous typer debug output into something readable.
It'll be pretty cool when I'm done, but for now let me show you which
implicits it goes looking for in a variety of scenarios, and that'll get
the brain juices flowing.

object Test {
class N[T]
class NE[T] extends N[T]
class NN[T] extends N[T]
class AA[A1,A2]
class BB[A]

implicit def conv1(i: Option[Float]) = new NE[Option[Float]]
implicit def conv2(i: Int) = new NE[Int]
implicit def conv3(op: AA[Byte,Byte]) = new N[Byte]
implicit def conv4(op: AA[Float,Int]) = new N[Float]
implicit def conv5(e: BB[java.util.Date]) = new N[java.util.Date]

def aFunc[A,B](a: NE[Option[A]], b: NE[B]) = new AA[A,B]
def aFunc[A](a: NN[Option[A]], b: NN[A]): AnyRef = new BB[A]
def bFunc[T1](e1: N[T1]) = {}

def f = bFunc(aFunc(Some(4F), 0))
}

// As-is.
//
// infer: Test.BB[?A] => Test.N[?]
// infer: Test.AA[?A,?B] => Test.N[?]
// infer: Some[Float] => Test.NN[Option[?A]]
// infer: => Some[Float] => Test.NN[Option[?A]]
// infer: Some[Float] => Test.NE[Option[?A]]
// infer: Int => Test.NE[?B]
// infer: Test.AA[?A,?B] => Test.N[?]
// [fail]

// Second aFunc commented out.
//
// infer: Some[Float] => Test.NE[Option[?]]
// infer: Int(0) => Test.NE[?]
// infer: Test.AA[Float,Int] => Test.N[?]
// [success]

// Second aFunc annotated with :AnyRef return type.
// infer: AnyRef => Test.N[?]
// infer: => AnyRef => Test.N[?]
// infer: Test.AA[?A,?B] => Test.N[?]
// infer: Some[Float] => Test.NE[Option[?]]
// infer: Int(0) => Test.NE[?]
// infer: Test.AA[Float,Int] => Test.N[?]
// [success]

// conv5 commented out.
//
// infer: Test.BB[?A] => Test.N[?]
// infer: => Test.BB[?A] => Test.N[?]
// infer: Test.AA[?A,?B] => Test.N[?]
// infer: Some[Float] => Test.NE[Option[?]]
// infer: Int(0) => Test.NE[?]
// infer: Test.AA[Float,Int] => Test.N[?]
// [success]

On 4/8/11 5:25 PM, Maxime Lévesque wrote:
> The way I see it, the compiler is contradicting itself :
> 1) the compiler says there is conversion ambiguity because conv3 and
> conv4 applies
> 2) when removing conv4, all of a sudden, no conversion apply, not even
> conv3 which was causing the ambiguity since it apparently was applicable.

The compiler is entitled to not make all possible inferences, so it's
unlikely to be a bug in the official sense. The question is where on
the "desirable enhancement" scale it will fall. Based on the early
guesses it makes, conv3 and conv4 appear ambiguous. That's not an
ironclad representation that either of them will be applicable; errors
arise at many different points during compilation and the earlier places
don't necessarily have any knowledge of the later places.

> Is the reasoning correct ?

Here's what I see happening. It sets out to type this expression:

bFunc(aFunc(Some(4F), 0))

And it doesn't know much. The parameter to bFunc is N[T] for any T, so
it says OK, it's N[?]. And at that point it looks like you are already
in trouble, because the successful inference avenues all seem to involve
it pinning something down up front. This guy:

implicit def conv5(e: BB[java.util.Date]) = new N[java.util.Date]

pushes the unknowns out farther than it presently solves for. Now given
where we see it ending up, it could rule some stuff out and take another
swing at it. But that's not how things are done at present.

BTW, you really need martin or adriaan to comment if you want remotely
reliable information. I'm more using this as an excuse to rewrite some
extremely messy pieces of the typechecker (which has been a lot of fun)
and hoping to stumble upon a fractional increase in understanding
somewhere along the way.

On 4/8/11 5:25 PM, Maxime Lévesque wrote:

The way I see it, the compiler is contradicting itself :
1) the compiler says there is conversion ambiguity because conv3 and
conv4 applies
2) when removing conv4, all of a sudden, no conversion apply, not even
conv3 which was causing the ambiguity since it apparently was applicable.

The compiler is entitled to not make all possible inferences, so it's unlikely to be a bug in the official sense.  The question is where on the "desirable enhancement" scale it will fall. Based on the early guesses it makes, conv3 and conv4 appear ambiguous.

That's not an ironclad representation that either of them will be applicable; errors arise at many different points during compilation and the earlier places don't necessarily have any knowledge of the later places.

Is the reasoning correct ?

Here's what I see happening.  It sets out to type this expression:

 bFunc(aFunc(Some(4F), 0))

And it doesn't know much.  The parameter to bFunc is N[T] for any T, so it says OK, it's N[?].  And at that point it looks like you are already in trouble, because the successful inference avenues all seem to involve it pinning something down up front.  This guy:

 implicit def conv5(e: BB[java.util.Date]) = new N[java.util.Date]

pushes the unknowns out farther than it presently solves for.  Now given where we see it ending up, it could rule some stuff out and take another swing at it. But that's not how things are done at present.

BTW, you really need martin or adriaan to comment if you want remotely reliable information.  I'm more using this as an excuse to rewrite some extremely messy pieces of the typechecker (which has been a lot of fun) and hoping to stumble upon a fractional increase in understanding somewhere along the way.

>>>>> "Maxime" == Maxime Lévesque writes:

Maxime> Also, doesn't it suck that this compiles :
Maxime> val x = aFunc(Some(4F), 0) bFunc(x)
Maxime> while this doesn't :
Maxime> bFunc(aFunc(Some(4F), 0))

Maxime> I often go about decomposing an expression into smaller chunks,
Maxime> working them out, and them recomposing them, I don't remember
Maxime> in 2.8.x having the compiler interpret decompositions or
Maxime> compositions differently at the type level.

Only semi-related, but: here's an example where "val x = a.b; x.c"
compiles but "a.b.c" doesn't (in 2.7, 2.8, and 2.9):

http://stackoverflow.com/questions/5544536/type-inference-on-set-failing/

On 4/9/11 6:23 AM, Maxime Lévesque wrote:
> If it's a "desirable enhancement", then it has been an undocumented
> feature all along in 2.8.x ;-)

You can think of it that way if you like. Everything which isn't
specified is unspecified. You can call a behavior outside the spec a
feature, a bug, a shazbot, a bumblefritz...

> Also, doesn't it suck that this compiles :
>
> val x = aFunc(Some(4F), 0)
> bFunc(x)
>
> while this doesn't :
>
> bFunc(aFunc(Some(4F), 0))

Nobody is suggesting it is a desirable feature that these have different
behaviors. If it's important to you that this not happen, I can propose
a "strict mode" where you have to use type annotations everywhere. You
will get very consistent behavior. Because the available alternative at
the present time is not more type inference, but less.

Also, as seth points out this is not new. There have been expressions
which evaluate differently if broken into subexpressions for as long as
I've been around. If you think about how the collections for instance
work (in that they choose a builder based on the expected type of the
result), this is inevitable.

On 4/9/11 6:23 AM, Maxime Lévesque wrote:

If it's a "desirable enhancement", then it has been an undocumented
feature all along in 2.8.x ;-)

You can think of it that way if you like.  Everything which isn't specified is unspecified.  You can call a behavior outside the spec a feature, a bug, a shazbot, a bumblefritz...

Also, doesn't it suck that this compiles :

 val x = aFunc(Some(4F), 0)
 bFunc(x)

while this doesn't :

 bFunc(aFunc(Some(4F), 0))

Nobody is suggesting it is a desirable feature that these have different behaviors.  If it's important to you that this not happen, I can propose a "strict mode" where you have to use type annotations everywhere.  You will get very consistent behavior.  Because the available alternative at the present time is not more type inference, but less.

Also, as seth points out this is not new.  There have been expressions which evaluate differently if broken into subexpressions for as long as I've been around.  If you think about how the collections for instance work (in that they choose a builder based on the expected type of the result), this is inevitable.

>> Also, doesn't it suck that this compiles :
>>
>>  val x = aFunc(Some(4F), 0)
>>  bFunc(x)
>>
>> while this doesn't :
>>
>>  bFunc(aFunc(Some(4F), 0))
>
> Nobody is suggesting it is a desirable feature that these have different
> behaviors.  If it's important to you that this not happen, I can propose a
> "strict mode" where you have to use type annotations everywhere.  You will
> get very consistent behavior.  Because the available alternative at the
> present time is not more type inference, but less.
>
> Also, as seth points out this is not new.  There have been expressions which
> evaluate differently if broken into subexpressions for as long as I've been
> around.  If you think about how the collections for instance work (in that
> they choose a builder based on the expected type of the result), this is
> inevitable.

I hit a similar case yesterday with 2.8.1 and spent an hour or two
trying to work around it. In the end of course there were multiple
possible workarounds: explicit result types, intermediate variables,
explicit conversions, but as Maxime said it "sucks" ;)

Now this is complicated stuff to get right, but what I am wondering is
whether type inference can be improved in such cases, at least to a
degree, of if there are fundamental reasons that it couldn't (besides
the man-hours needed to work on it).

-Erik

i agree with you. but i have seen this already in 2.8.0. or 2.8.1. something like

call( a, b, x( y ))

doesn't work, but

call( a, b, { val c = x( y ); c })

does work. i can hardly see this as a "feature" of the compiler.

best, -sciss-

On 9 Apr 2011, at 14:23, Maxime Lévesque wrote:

> Also, doesn't it suck that this compiles :
>
> val x = aFunc(Some(4F), 0)
> bFunc(x)
>
> while this doesn't :
>
> bFunc(aFunc(Some(4F), 0))
>
> ?
>
> I'm *very* surprised that these two expressions differ at the type level.

i've seen something similar, just the other way round (which makes
perfectly sense)
list.filter(e => e+1) // type if e inferred

val aFunc = e => e+1
list.filter(aFunc) // whoopsy

might type inference mess up something in your example?

On 9 Apr 2011, at 14:23, Maxime Lévesque wrote:
>> Also, doesn't it suck that this compiles :
>>
>> val x = aFunc(Some(4F), 0)
>> bFunc(x)
>>
>> while this doesn't :
>>
>> bFunc(aFunc(Some(4F), 0))
>>
>> ?
>>
>> I'm *very* surprised that these two expressions differ at the type level.
>