Re: Riddle -- how can contravariant types be specialized at all?

  private class Node( var value: A ) {     def compare( b: Node ) : Int = compareFun( value, b.value )
  }
class Node is not specialized. You would have to do: class Node[@specialized(Int) B <: A]( var value: B ) or something along those lines.


On Mon, Nov 7, 2011 at 5:16 PM, Sciss <contact@sciss.de> wrote:

(was missing the scala-user CC)

oh.... so it did _not_ work?

here is another example that i thought should work:

final case class Compare[ @specialized( Int ) A ]( fun: (A, A) => Int ) { def apply( a: A, b: A ) : Int = fun( a, b )}

trait SkipList[ @specialized( Int ) A ] {
 def compareFun: Compare[ A ]
 private class Node( var value: A ) {
    def compare( b: Node ) : Int = compareFun( value, b.value )
 }

 def test( a: A, b: A ) : Int = {
    val n = new Node( a )
    val m = new Node( b )
    n compare m
 }
}

object SkipList { def apply[ A ]( _compareFun: Compare[ A ]) : SkipList[ A ] = new SkipList[ A ] { def compareFun = _compareFun }}

val l = SkipList( Compare[ Int ] { (a, b) => sys.error( "here" )})

l.test( 1, 2 )

java.lang.RuntimeException: here
       at scala.sys.package$.error(package.scala:27)
       at $anonfun$1.apply(<console>:11)
       at $anonfun$1.apply(<console>:11)
       at scala.Function2$class.apply$mcIII$sp(Function2.scala:33)
       at scala.runtime.AbstractFunction2.apply$mcIII$sp(AbstractFunction2.scala:12)
       at Compare$mcI$sp.apply$mcI$sp(<console>:7)
       at Compare$mcI$sp.apply(<console>:7)
       at Compare$mcI$sp.apply(<console>:7)
       at SkipList$Node.compare(<console>:12)
       at SkipList$class.test(<console>:18)
       at SkipList$$anon$1.test(<console>:10)
       at SkipList$class.test$mcI$sp(<console>:15)
       at SkipList$$anon$1.test$mcI$sp(<console>:10)
       at .<init>(<console>:13)
...

again here is your pattern:

       at Compare$mcI$sp.apply$mcI$sp(<console>:7)
       at Compare$mcI$sp.apply(<console>:7)
       at Compare$mcI$sp.apply(<console>:7)

however not in the actual function call  (scala.runtime.AbstractFunction2.apply$mcIII$sp)

but Node is an _inner_ class of a type which is specialized in A. i really cannot add the type parameter again, that would mean i rewrite 90% of my code, plus it seriously damaged readability.

On 7 Nov 2011, at 23:28, Aleksey Nikiforov wrote:

> private class Node( var value: A ) {
> def compare( b: Node ) : Int = compareFun( value, b.value )
> }
>
> class Node is not specialized. You would have to do: class Node[@specialized(Int) B <: A]( var value: B ) or something along those lines.
>
>
> On Mon, Nov 7, 2011 at 5:16 PM, Sciss wrote:
> (was missing the scala-user CC)
>
> oh.... so it did _not_ work?
>
> here is another example that i thought should work:
>
> final case class Compare[ @specialized( Int ) A ]( fun: (A, A) => Int ) { def apply( a: A, b: A ) : Int = fun( a, b )}
>
> trait SkipList[ @specialized( Int ) A ] {
> def compareFun: Compare[ A ]
> private class Node( var value: A ) {
> def compare( b: Node ) : Int = compareFun( value, b.value )
> }
>
> def test( a: A, b: A ) : Int = {
> val n = new Node( a )
> val m = new Node( b )
> n compare m
> }
> }
>
> object SkipList { def apply[ A ]( _compareFun: Compare[ A ]) : SkipList[ A ] = new SkipList[ A ] { def compareFun = _compareFun }}
>
> val l = SkipList( Compare[ Int ] { (a, b) => sys.error( "here" )})
>
> l.test( 1, 2 )
>
> java.lang.RuntimeException: here
> at scala.sys.package$.error(package.scala:27)
> at $anonfun$1.apply(:11)
> at $anonfun$1.apply(:11)
> at scala.Function2$class.apply$mcIII$sp(Function2.scala:33)
> at scala.runtime.AbstractFunction2.apply$mcIII$sp(AbstractFunction2.scala:12)
> at Compare$mcI$sp.apply$mcI$sp(:7)
> at Compare$mcI$sp.apply(:7)
> at Compare$mcI$sp.apply(:7)
> at SkipList$Node.compare(:12)
> at SkipList$class.test(:18)
> at SkipList$$anon$1.test(:10)
> at SkipList$class.test$mcI$sp(:15)
> at SkipList$$anon$1.test$mcI$sp(:10)
> at .(:13)
> ...
>
> again here is your pattern:
>
> at Compare$mcI$sp.apply$mcI$sp(:7)
> at Compare$mcI$sp.apply(:7)
> at Compare$mcI$sp.apply(:7)
>
> however not in the actual function call (scala.runtime.AbstractFunction2.apply$mcIII$sp)
>

Specialization works by generating every combination of primitive types. This can cause an explosion of type signatures. Scala takes a very conservative approach and makes it an opt-in feature rather than opt-out. As a result you have to annotate every class and every type parameter you want specialized.

Try compiling this one-liner to get the idea how crazy specialization can get:
class S[@specialized T0, @specialized T1, @specialized T2, @specialized T3](v0: T0, v1: T1, v2: T2, v3: T3)
It will take a while, generate 6564 classes, and consume 5.4 Mb of disk space. All from one line of code.

On Mon, Nov 7, 2011 at 5:31 PM, Sciss <contact@sciss.de> wrote:

but Node is an _inner_ class of a type which is specialized in A. i really cannot add the type parameter again, that would mean i rewrite 90% of my code, plus it seriously damaged readability.

i understand. but if i have

trait X[ @specialized( Int ) A ] {
private class Node {
var payload: A
}
}

i don't get why that wouldn't be specialized. this is by no means creating any sort of explosion, because Node as an inner class is inherently tied to X, so the maximum number of classes equals the number of specializations in A, not to the power of two.

this just shows for me that type parameters and path-dependent types are second-class citizens in scala, and i find that very implausible.

best, -sciss-

On 8 Nov 2011, at 00:15, Aleksey Nikiforov wrote:

> Specialization works by generating every combination of primitive types. This can cause an explosion of type signatures. Scala takes a very conservative approach and makes it an opt-in feature rather than opt-out. As a result you have to annotate every class and every type parameter you want specialized.
>
> Try compiling this one-liner to get the idea how crazy specialization can get:
>
> class S[@specialized T0, @specialized T1, @specialized T2, @specialized T3](v0: T0, v1: T1, v2: T2, v3: T3)
>
> It will take a while, generate 6564 classes, and consume 5.4 Mb of disk space. All from one line of code.
>
>
> On Mon, Nov 7, 2011 at 5:31 PM, Sciss wrote:
> but Node is an _inner_ class of a type which is specialized in A. i really cannot add the type parameter again, that would mean i rewrite 90% of my code, plus it seriously damaged readability.
>
>

abstract type _members_ i wanted to say.

trait X {
type Y
}

versus

trait X[ Y ]

On 8 Nov 2011, at 00:43, Sciss wrote:

> this just shows for me that type parameters and path-dependent types are second-class citizens in scala, and i find that very implausible.

Abstract types and inner classes are different concepts. So I am not quite sure what abstract and path-dependent types have to do with specialization.
As for inner classes, each inner class becomes a java class file after compilation. If you Node would keep specialization of the parent class, it would effectively double the total number of classes, provided it uses all the specialized types. Now, if you have 10 inner classes, you get 10 times more classes.
Specialization works well when you have a limited number of specialized types, otherwise the number of combinations can explode very quickly. So specialization is made explicit, to make sure you know what you are doing, and the very least prevent accidental explosion in generated number of classes. Also this approach makes it easy to track down the causes by simply searching for @specialized annotation.
Overall, given how unpredictable specialization results can be, I think it was the right decision to make it so explicit. I do not believe it has anything to do with second-class citizens, but more to do with potential problems with specialization when used without caution.


On Mon, Nov 7, 2011 at 6:45 PM, Sciss <contact@sciss.de> wrote:

abstract type _members_ i wanted to say.

trait X {
  type Y
}

versus

trait X[ Y ]


On 8 Nov 2011, at 00:43, Sciss wrote:

> this just shows for me that type parameters and path-dependent types are second-class citizens in scala, and i find that very implausible.

i disagree with judging this situation as explosion. that for me implies exponential behaviour. as if you'd have 4 * 4 = 16 combinations for an inner and an outer class specializing in four types, where you would only need 4 + 4. so the number of classes generated is linear. wouldn't it be possible at least to get proper specialization like this:

trait SkipList[ @specialized( Int, Long) A ] {
@specialized private class Node {
var value A
...
}
}

the idea of specialization for me is to get maximum performance, that's the only reason why you do the effort to put the annotations. so if get a half-way specialized result with numerous boxing, unboxing, that is clearly unintended and counter-intuitive. the compiler should at least tell me

Warning: inner class Node is not specialized in A.
enforce specialization by annotating this class as @specialized

or similar

?

The current behaviour just means Scala is unsuitable for creating elegant high-performance data structures. When i start now to move around my classes and annotate methods and so forth to eventually end up in a fully specialized data structure -- which seems almost impossible to observe --, probably the result would be more readable if i go back to Java. I'm very disappointed by this.

best, -sciss-

On 8 Nov 2011, at 04:58, Aleksey Nikiforov wrote:

> Abstract types and inner classes are different concepts. So I am not quite sure what abstract and path-dependent types have to do with specialization.
>
> As for inner classes, each inner class becomes a java class file after compilation. If you Node would keep specialization of the parent class, it would effectively double the total number of classes, provided it uses all the specialized types. Now, if you have 10 inner classes, you get 10 times more classes.
>
> Specialization works well when you have a limited number of specialized types, otherwise the number of combinations can explode very quickly. So specialization is made explicit, to make sure you know what you are doing, and the very least prevent accidental explosion in generated number of classes. Also this approach makes it easy to track down the causes by simply searching for @specialized annotation.
>
> Overall, given how unpredictable specialization results can be, I think it was the right decision to make it so explicit. I do not believe it has anything to do with second-class citizens, but more to do with potential problems with specialization when used without caution.
>
>
> On Mon, Nov 7, 2011 at 6:45 PM, Sciss wrote:
> abstract type _members_ i wanted to say.
>
> trait X {
> type Y
> }
>
> versus
>
> trait X[ Y ]
>
>
> On 8 Nov 2011, at 00:43, Sciss wrote:
>
> > this just shows for me that type parameters and path-dependent types are second-class citizens in scala, and i find that very implausible.
>
>

On 8 November 2011 15:59, Sciss <contact@sciss.de> wrote:

The current behaviour just means Scala is unsuitable for creating elegant high-performance data structures.

Unfortunately, I came to the same conclusion for my applications. @specialize is currently too buggy, not calling specialized calls in several important cases. The compiler doesn't help you track down issues where non-specialized things are called from specialized code and vice-versa - you end up crawling through (decompiled) bytecode and multi-megabyte compiler logs. It also doesn't give you fine-grained control over what combinations are specialized for - it expands the cross-product of the type parameters, but often you only want the 'same type' cases specialized. Specialization and derived classes/traits seem to not play at all well with each other. None of this can be fixed without some resources being thrown at it, but the pool of people who understand the specialization plugin is small.  

When i start now to move around my classes and annotate methods and so forth to eventually end up in a fully specialized data structure -- which seems almost impossible to observe --, probably the result would be more readable if i go back to Java. I'm very disappointed by this.

For now, when I find that boxing is the bottleneck and @specialize doesn't work, I end up hand-rolling an implementation for the concrete primitives that I need for that application. It's far from ideal.
I'd be happy to work with someone to fix the specialization plugin, but I'd need mentoring, and I expect the pool of available mentors is even smaller than those who understand the plugin to begin with.
Matthew 

best, -sciss-

Matthew 

--
Dr Matthew PocockIntegrative Bioinformatics Group, School of Computing Science, Newcastle Universitymailto: turingatemyhamster@gmail.com gchat: turingatemyhamster@gmail.commsn: matthew_pocock@yahoo.co.uk irc.freenode.net: drdozerskype: matthew.pococktel: (0191) 2566550mob: +447535664143

On Tue, Nov 8, 2011 at 10:59 AM, Sciss <contact@sciss.de> wrote:

The current behaviour just means Scala is unsuitable for creating elegant high-performance data structures. When i start now to move around my classes and annotate methods and so forth to eventually end up in a fully specialized data structure -- which seems almost impossible to observe --, probably the result would be more readable if i go back to Java. I'm very disappointed by this.

I sympathize.  I've found the same thing.  Specialization was a good idea, but the current implementation falls on its face in so many cases that it's almost unusable for anything with nontrivial structure.  Also, the bugs do not seem to be getting fixed very quickly.  Still, if you find a new bug that isn't in JIRA, it would be great if you could add it for the benefit of the rest of us who have tried to use specialization.

I write code generators instead, most of the time.

I think I may have managed to write a specialized mutable tuple library using specialization without it completely breaking, but my efforts to create a specialized mutable collection library have thus far been pretty unsuccessful.

  --Rex
 

another 'solution' is to copy the source code, use the very well working refactoring tool of IDEA to translate the type parameter 'A' to 'Int' and 'Long', and then rename SkipListImpl[Int] to IntSkipListImpl.

but that doesn't really scale, because once you need to change code afterwards, you need to change it three times.

i wonder if there is maybe a capable source-rewriting tool around that can do this refactoring automatically, like a nice little sbt plugin... ?

best, -sciss-

On 8 Nov 2011, at 16:24, Matthew Pocock wrote:

>
>
> On 8 November 2011 15:59, Sciss wrote:
>
> The current behaviour just means Scala is unsuitable for creating elegant high-performance data structures.
>
> Unfortunately, I came to the same conclusion for my applications. @specialize is currently too buggy, not calling specialized calls in several important cases. The compiler doesn't help you track down issues where non-specialized things are called from specialized code and vice-versa - you end up crawling through (decompiled) bytecode and multi-megabyte compiler logs. It also doesn't give you fine-grained control over what combinations are specialized for - it expands the cross-product of the type parameters, but often you only want the 'same type' cases specialized. Specialization and derived classes/traits seem to not play at all well with each other. None of this can be fixed without some resources being thrown at it, but the pool of people who understand the specialization plugin is small.
>
> When i start now to move around my classes and annotate methods and so forth to eventually end up in a fully specialized data structure -- which seems almost impossible to observe --, probably the result would be more readable if i go back to Java. I'm very disappointed by this.
>
>
> For now, when I find that boxing is the bottleneck and @specialize doesn't work, I end up hand-rolling an implementation for the concrete primitives that I need for that application. It's far from ideal.
>
> I'd be happy to work with someone to fix the specialization plugin, but I'd need mentoring, and I expect the pool of available mentors is even smaller than those who understand the plugin to begin with.
>
> Matthew
>
> best, -sciss-
>
>
> Matthew
>
>

tada..... that is what i was thinking now. can you recommend an approach here that i could use?

best, -sciss-

On 8 Nov 2011, at 16:33, Rex Kerr wrote:

> I write code generators instead, most of the time.

For some code I just use search-and-replace (generally using $x$ or $$x$$ as the target pattern); just write the code in strings (and include the compile-and-pipe-to-output as an initial step for compiling the code).  For example,

  lowup.zipWithIndex.map { case ((l,u),i) =>
    "trait Oople%s[@specialized %s] extends Muple%s[%s] with Oople {\n".format(u,u,u,u) +
    "  def op%s = if (ok) Some(%s) else None\n".format(u,l) +
    "  def map%s(fn: %s => %s) = { if (ok) %s = fn(%s); this }\n".format(u,u,u,l,l) +
    "}"
  }

produces a bunch of traits from other traits.  (In this case, specialization seems to work, but I generated code for different numbers of arguments; if you *just* want to use it to replace specialization, this approach is usually good enough, I've found.)

For cases with more complex structure, I typically create a class hierarchy representing the types I want to specialize and manipulations between.  For example,

  object GenericType extends RType("G","Generic") {
    override def defmods = "abstract "; def supertype = None
  }
  object ImmutableType extends RType("I","Immutable") {
    override def defmods = "final case "; def supertype = Some(GenericType)
  }
  object AssignableType extends RType("A","Assignable") {
    override def mutates = true; override def defmods = "abstract "
    def supertype = Some(GenericType)
  }
  object MutableType extends RType("M","Mutable") {
    override def mutates = true; override def defmods = "final "
    def supertype = Some(AssignableType)
  }
  object BackedType extends RType("B","Backed") {
    override def mutates = true; def supertype = Some(AssignableType)
  }
  val reps = List(GenericType, ImmutableType, AssignableType, MutableType, BackedType)
  val canon = reps.map(r =>
    r -> (if (r==GenericType || r==ImmutableType) ImmutableType else MutableType)
  ).toMap

is part of a system to represent the inheritance hierarchy that I want to automatically generate.

So far I haven't really needed a powerful generic system for code generation; Scala is sufficiently powerful so I can just come up with an ad-hoc solution each time.

I am hoping that macro facilities will obviate the need for some of these things.

  --Rex


On Tue, Nov 8, 2011 at 12:10 PM, Sciss <contact@sciss.de> wrote:

tada..... that is what i was thinking now. can you recommend an approach here that i could use?

best, -sciss-



On 8 Nov 2011, at 16:33, Rex Kerr wrote:

> I write code generators instead, most of the time.

of course, i invested energy again to remove the inner classes in favour of outer classes with type parameters so specialization kicks in again. i should have known:

[info] Compiling 1 Scala source to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
[error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
[error] found : de.sciss.collection.txn.HASkipList.Node[S(in class Leaf),A]
[error] required: de.sciss.collection.txn.HASkipList.Child[S(in class Leaf$mcI$sp),Int]
[error] Total time: 1 s, completed Nov 15, 2011 2:28:05 PM

are there any tricks to get more information from this -- like which is the line that crashes the compiler?

best, -sciss-

On 8 Nov 2011, at 16:33, Rex Kerr wrote:

>
> On Tue, Nov 8, 2011 at 10:59 AM, Sciss wrote:
>
> The current behaviour just means Scala is unsuitable for creating elegant high-performance data structures. When i start now to move around my classes and annotate methods and so forth to eventually end up in a fully specialized data structure -- which seems almost impossible to observe --, probably the result would be more readable if i go back to Java. I'm very disappointed by this.
>
> I sympathize. I've found the same thing. Specialization was a good idea, but the current implementation falls on its face in so many cases that it's almost unusable for anything with nontrivial structure. Also, the bugs do not seem to be getting fixed very quickly. Still, if you find a new bug that isn't in JIRA, it would be great if you could add it for the benefit of the rest of us who have tried to use specialization.
>
> I write code generators instead, most of the time.
>
> I think I may have managed to write a specialized mutable tuple library using specialization without it completely breaking, but my efforts to create a specialized mutable collection library have thus far been pretty unsuccessful.
>
> --Rex
>

I was told to use  -Ylog:icode -Ydebug to find where compiler has crashed, but depending at what stage it crashed you may not get any useful info. The only 100% reliable method is removing parts of your project until it start compiling again. Rinse and repeat with the last removed part until you narrow down the cause.

You could also try scala nightly. I have encountered several compiler crashes recently, and most of them were fixed in unrleased 2.10.


On Tue, Nov 15, 2011 at 8:33 AM, Sciss <contact@sciss.de> wrote:

of course, i invested energy again to remove the inner classes in favour of outer classes with type parameters so specialization kicks in again. i should have known:

[info] Compiling 1 Scala source to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
[error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
[error]  found   : de.sciss.collection.txn.HASkipList.Node[S(in class Leaf),A]
[error]  required: de.sciss.collection.txn.HASkipList.Child[S(in class Leaf$mcI$sp),Int]
[error] Total time: 1 s, completed Nov 15, 2011 2:28:05 PM

are there any tricks to get more information from this -- like which is the line that crashes the compiler?

best, -sciss-

ok, the first thing doesn't give me any additional info:

[info] Compiling 47 Scala sources and 11 Java sources to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
[info] [running phase parser on 58 compilation units]
[info] [running phase namer on 58 compilation units]
[info] [running phase packageobjects on 58 compilation units]
[info] [running phase typer on 58 compilation units]
[info] [running phase superaccessors on 58 compilation units]
[info] [running phase pickler on 58 compilation units]
[info] [running phase refchecks on 58 compilation units]
[info] [running phase liftcode on 58 compilation units]
[info] [running phase uncurry on 58 compilation units]
[info] [running phase tailcalls on 58 compilation units]
[info] [running phase specialize on 58 compilation units]
[error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
[error] found : HASkipList.this.Node[S(in class Leaf),A]
[error] required: HASkipList.this.Child[S(in class Leaf$mcI$sp),scala.this.Int]
[error] Total time: 9 s, completed Nov 15, 2011 3:03:34 PM

i will see if i can build with 2.10, but i have various dependencies, so probably not. might need to add -no-specialization and screw this wasted attempt...

On 15 Nov 2011, at 15:01, Aleksey Nikiforov wrote:

> I was told to use -Ylog:icode -Ydebug to find where compiler has crashed, but depending at what stage it crashed you may not get any useful info. The only 100% reliable method is removing parts of your project until it start compiling again. Rinse and repeat with the last removed part until you narrow down the cause.
>
> You could also try scala nightly. I have encountered several compiler crashes recently, and most of them were fixed in unrleased 2.10.
>
>
> On Tue, Nov 15, 2011 at 8:33 AM, Sciss wrote:
> of course, i invested energy again to remove the inner classes in favour of outer classes with type parameters so specialization kicks in again. i should have known:
>
> [info] Compiling 1 Scala source to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
> [error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
> [error] found : de.sciss.collection.txn.HASkipList.Node[S(in class Leaf),A]
> [error] required: de.sciss.collection.txn.HASkipList.Child[S(in class Leaf$mcI$sp),Int]
> [error] Total time: 1 s, completed Nov 15, 2011 2:28:05 PM
>
> are there any tricks to get more information from this -- like which is the line that crashes the compiler?
>
> best, -sciss-
>

I tend to lose my cool when encountering rough edges in Scala, espectially if the problem involves a compiler crash. However, I have figured that Scala makes me about twice as productive when compared to Java (a factor of 2.0). So if I spend as much as 10% of my overall time chasing down and working around bugs in the Scala itself, then my overall efficiency is: 0.9*2.0 = 1.8. Still 80% higher than using Java.

That said, you should try to track down the problem and file the bug report. As a bonus, once you know the problem you may be able to work around it.


On Tue, Nov 15, 2011 at 9:05 AM, Sciss <contact@sciss.de> wrote:

ok, the first thing doesn't give me any additional info:

[info] Compiling 47 Scala sources and 11 Java sources to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
[info] [running phase parser on 58 compilation units]
[info] [running phase namer on 58 compilation units]
[info] [running phase packageobjects on 58 compilation units]
[info] [running phase typer on 58 compilation units]
[info] [running phase superaccessors on 58 compilation units]
[info] [running phase pickler on 58 compilation units]
[info] [running phase refchecks on 58 compilation units]
[info] [running phase liftcode on 58 compilation units]
[info] [running phase uncurry on 58 compilation units]
[info] [running phase tailcalls on 58 compilation units]
[info] [running phase specialize on 58 compilation units]
[error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
[error]  found   : HASkipList.this.Node[S(in class Leaf),A]
[error]  required: HASkipList.this.Child[S(in class Leaf$mcI$sp),scala.this.Int]
[error] Total time: 9 s, completed Nov 15, 2011 3:03:34 PM


i will see if i can build with 2.10, but i have various dependencies, so probably not. might need to add -no-specialization and screw this wasted attempt...

ok, i was able to 'avoid' the bug. it seems to have been a problem with specialization and pattern matching. before i had:

@tailrec def step( num: Int, n: Node[ S, A ]) : Int = n match {
case Leaf( _ ) => num
case Branch( b ) => step( num + 1, b.down( 0 ))
}

with my extractor objects like this to avoid warnings due to type erasure:

object Node {
def unapply[ S <: Sys[ S ], @specialized( Int ) A ]( child: Child[ S, A ]) : Option[ Node[ S, A ]] = child.nodeOption
}

so this was somehow crashing the compiler/specialization. how if i go back to crappy warnings:

@tailrec def step( num: Int, n: Node[ S, A ]) : Int = n match {
case _: Leaf[ _, _ ] => num
case b: Branch[ S, A ] => step( num + 1, b.down( 0 ))
}

i get a strange warning:

warn] /Users/hhrutz/Documents/devel/TreeTests/src/main/scala/de/sciss/collection/txn/HASkipList.scala:155: non variable type-argument S in type pattern de.sciss.collection.txn.HASkipList.Branch[S,A] is unchecked since it is eliminated by erasure
[warn] case b: Branch[ S, A ] => step( num + 1, b.down( 0 ))
[warn] ^
...
[warn] missing combination Leaf
[warn] missing combination Leaf$mcI$sp
[warn] @tailrec def step( num: Int, n: Node[ S, A ]) : Int = n match {
[warn] ^

a guess this latter warning is a bug with the interaction between pattern matching and specialization?

also i'm curious why i only get a warning about the erasure of `S` (my list's transactional system), but not because of `A` (my list's element type, specialized)

best, -sciss-

On 15 Nov 2011, at 15:05, Sciss wrote:

> ok, the first thing doesn't give me any additional info:
>
> [info] Compiling 47 Scala sources and 11 Java sources to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
> [info] [running phase parser on 58 compilation units]
> [info] [running phase namer on 58 compilation units]
> [info] [running phase packageobjects on 58 compilation units]
> [info] [running phase typer on 58 compilation units]
> [info] [running phase superaccessors on 58 compilation units]
> [info] [running phase pickler on 58 compilation units]
> [info] [running phase refchecks on 58 compilation units]
> [info] [running phase liftcode on 58 compilation units]
> [info] [running phase uncurry on 58 compilation units]
> [info] [running phase tailcalls on 58 compilation units]
> [info] [running phase specialize on 58 compilation units]
> [error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
> [error] found : HASkipList.this.Node[S(in class Leaf),A]
> [error] required: HASkipList.this.Child[S(in class Leaf$mcI$sp),scala.this.Int]
> [error] Total time: 9 s, completed Nov 15, 2011 3:03:34 PM
>
>
> i will see if i can build with 2.10, but i have various dependencies, so probably not. might need to add -no-specialization and screw this wasted attempt...
>
>
> On 15 Nov 2011, at 15:01, Aleksey Nikiforov wrote:
>
>> I was told to use -Ylog:icode -Ydebug to find where compiler has crashed, but depending at what stage it crashed you may not get any useful info. The only 100% reliable method is removing parts of your project until it start compiling again. Rinse and repeat with the last removed part until you narrow down the cause.
>>
>> You could also try scala nightly. I have encountered several compiler crashes recently, and most of them were fixed in unrleased 2.10.
>>
>>
>> On Tue, Nov 15, 2011 at 8:33 AM, Sciss wrote:
>> of course, i invested energy again to remove the inner classes in favour of outer classes with type parameters so specialization kicks in again. i should have known:
>>
>> [info] Compiling 1 Scala source to /Users/hhrutz/Documents/devel/TreeTests/target/scala-2.9.1/classes...
>> [error] {file:/Users/hhrutz/Documents/devel/TreeTests/}default-92482a/compile:compile: scala.tools.nsc.symtab.Types$TypeError: type mismatch;
>> [error] found : de.sciss.collection.txn.HASkipList.Node[S(in class Leaf),A]
>> [error] required: de.sciss.collection.txn.HASkipList.Child[S(in class Leaf$mcI$sp),Int]
>> [error] Total time: 1 s, completed Nov 15, 2011 2:28:05 PM
>>
>> are there any tricks to get more information from this -- like which is the line that crashes the compiler?
>>
>> best, -sciss-
>>
>

specialization is like an invitation to shoot yourself in the foot

class Test1[ @specialized( Int ) A ] {
def test( a: A ) {
sys.error( "here" )
}
}

val t1 = new Test1[ Int ]
t1.test(0)
// ok:
// java.lang.RuntimeException: here
// at scala.sys.package$.error(package.scala:27)
// at Test1$mcI$sp.test$mcI$sp(:34)

abstract class Test2[ @specialized( Int ) A ] {
def test( a: A ) {
sys.error( "here" )
}
}

def ano[ @specialized( Int ) A ] : Test2[ A ] = new Test2[ A ] {}

val t2 = ano[ Int ]
t2.test( 1 )
// not ok:
// java.lang.RuntimeException: here
// at scala.sys.package$.error(package.scala:27)
// at Test2.test(:34)
// at Test2.test$mcI$sp(:33)

:-(

best, -sciss-

On 7 Nov 2011, at 23:04, Aleksey Nikiforov wrote:

> Seem you have posted more while I was typing the response. When you see something like this in the stack trace, it means specialization did not work out:
>
> at KeyTest$mcI$sp.up$mcI$sp(:8)
> at KeyTest$mcI$sp.up(:8)
> at KeyTest$mcI$sp.up(:8)
>
> When specialization works you will see only one specialized call:
>
> at KeyTest$mcI$sp.up$mcI$sp(:8)
>
>

Is that one not in the bug tracker?  Better add it, at least to have it throw out the same "can't inherit specialization" warning that it does when you try
  class Test3[@specialized(Int) A] extends Test2[A] {}
(which is really what's going on here).

  --Rex

On Tue, Nov 15, 2011 at 12:18 PM, Sciss <contact@sciss.de> wrote:

specialization is like  an invitation to shoot yourself in the foot

class Test1[ @specialized( Int ) A ] {
  def test( a: A ) {
     sys.error( "here" )
  }
}

val t1 = new Test1[ Int ]
t1.test(0)
// ok:
// java.lang.RuntimeException: here
//      at scala.sys.package$.error(package.scala:27)
//      at Test1$mcI$sp.test$mcI$sp(<console>:34)

abstract class Test2[ @specialized( Int ) A ] {
  def test( a: A ) {
     sys.error( "here" )
  }
}


def ano[ @specialized( Int ) A ] : Test2[ A ] = new Test2[ A ] {}

val t2 = ano[ Int ]
t2.test( 1 )
// not ok:
// java.lang.RuntimeException: here
//      at scala.sys.package$.error(package.scala:27)
//      at Test2.test(<console>:34)
//      at Test2.test$mcI$sp(<console>:33)


:-(

best, -sciss-


On 7 Nov 2011, at 23:04, Aleksey Nikiforov wrote:

> Seem you have posted more while I was typing the response. When you see something like this in the stack trace, it means specialization did not work out:
>
>        at KeyTest$mcI$sp.up$mcI$sp(<console>:8)
>        at KeyTest$mcI$sp.up(<console>:8)
>        at KeyTest$mcI$sp.up(<console>:8)
>
> When specialization works you will see only one specialized call:
>
>        at KeyTest$mcI$sp.up$mcI$sp(<console>:8)
>
>

btw, this seems independent of whether an abstract class or a trait is extended:

trait Test4[ @specialized( Int ) A ] {
def test( a: A ) {
sys.error( "here" )
}
}

def ano2[ @specialized( Int ) A ] : Test4[ A ] = new Test4[ A ] {}
val t4 = ano2[ Int ]
t4.test(0) // OK!

class Test5[ @specialized( Int ) A ] extends Test4[ A ]
val t5 = new Test5[ Int ]
t5.test(0) // Not OK!

On 15 Nov 2011, at 17:36, Rex Kerr wrote:

> Is that one not in the bug tracker? Better add it, at least to have it throw out the same "can't inherit specialization" warning that it does when you try
> class Test3[@specialized(Int) A] extends Test2[A] {}
> (which is really what's going on here).
>
> --Rex
>
> On Tue, Nov 15, 2011 at 12:18 PM, Sciss wrote:
> specialization is like an invitation to shoot yourself in the foot
>
> class Test1[ @specialized( Int ) A ] {
> def test( a: A ) {
> sys.error( "here" )
> }
> }
>
> val t1 = new Test1[ Int ]
> t1.test(0)
> // ok:
> // java.lang.RuntimeException: here
> // at scala.sys.package$.error(package.scala:27)
> // at Test1$mcI$sp.test$mcI$sp(:34)
>
> abstract class Test2[ @specialized( Int ) A ] {
> def test( a: A ) {
> sys.error( "here" )
> }
> }
>
>
> def ano[ @specialized( Int ) A ] : Test2[ A ] = new Test2[ A ] {}
>
> val t2 = ano[ Int ]
> t2.test( 1 )
> // not ok:
> // java.lang.RuntimeException: here
> // at scala.sys.package$.error(package.scala:27)
> // at Test2.test(:34)
> // at Test2.test$mcI$sp(:33)
>
>
> :-(
>
> best, -sciss-
>
>
> On 7 Nov 2011, at 23:04, Aleksey Nikiforov wrote:
>
> > Seem you have posted more while I was typing the response. When you see something like this in the stack trace, it means specialization did not work out:
> >
> > at KeyTest$mcI$sp.up$mcI$sp(:8)
> > at KeyTest$mcI$sp.up(:8)
> > at KeyTest$mcI$sp.up(:8)
> >
> > When specialization works you will see only one specialized call:
> >
> > at KeyTest$mcI$sp.up$mcI$sp(:8)
> >
> >
>
>

Everything worked as expected.

// at scala.sys.package$.error(package.scala:27) <-- Your error call happens last
//      at Test2.test(<console>:34) <- Generic test call happens just before the error call.
//      at Test2.test$mcI$sp(<console>:33) <- Specialized call happens, first, but it delegates to generic implementation.

So specialized call happened first, means it worked. The big question is: can you really do anything with "a: A" at the abstract level without avoiding boxing? I mean, you dont know what A will be, so there is no way to specialize around that. The only thing you can do is override test(), and provide a specialized implementation. But in that case the specialized call will delegate to your override instead of the generic implementation of the abstract class.


On Tue, Nov 15, 2011 at 11:18 AM, Sciss <contact@sciss.de> wrote:

specialization is like  an invitation to shoot yourself in the foot

class Test1[ @specialized( Int ) A ] {
  def test( a: A ) {
     sys.error( "here" )
  }
}

val t1 = new Test1[ Int ]
t1.test(0)
// ok:
// java.lang.RuntimeException: here
//      at scala.sys.package$.error(package.scala:27)
//      at Test1$mcI$sp.test$mcI$sp(<console>:34)

abstract class Test2[ @specialized( Int ) A ] {
  def test( a: A ) {
     sys.error( "here" )
  }
}


def ano[ @specialized( Int ) A ] : Test2[ A ] = new Test2[ A ] {}

val t2 = ano[ Int ]
t2.test( 1 )
// not ok:
// java.lang.RuntimeException: here
//      at scala.sys.package$.error(package.scala:27)
//      at Test2.test(<console>:34)
//      at Test2.test$mcI$sp(<console>:33)


:-(

best, -sciss-


On 7 Nov 2011, at 23:04, Aleksey Nikiforov wrote:

> Seem you have posted more while I was typing the response. When you see something like this in the stack trace, it means specialization did not work out:
>
>        at KeyTest$mcI$sp.up$mcI$sp(<console>:8)
>        at KeyTest$mcI$sp.up(<console>:8)
>        at KeyTest$mcI$sp.up(<console>:8)
>
> When specialization works you will see only one specialized call:
>
>        at KeyTest$mcI$sp.up$mcI$sp(<console>:8)
>
>