releasing 2.8

Hi,

I think that #2850 should be fixed before an official Beta. Which,
with none useful error messages, spent me almost one day to narrow
down the cause, I almost gave up to port a lib to Scala.

Cheers,
-Caoyuan

On Thu, Jan 7, 2010 at 2:18 AM, martin odersky wrote:
> Hi all,
>
> At yesterdays Scala meeting we did a quick rundown of what needs to be
> done to release the 2.8 beta.
>
> We decided on the following roadmap:
>
> First, there were two regressions failed against 2.8, which are still open:
>
> - A few bugs were reported against RC6. Currently open: #2850 and
> #2835, which was marked as a duplicate of #2310. They do not seem
> critical, but we'll try to fix them before the beta
>
> They are assigned to Iulian and Paul. Neither looks like a
> showstopper, but maybe Iulian and Paul can see whether they can come
> up with a fix in the next days before we freeze the code. Or else let
> us know that it will stay as it is.
>
> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!
>
> Once we have sorted this out, we plan to do another RC, hopefully by
> the need of this week or beginning next week.
>
> Unless something serious comes up this RC will be the public beta. ETA
> if all goes well: end of next week.
>
> Cheers
>
>  -- Martin
>

I wonder about the library bugs that have been reported. There's a bug on RedBlack which can be pretty detrimental to TreeSet/SortedSet, but that's not a regression. There are a few other library bugs, some of which are regressions. But I'm mostly concerned about immutable.HashMap, of which there are many tickets open. The general sentiment on these tickets, too, is that it is going to be replaced for 2.8, so there's no need to worry about it. I have seen on mailing lists people expecting that too, or the reinstatement of TreeHashMap.

On Wed, Jan 6, 2010 at 4:18 PM, martin odersky <martin.odersky@epfl.ch> wrote:

Hi all,

At yesterdays Scala meeting we did a quick rundown of what needs to be
done to release the 2.8 beta.

We decided on the following roadmap:

First, there were two regressions failed against 2.8, which are still open:

- A few bugs were reported against RC6. Currently open: #2850 and
#2835, which was marked as a duplicate of #2310. They do not seem
critical, but we'll try to fix them before the beta

They are assigned to Iulian and Paul. Neither looks like a
showstopper, but maybe Iulian and Paul can see whether they can come
up with a fix in the next days before we freeze the code. Or else let
us know that it will stay as it is.

Also if there are still uniresolved issues in community projects
relative to 2.8 RC6, we need to know _now_!

Once we have sorted this out, we plan to do another RC, hopefully by
the need of this week or beginning next week.

Unless something serious comes up this RC will be the public beta. ETA
if all goes well: end of next week.

Cheers

 -- Martin

--
Daniel C. Sobral

I travel to the future all the time.

On Wed, Jan 6, 2010 at 12:30 PM, Daniel Sobral <dcsobral@gmail.com> wrote:

I wonder about the library bugs that have been reported. There's a bug on RedBlack which can be pretty detrimental to TreeSet/SortedSet, but that's not a regression. There are a few other library bugs, some of which are regressions. But I'm mostly concerned about immutable.HashMap, of which there are many tickets open. The general sentiment on these tickets, too, is that it is going to be replaced for 2.8, so there's no need to worry about it. I have seen on mailing lists people expecting that too, or the reinstatement of TreeHashMap.

My 2 cents:

I'd like to see Beta1 ship as long as the usual suspects (Lift, Specs, ScalaTest, ScalaCheck, sbt, etc.) compile and run with it.  Let's get it out there and start banging on it.  I'm not overly concerned about some library bugs (including the immutable.HashMap bug which I think does need to get fixed for 2.8 final) during the early betas.  Let's get this baby out the door and have something to work against.
 

On Wed, Jan 6, 2010 at 4:18 PM, martin odersky <martin.odersky@epfl.ch> wrote:

Hi all,

At yesterdays Scala meeting we did a quick rundown of what needs to be
done to release the 2.8 beta.

We decided on the following roadmap:

First, there were two regressions failed against 2.8, which are still open:

- A few bugs were reported against RC6. Currently open: #2850 and
#2835, which was marked as a duplicate of #2310. They do not seem
critical, but we'll try to fix them before the beta

They are assigned to Iulian and Paul. Neither looks like a
showstopper, but maybe Iulian and Paul can see whether they can come
up with a fix in the next days before we freeze the code. Or else let
us know that it will stay as it is.

Also if there are still uniresolved issues in community projects
relative to 2.8 RC6, we need to know _now_!

Once we have sorted this out, we plan to do another RC, hopefully by
the need of this week or beginning next week.

Unless something serious comes up this RC will be the public beta. ETA
if all goes well: end of next week.

Cheers

 -- Martin

--
Daniel C. Sobral

I travel to the future all the time.

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

On Wednesday 06 January 2010 21:18:27 martin odersky wrote:
> ...the next days before we freeze the code.

Will Ingo have an opportunity to fix few scala.swing issues? I worry he will
be back after freezing.

Are we talking code freeze here, or feature freeze?

Either way, bugs must surely get fixed between the first beta and
release, otherwise they're identical!

On Thursday, January 7, 2010, Andrew Gaydenko wrote:
> On Wednesday 06 January 2010 21:18:27 martin odersky wrote:
>> ...the next days before we freeze the code.
>
> Will Ingo have an opportunity to fix few scala.swing issues? I worry he will
> be back after freezing.
>

I've finally gotten around to an isolated test case for #2741. It is
another example of code that compiles jointly, but fails if compiled
separately. Could be worth a quick look pre-beta in case the fix
requires changes to the pickler/unpickler. We worked around the
problem in Scalaz by defining methods directly in an object instead of
using a trait. But it both ugly and fragile.

Furthermore, partest really needs the capability to test separate compilation.

There are two in the pattern matcher that warrant some attention pre-final.

#2337 - wrong branch chosen. (See attachment for small test case.)
#2310 - crasher performing simple match against Stream.

The first one is quite dangerous -- I don't use extractor objects now
because I don't trust them.

-jason

http://lampsvn.epfl.ch/trac/scala/ticket/2741
http://lampsvn.epfl.ch/trac/scala/ticket/2337
http://lampsvn.epfl.ch/trac/scala/ticket/2310

On Wed, Jan 6, 2010 at 7:18 PM, martin odersky wrote:
> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!

On Thu, Jan 7, 2010 at 10:51, Jason Zaugg <jzaugg@gmail.com> wrote:

I've finally gotten around to an isolated test case for #2741. It is
another example of code that compiles jointly, but fails if compiled
separately. Could be worth a quick look pre-beta in case the fix
requires changes to the pickler/unpickler.  We worked around the
problem in Scalaz by defining methods directly in an object instead of
using a trait. But it both ugly and fragile.

Furthermore, partest really needs the capability to test separate compilation.

it can, for instancehttp://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/test/files/pos/t1942  

There are two in the pattern matcher that warrant some attention pre-final.

#2337 - wrong branch chosen. (See attachment for small test case.)
#2310 - crasher performing simple match against Stream.

The first one is quite dangerous -- I don't use extractor objects now
because I don't trust them.

-jason

http://lampsvn.epfl.ch/trac/scala/ticket/2741
http://lampsvn.epfl.ch/trac/scala/ticket/2337
http://lampsvn.epfl.ch/trac/scala/ticket/2310

On Wed, Jan 6, 2010 at 7:18 PM, martin odersky <martin.odersky@epfl.ch> wrote:
> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!

By testing separate compilation, I don't mean passing a set of files to scalac.

Rather, the following is required:

val outDir = new File("./out")
val groups = List(List("foo.scala", "bar.scala"), List("baz.scala"))
for (group <- group) {
compile(group, classpath = List(out.getPath), output = out)
}

def compile(files: List[String], classpath: List[String], output: File) = {
// start a new instance of compiler
}

Is this possible already?

I think that an extended version of this style of testing has been
proposed by Mark Harrah for testing the build manager. It would also
touch source files in between runs of the compiler.

-jason

On Thu, Jan 7, 2010 at 11:05 AM, Lukas Rytz wrote:
>> Furthermore, partest really needs the capability to test separate
>> compilation.
>
> it can, for instance
> http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/test/files/pos/t1942

yes, the groups are done by filename ending _N.scala / _N.java
Forwarding a text by Philipp Haller:

You can now compile files in sequential groups where each group is identified by a file name suffix. For instance, having the files
J_1.java test_2.scala
in a multi-file test will first compile the java file and then the scala file. Files with no suffix are in the very first group(before _1). You can have both scala and java files in the same group. In general, compilation proceeds as follows:
for each group: if there are scala files: - compile all files (scala and java) together with scalac if there are java files: - compile all java files together with javac  if there are scala files:   - compile all scala files together with scalac
This works for both "pos" as well as the "run" tests (including "jvm" etc.).

On Thu, Jan 7, 2010 at 11:41, Jason Zaugg <jzaugg@gmail.com> wrote:

By testing separate compilation, I don't mean passing a set of files to scalac.

Rather, the following is required:

val outDir = new File("./out")
val groups = List(List("foo.scala", "bar.scala"), List("baz.scala"))
for (group <- group) {
  compile(group, classpath = List(out.getPath), output = out)
}

def compile(files: List[String], classpath: List[String], output: File) = {
 // start a new instance of compiler
}

Is this possible already?

I think that an extended version of this style of testing has been
proposed by Mark Harrah for testing the build manager. It would also
touch source files in between runs of the compiler.

-jason

On Thu, Jan 7, 2010 at 11:05 AM, Lukas Rytz <lukas.rytz@epfl.ch> wrote:
>> Furthermore, partest really needs the capability to test separate
>> compilation.
>
> it can, for instance
> http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/test/files/pos/t1942

Jason Zaugg wrote:
> By testing separate compilation, I don't mean passing a set of files to scalac.
>
> Rather, the following is required:
>
> val outDir = new File("./out")
> val groups = List(List("foo.scala", "bar.scala"), List("baz.scala"))
> for (group <- group) {
> compile(group, classpath = List(out.getPath), output = out)
> }
>
> def compile(files: List[String], classpath: List[String], output: File) = {
> // start a new instance of compiler
> }
>
> Is this possible already?

Yes, in fact the example Lukas gave has two groups, each consisting of a
single file that will be compiled separately.

Your example can be done by naming the files as follows:
foo_1.scala, bar_1.scala, baz_2.scala

Cheers,
Philipp

> On Thu, Jan 7, 2010 at 11:05 AM, Lukas Rytz wrote:
>>> Furthermore, partest really needs the capability to test separate
>>> compilation.
>> it can, for instance
>> http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/test/files/pos/t1942

Hi Martin,
> Also if there are still uniresolved issues in community projects
relative to 2.8 RC6, we need to know _now_!
 I have deployed specs for scala-2.8.0.Beta1-RC6 today and haven't found any significant issue. Besides #2733 is also working for me now.
Eric.
----------------------------------------------
Eric TORREBORRE
T +61 411 707 402
E etorreborre@yahoo.com
B http://etorreborre.blogspot.com
P http://specs.googlecode.com
----------------------------------------------

De : martin odersky <martin.odersky@epfl.ch>
À : scala-internals@listes.epfl.ch
Envoyé le : Jeu 7 Janvier 2010, 5 h 18 min 27 s
Objet : [scala-internals] releasing 2.8

Hi all,

At yesterdays Scala meeting we did a quick rundown of what needs to be
done to release the 2.8 beta.

We decided on the following roadmap:

First, there were two regressions failed against 2.8, which are still open:

- A few bugs were reported against RC6. Currently open: #2850 and
#2835, which was marked as a duplicate of #2310. They do not seem
critical, but we'll try to fix them before the beta

They are assigned to Iulian and Paul. Neither looks like a
showstopper, but maybe Iulian and Paul can see whether they can come
up with a fix in the next days before we freeze the code. Or else let
us know that it will stay as it is.

Also if there are still uniresolved issues in community projects
relative to 2.8 RC6, we need to know _now_!

Once we have sorted this out, we plan to do another RC, hopefully by
the need of this week or beginning next week.

Unless something serious comes up this RC will be the public beta. ETA
if all goes well: end of next week.

Cheers

-- Martin

On Thu, Jan 7, 2010 at 12:56 PM, Eric Torreborre wrote:
> Hi Martin,
>> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!
>
> I have deployed specs for scala-2.8.0.Beta1-RC6 today and haven't found any
> significant issue.
> Besides #2733 is also working for me now.

Excellent! -- Martin

> Eric.
> ----------------------------------------------
> Eric TORREBORRE
> T +61 411 707 402
> E etorreborre@yahoo.com
> B http://etorreborre.blogspot.com
> P http://specs.googlecode.com
> ----------------------------------------------
>
> ________________________________
> De : martin odersky
> À : scala-internals@listes.epfl.ch
> Envoyé le : Jeu 7 Janvier 2010, 5 h 18 min 27 s
> Objet : [scala-internals] releasing 2.8
>
> Hi all,
>
> At yesterdays Scala meeting we did a quick rundown of what needs to be
> done to release the 2.8 beta.
>
> We decided on the following roadmap:
>
> First, there were two regressions failed against 2.8, which are still open:
>
> - A few bugs were reported against RC6. Currently open: #2850 and
> #2835, which was marked as a duplicate of #2310. They do not seem
> critical, but we'll try to fix them before the beta
>
> They are assigned to Iulian and Paul. Neither looks like a
> showstopper, but maybe Iulian and Paul can see whether they can come
> up with a fix in the next days before we freeze the code. Or else let
> us know that it will stay as it is.
>
> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!
>
> Once we have sorted this out, we plan to do another RC, hopefully by
> the need of this week or beginning next week.
>
> Unless something serious comes up this RC will be the public beta. ETA
> if all goes well: end of next week.
>
> Cheers
>

On Thu, Jan 7, 2010 at 2:46 AM, Caoyuan wrote:
> Hi,
>
> I think that #2850 should be fixed before an official Beta. Which,
> with none useful error messages, spent me almost one day to narrow
> down the cause, I almost gave up to port a lib to Scala.

I noticed #2850 just fixed by dragos, thanks a lot. BTW, I ported my
AIOTrade project to Scala, which has lots swing code and heavy
computing, there is no issue under current RC now.

Cheers,
-Caoyuan

>
> Cheers,
> -Caoyuan
>
> On Thu, Jan 7, 2010 at 2:18 AM, martin odersky wrote:
>> Hi all,
>>
>> At yesterdays Scala meeting we did a quick rundown of what needs to be
>> done to release the 2.8 beta.
>>
>> We decided on the following roadmap:
>>
>> First, there were two regressions failed against 2.8, which are still open:
>>
>> - A few bugs were reported against RC6. Currently open: #2850 and
>> #2835, which was marked as a duplicate of #2310. They do not seem
>> critical, but we'll try to fix them before the beta
>>
>> They are assigned to Iulian and Paul. Neither looks like a
>> showstopper, but maybe Iulian and Paul can see whether they can come
>> up with a fix in the next days before we freeze the code. Or else let
>> us know that it will stay as it is.
>>
>> Also if there are still uniresolved issues in community projects
>> relative to 2.8 RC6, we need to know _now_!
>>
>> Once we have sorted this out, we plan to do another RC, hopefully by
>> the need of this week or beginning next week.
>>
>> Unless something serious comes up this RC will be the public beta. ETA
>> if all goes well: end of next week.
>>
>> Cheers
>>
>>  -- Martin
>>
>

On Wednesday 06 January 2010, martin odersky wrote:
> ...
>
> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!

Jason pointed out that sbt's analysis phase was taking an unreasonable amount
of time on scalaz-core. It is taking something like 20 seconds when it should
take ~1 second. I checked it out and it looks like `classPath.findClass` in
2.8 takes much longer than the corresponding `root.find` did in 2.7.

I modified scala.tools.nsc.util.ClassPath to include:

private lazy val classes0 = classes
private lazy val packages0 = packages

and used classes0 and packages0 in the implementation of ClassPath.findClass
instead of classes and packages.

This returned the runtime of sbt's analysis phase to ~1s. The effect is more
pronounced with scalaz-http, which has scalaz-core's output directory on its
classpath. With scalaz-http, the time goes from 89 seconds down to 1 second.

It isn't obvious to me from looking at ClassPath in 2.7.x that it cached the
classpath similar to what I've done, so I don't know that this is the proper
fix.

Thanks,
Mark

I just noticed that scalap (and consequently, IntelliJ) incorrectly decompiles signatures with default parameters. The parameter is omittted, and, in addition, synthetic methods are displayed. These synthetic methods also appear in the REPL tab completion.

scalac -Xshowclass reports 'BAD TAG'. Furthermore, compilation with -Xshowclass fails (independent of using default parameters) with the exception: "scala.tools.nsc.MissingRequirementError: object test not found."

-jason

https://lampsvn.epfl.ch/trac/scala/ticket/2885


On Wed, Jan 6, 2010 at 7:18 PM, martin odersky <martin.odersky@epfl.ch> wrote:

Also if there are still uniresolved issues in community projects
relative to 2.8 RC6, we need to know _now_!

On Sun, Jan 10, 2010 at 03:29:33PM +0100, Jason Zaugg wrote:
> I just noticed that scalap (and consequently, IntelliJ) incorrectly
> decompiles signatures with default parameters. The parameter is
> omittted, and, in addition, synthetic methods are displayed. These
> synthetic methods also appear in the REPL tab completion.
>
> scalac -Xshowclass reports 'BAD TAG'. Furthermore, compilation with
> -Xshowclass fails (independent of using default parameters) with the
> exception: "scala.tools.nsc.MissingRequirementError: object test not
> found."

FYI so nobody sinks too much work into scalap, I have reimplemented it
with the standard library parser combinators and pending approval intend
to replace the one in trunk with mine. (I realize intellij may want to
keep using the existing one.) It's a complete bytecode parser in fact,
usable as a superior javap as well as scalap. It's not quite ready to
go yet but I will have to ship it soon one way or another so I can get
back to all these lovely bugs.

2010/1/10 Paul Phillips <paulp@improving.org>

On Sun, Jan 10, 2010 at 03:29:33PM +0100, Jason Zaugg wrote:
> I just noticed that scalap (and consequently, IntelliJ) incorrectly
> decompiles signatures with default parameters. The parameter is
> omittted, and, in addition, synthetic methods are displayed. These
> synthetic methods also appear in the REPL tab completion.
>
> scalac -Xshowclass reports 'BAD TAG'. Furthermore, compilation with
> -Xshowclass fails (independent of using default parameters) with the
> exception: "scala.tools.nsc.MissingRequirementError: object test not
> found."

FYI so nobody sinks too much work into scalap, I have reimplemented it
with the standard library parser combinators and pending approval intend
to replace the one in trunk with mine.  (I realize intellij may want to
keep using the existing one.) It's a complete bytecode parser in fact,
usable as a superior javap as well as scalap.  It's not quite ready to
go yet but I will have to ship it soon one way or another so I can get
back to all these lovely bugs.

Great news!  I'm looking forward to seeing it on the review board. :)Bring on the deep-fried locusts... 

--
Paul Phillips      | One way is to make it so simple that there are
Analgesic          | obviously no deficiencies. And the other way is to make
Empiricist         | it so complicated that there are no obvious deficiencies.
pull his pi pal!   |     -- Hoare

--
Kevin Wright

mail/google talk: kev.lee.wright@googlemail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

Paul,

The idea to re-implement scalap using default parser combinator library was pending for a long time, so it's really nice that you've finally done it. All the functionality IntelliJ needs from the existing scalap implementation is covered by the set of scalap test included into the partest suite. Also, to not break compatibility, could you not to change signatures of classes and objects in the "tools.scalap.scalax.rules.scalasig" package?

Ilya

2010/1/10 Paul Phillips <paulp@improving.org>

On Sun, Jan 10, 2010 at 03:29:33PM +0100, Jason Zaugg wrote:
> I just noticed that scalap (and consequently, IntelliJ) incorrectly
> decompiles signatures with default parameters. The parameter is
> omittted, and, in addition, synthetic methods are displayed. These
> synthetic methods also appear in the REPL tab completion.
>
> scalac -Xshowclass reports 'BAD TAG'. Furthermore, compilation with
> -Xshowclass fails (independent of using default parameters) with the
> exception: "scala.tools.nsc.MissingRequirementError: object test not
> found."

FYI so nobody sinks too much work into scalap, I have reimplemented it
with the standard library parser combinators and pending approval intend
to replace the one in trunk with mine.  (I realize intellij may want to
keep using the existing one.) It's a complete bytecode parser in fact,
usable as a superior javap as well as scalap.  It's not quite ready to
go yet but I will have to ship it soon one way or another so I can get
back to all these lovely bugs.

--
Paul Phillips      | One way is to make it so simple that there are
Analgesic          | obviously no deficiencies. And the other way is to make
Empiricist         | it so complicated that there are no obvious deficiencies.
pull his pi pal!   |     -- Hoare

On Thu, Jan 07, 2010 at 10:51:54AM +0100, Jason Zaugg wrote:
> There are two in the pattern matcher that warrant some attention pre-final.
>
> #2337 - wrong branch chosen. (See attachment for small test case.)
> #2310 - crasher performing simple match against Stream.
>
> The first one is quite dangerous -- I don't use extractor objects now
> because I don't trust them.

True that. I've not been a fan of #2337 since all the way back when we
called him #1697. (And we had a less printable name for him before
that, but that was the first report to pinpoint it.) A great deal of the
work I've done on the pattern matcher has been in an attempt to get this
bug fixed. I have a lot of code from October which is close -- but as
with #313131 and its buddy #3e3e3e I tend to get stymied in the last mile
because other things start breaking.

In case people are wondering why I can't just fix the freaking thing,
let me explain what happens. It helps to realize that the pattern
matcher was not all written at once, and extractors were bolted on after
the fact. Extractors (unapplies) are encoded in a tree like this:

UnApply(Apply(_, _), _)

which works fine if the unapply is written to take an "Any" as a
parameter. However the spec lets us tighten that, and the pattern
matcher needs to notice if the scrutinee is type incompatible with the
argument type. So somebody said (and I'm paraphrasing here because I
have no idea what really happened) "hey look, we already do type tests
in the pattern matcher. Let's just wrap one around the unapply, toss it
back into the hopper, and pattern match our way to goodness. What could
possibly go wrong?" The thought being that you could look for this
nested tree:

Typed(UnApply(Apply(_, _), _), tpt)

and test against tpt. In fact let's look at exactly that code from back
before I got my hands on the matcher. Watch it wrap the unapply and
then prepend the new pattern to the front of the list.

case ua @ UnApply(Apply(fn, _), _) =>
DBG("Unapply(Apply())")
fn.tpe match {
case MethodType(List(argtpe,_*),_) =>
val npat = (if (temp(j).tpe <:< argtpe) ua else Typed(ua,TypeTree(argtpe)).setType(argtpe))
pats = (makeBind(vs, npat) setType argtpe)::pats
}

OK, sounds great. So now we have the other piece of the puzzle: the
matching algorithm. It's based on this paper (and this is something I
didn't discover until months into the process of deconstructing all the
knobs - a comment would have really helped.)

"A Term Pattern-Match Compiler Inspired by Finite Automata Theory"
Mikael Pettersson
ftp://ftp.ida.liu.se/pub/labs/pelab/papers/cc92pmc.ps.gz

The paper, while useful and reasonably understandable, does not grapple
with such real-world perversions as bolted-on extractors which may or
may not match. It has constructor patterns (case classes) and
variables, that's about it: so the algorithm assumes that if the current
case doesn't match, then the next case isn't going to match if it's
strictly more specific than this case. (And you don't even want to know
how case class inheritance interacts with it.)

So under other conditions the aforedescribed approach might have been
reasonable enough, except now we get here. I fixed this simple version
of the bug by the way, so it's only for illustration. The remaining
ones go deeper, and it doesn't seem like I'm going to successfully root
them out until I understand every single line of the thing. And I
don't, yet.

object Extractor {
def unapply(x: AnyRef): Option[Int] = None
}

"abc" match {
case Extractor(x) => x
case x: String => 2
case _ => error("")
}

In this simplified example, after the unapply is rewrapped in Typed, it
looks to the algorithm like two consecutive type tests where if the
first one fails (AnyRef) the second one cannot match. So it skips the
next case altogether. The type test and the unapply cannot be
considered separately without modifying the algorithm. Either they have
to be joined at the hip or the algorithm has to change.

I went with marrying them, and started rearchitecting the pattern
matcher to make that possible. Now the problem became that the whole
pattern matcher was operating on raw AST trees, which map semantically
onto the various pattern definitions with the elegance of a camel
climbing a tree. An example:

case UnApply(Apply(TypeApply(sel @ Select(stor, nme.unapplySeq),List(tptArg)),_),ArrayValue(_,xs)::Nil) if (stor.symbol eq definitions.ListModule) =>

So I had to reverse engineer behavior from how the ASTs were used,
remembering that *we're afraid to use extractors* and in fact before
#1697 showed up in February 2009 I walked straight into that one.
Here's a comment I wrote in r16553 from November 2008, back when I was
still funneling my patches through DRMacIver. Oh goodness has this been
going on for that long.

+ // TODO - this doesn't work! For some reason this sometimes returns false when encapsulated
+ // in an object like this, when it returns true if the same logic is applied literally in
+ // the classifyPat match in ParallelMatching. I couldn't figure out why, and it concerns
+ // me - if this isn't working maybe other unapply objects aren't working right either. The big
+ // problem with using unapply and type matching is that if something's amiss, it will simply fail
+ // silently, with the bug most likely manifesting at some unknown later point.
+ object Ident_Or_Empty {
+ def unapply(x: Any) = x match {
+ case Ident(nme.WILDCARD) | EmptyTree | _: Typed | _: Literal => true
+ // this returns false, and then a line identical the one above will match
+ // back in PM.scala.
+ case _ => false
+ }

Now maybe armed with this explanation a fresh pair of eyes can go in
there and do some damage. Don't worry about showing me up, just because
I haven't managed to fix it in a year and a half doesn't mean you can't
fix it in five minutes. I'd be thrilled.

Summary: absolutely yes, #1697 / #2337 should be considered blockers for
2.8. I probably have another 50 bugs I think should be blockers, but if
I had to pick just one, this would be it because there is no crash or
visible failure: merely silent doom.

Paul,

Thanks for the war story. I feel like we should have bought you a beer, or something stonger, to help you recount it.

This might seem naive, but would it be possible, perhaps via a compiler option or an annotation on the scrutinee, to compile matches involving extractor objects to something as terribly performant but clearly correct as the following? (Assuming it is correct!) It might help flush out meta-circularity bugs and the paralysing fear thereof.

-jason

import Stream._
object diyMatcher {
  def test[A](lefts: Stream[A], rights: Stream[A]) = {
    (lefts, rights) `match` (
        {case (Stream.Empty, Stream.Empty) => None},

        {case (l #:: ls, rs) => None}
    )
  }

  trait Matchable[A] { 
    def `match`[B](cases: PartialFunction[A, B]*): B
  }
  
  implicit def anyToMatchable[A](a: A): Matchable[A] = new Matchable[A]{

    def `match`[B](cases: PartialFunction[A, B]*) = {
      cases.toList match {
        case Nil => throw new MatchError(a)
        case x :: _ if x.isDefinedAt(a) => x(a)
        case _ :: xs => a `match` (xs : _*) 

      }
    }
  }
}

On Mon, Jan 11, 2010 at 7:25 AM, Paul Phillips <paulp@improving.org> wrote:

Now maybe armed with this explanation a fresh pair of eyes can go in
there and do some damage.  Don't worry about showing me up, just because
I haven't managed to fix it in a year and a half doesn't mean you can't
fix it in five minutes.  I'd be thrilled.

Summary: absolutely yes, #1697 / #2337 should be considered blockers for
2.8.  I probably have another 50 bugs I think should be blockers, but if
I had to pick just one, this would be it because there is no crash or
visible failure: merely silent doom.

On Mon, Jan 11, 2010 at 07:50:29AM +0100, Jason Zaugg wrote:
> This might seem naive, but would it be possible, perhaps via a
> compiler option or an annotation on the scrutinee, to compile matches
> involving extractor objects to something as terribly performant but
> clearly correct as the following? (Assuming it is correct!) It might
> help flush out meta-circularity bugs and the paralysing fear thereof.

I have made a couple half-hearted attempts at the baby step of a super
straightforward matcher. Each time I ended up finding that there are
things happening which I don't understand well enough to falsify. It
doesn't help that explicitouter is one of those middle stages where
symbols and types and trees are in various states of dishevel. Maybe
I'd have better luck coming in at the parser.

Still, it's hard to get motivated for such a task when it solves very
little of the real problem. My real issue with the pattern matcher now
is that it has eaten so many days with near zero payoff, I have come to
loathe opening up those files.

Can you please try again with r20438?http://lampsvn.epfl.ch/trac/scala/changeset/20438
Thanks for reporting!Lukas

On Sun, Jan 10, 2010 at 00:21, Mark Harrah <harrah@bu.edu> wrote:

On Wednesday 06 January 2010, martin odersky wrote:
> ...
>
> Also if there are still uniresolved issues in community projects
> relative to 2.8 RC6, we need to know _now_!

Jason pointed out that sbt's analysis phase was taking an unreasonable amount
of time on scalaz-core.  It is taking something like 20 seconds when it should
take ~1 second.  I checked it out and it looks like `classPath.findClass` in
2.8 takes much longer than the corresponding `root.find` did in 2.7.

I modified scala.tools.nsc.util.ClassPath to include:

 private lazy val classes0 = classes
 private lazy val packages0 = packages

and used classes0 and packages0 in the implementation of ClassPath.findClass
instead of classes and packages.

This returned the runtime of sbt's analysis phase to ~1s.  The effect is more
pronounced with scalaz-http, which has scalaz-core's output directory on its
classpath.  With scalaz-http, the time goes from 89 seconds down to 1 second.

It isn't obvious to me from looking at ClassPath in 2.7.x that it cached the
classpath similar to what I've done, so I don't know that this is the proper
fix.

Thanks,
Mark

Lukas,

r20438 fixes the issue for me. Thanks very much for the quick response.

-Mark

Quoting Lukas Rytz :

> Can you please try again with r20438?
> http://lampsvn.epfl.ch/trac/scala/changeset/20438
>
> Thanks for reporting!
> Lukas
>
>
> On Sun, Jan 10, 2010 at 00:21, Mark Harrah wrote:
>
>> On Wednesday 06 January 2010, martin odersky wrote:
>> > ...
>> >
>> > Also if there are still uniresolved issues in community projects
>> > relative to 2.8 RC6, we need to know _now_!
>>
>> Jason pointed out that sbt's analysis phase was taking an unreasonable
>> amount
>> of time on scalaz-core. It is taking something like 20 seconds when it
>> should
>> take ~1 second. I checked it out and it looks like `classPath.findClass`
>> in
>> 2.8 takes much longer than the corresponding `root.find` did in 2.7.
>>
>> I modified scala.tools.nsc.util.ClassPath to include:
>>
>> private lazy val classes0 = classes
>> private lazy val packages0 = packages
>>
>> and used classes0 and packages0 in the implementation of
>> ClassPath.findClass
>> instead of classes and packages.
>>
>> This returned the runtime of sbt's analysis phase to ~1s. The effect is
>> more
>> pronounced with scalaz-http, which has scalaz-core's output directory on
>> its
>> classpath. With scalaz-http, the time goes from 89 seconds down to 1
>> second.
>>
>> It isn't obvious to me from looking at ClassPath in 2.7.x that it cached
>> the
>> classpath similar to what I've done, so I don't know that this is the
>> proper
>> fix.
>>
>> Thanks,
>> Mark
>>
>>
>

Fate seems to be interested in illustrating my difficulties. You are
familiar with #2310, a regression I now know I introduced in r18184 with
commit message:

Creating case classes in preference to passing around a variety
of inscrutable tuples. And, fix and test case for #1697. There
remain serious extractor issues which I hope to have fully
diagnosed in the near future.

So even fixing the "simple" variation of #1697 overcame the regression
defenses.