Re: scalac failed to load class file produced by javac in RC7

I found out some more about the mechanics of the problem.
In phase Typer, we always load first the symbol of the enclosing class
(i.e. Tuppel), and only then load symbols of members. This is
essential because Tuppel contains type parameters that need to be
entered into the ClassTParams map before proceeding to members.

When we reload that class in specialize, we do not re-load Tuppel
first, but instead immediately load Tuppel$1, the anonymous inner
class in Tuppel.get. So the ClassTParams map is empty.

We do not re-load Tuppel's type because it has already been computed.
On the other hand, the anonymous inner class Tuppel$1 has not been
completed, in fact it does not even show up in the interface.

So the problems seem to be

- why do we complete Tupel$1 in phase specialize?
- why do we reload the class file doing that instead of just using the old one?

Puzzled

On Tue, Jul 6, 2010 at 3:54 PM, martin odersky <martin.odersky@epfl.ch> wrote:

On Tue, Jul 6, 2010 at 2:55 PM, iulian dragos <iulian.dragos@epfl.ch> wrote:
> This bug is technically a regression, so we need to decide if we're going to
> fix it and have an RC8 or not.
> It is not clear currently if this is can be truly fixed by turning off
> specialization. The type *is* forced by specialization, but I think it is a
> symptom of a deeper problem inside the classfile parser. We could, however,
> make specialization more careful about what types it forces, but I have the
> feeling it only takes a bit more sophistication to produce a program that
> does not need it to force that type.
> So the question is what to do: have a new RC, or go on with the release?

Part of the problem seems to be that the class file is read twice. If
you turn -verbose on for Adriaan's example you see that Tuppel is
successfully read in phase typer, but then reading fails in phase
specialize. In principle classfile reading is not made explicitly to
be repeatable; there was no need to do this so far. I'd like to
understand why we read the file a second time. Is that necessary, or
an accident? If it is an accident, it would be better to fix that
because, bug or not, this repeated reading is wasteful.

Absolutely. Without really looking at the code, I think the following happens (it would be good if someone checked it, as I don't think it is extremely difficult, and I am very time-constrained): The symbol that fails loading is an inner class, and the two classfile readers are different, for /two different symbols/. Note that, at least for a while, there *are* two symbols for one inner class: one that is created for each classfile in a directory (package), and another one that is created inside the owner class, when the owner class is fully loaded. Normally, the top-level symbol of the inner class is removed (unlinked) from the package scope once the inner class symbol has been properly inserted in its owner. Somehow, this one hangs around and trips the compiler.
We should find out why the old symbol still appears in a scope, and how come the info transform in specialization steps on it.
Specialization only needs to know the type parameters of a symbol. We can either guard the transformInfo against Java symbols (which I'm not sure it works, since before loading the classfile it's hard to tell if a symbol comes from Java or not), or find why the other symbol is there.
iulian
 

Now, to understand whether we need an RC8, it would be good to get a
feeling for the precise combination of features that trigger the
error. Can someone find out the exact conditions when this happens?
Only then we can judge how common these are, and whether there is an
easy fix.

Cheers

On Tue, Jul 6, 2010 at 4:10 PM, martin odersky <martin.odersky@epfl.ch> wrote:

I found out some more about the mechanics of the problem.
In phase Typer, we always load first the symbol of the enclosing class
(i.e. Tuppel), and only then load symbols of members. This is
essential because Tuppel contains type parameters that need to be
entered into the ClassTParams map before proceeding to members.

When we reload that class in specialize, we do not re-load Tuppel
first, but instead immediately load Tuppel$1, the anonymous inner
class in Tuppel.get. So the ClassTParams map is empty.

We do not re-load Tuppel's type because it has already been computed.
On the other hand, the anonymous inner class Tuppel$1 has not been
completed, in fact it does not even show up in the interface.

That doesn't seem right. The inner classes should have been entered by 'enterOwnInnerClasses' in the classfile parser. Maybe that's the point where class type parameters should be attached to their lazy types, so that the classfile parser may get to them later, when the type is forced.  

So the problems seem to be

 - why do we complete Tupel$1 in phase specialize?

to know its type parameters. 

 - why do we reload the class file doing that instead of just using the old one?

 

Puzzled

On Tue, Jul 6, 2010 at 4:20 PM, iulian dragos wrote:
>
>
> On Tue, Jul 6, 2010 at 4:10 PM, martin odersky
> wrote:
>>
>> I found out some more about the mechanics of the problem.
>> In phase Typer, we always load first the symbol of the enclosing class
>> (i.e. Tuppel), and only then load symbols of members. This is
>> essential because Tuppel contains type parameters that need to be
>> entered into the ClassTParams map before proceeding to members.
>>
>> When we reload that class in specialize, we do not re-load Tuppel
>> first, but instead immediately load Tuppel$1, the anonymous inner
>> class in Tuppel.get. So the ClassTParams map is empty.
>>
>> We do not re-load Tuppel's type because it has already been computed.
>> On the other hand, the anonymous inner class Tuppel$1 has not been
>> completed, in fact it does not even show up in the interface.
>
> That doesn't seem right. The inner classes should have been entered by
> 'enterOwnInnerClasses' in the classfile parser. Maybe that's the point where
> class type parameters should be attached to their lazy types, so that the
> classfile parser may get to them later, when the type is forced.
>

I believe if we would use the lazy type everything would be fine. But
we don't. Instead we prompt a full
reload of the class file (in fact we even prompt a full reload of all
enclosiung package loaders). Only this time we do not read the
enclosing class first.

Cheers

I haven't looked what happens outside ClassfileParser, but I feel pretty confident the bug is restricted to that class.
From my comment on the ticket:

I think the underlying defect is that we don't take the EnclosingMethod? attribute into account when parsing an inner class:

parsed output from javap:
final class Tuppel$1 extends Tuppel
  Signature: "LTuppel<TA;>;;"
  EnclosingMethod: "Tuppel::get:()LTuppel;"
  InnerClass: "Tuppel$1"

the typeparameters of the method get:()LTuppel in Tuppel need to be brought into scope when sigToType is called on Tuppel$1 (as far as I can see, this never happens now, irrespective of the order in which these symbols are completed). Unfortunately, that requires a significant rewrite of parseAttributes since it currently parses the attributes in order. However, when it encounters the Signature attribute it should look ahead to see whether there is an EnclosingMethod attribute, and enter that method's type parameters into scope

cheers

adriaan

On Tue, Jul 6, 2010 at 4:10 PM, martin odersky <martin.odersky@epfl.ch> wrote:

I found out some more about the mechanics of the problem.
In phase Typer, we always load first the symbol of the enclosing class
(i.e. Tuppel), and only then load symbols of members. This is
essential because Tuppel contains type parameters that need to be
entered into the ClassTParams map before proceeding to members.

When we reload that class in specialize, we do not re-load Tuppel
first, but instead immediately load Tuppel$1, the anonymous inner
class in Tuppel.get. So the ClassTParams map is empty.

We do not re-load Tuppel's type because it has already been computed.
On the other hand, the anonymous inner class Tuppel$1 has not been
completed, in fact it does not even show up in the interface.

So the problems seem to be

 - why do we complete Tupel$1 in phase specialize?
 - why do we reload the class file doing that instead of just using the old one?

Puzzled

> On Tue, Jul 6, 2010 at 2:55 PM, iulian dragos wrote:
> > So the question is what to do: have a new RC, or go on with the release?

It's weird being the guy saying this, but we should release no matter
what we find now. I mean that quite literally and unconditionally. We
can enjoy this trickle of troublesome issues indefinitely, and then when
we finally release and see what is uncovered immediately afterward, we
will feel pretty silly for dragging it out like we were converging on
some platonic ideal.

RC7 basically works, it has built some big projects, and it is at a
plateau in terms of improvement. We must ship something so we can go
after the next plateau rather than pushing a sisyphean changelog up a
few more commits only to see it steamroll us once we let go.

(Sorry to drag Plato and Sisyphus into it, but I'm only trying to keep
up with Mr. "Skylla and Charybdis" over here.)

http://github.com/paulp/scala/commit/4629f3c79b32ea8fe7ac1fc4fc86b676d38...

On Tue, Jul 6, 2010 at 5:29 PM, martin odersky <martin.odersky@epfl.ch> wrote:

On Tue, Jul 6, 2010 at 4:20 PM, iulian dragos <iulian.dragos@epfl.ch> wrote:
>
>
> On Tue, Jul 6, 2010 at 4:10 PM, martin odersky <martin.odersky@epfl.ch>
> wrote:
>>
>> I found out some more about the mechanics of the problem.
>> In phase Typer, we always load first the symbol of the enclosing class
>> (i.e. Tuppel), and only then load symbols of members. This is
>> essential because Tuppel contains type parameters that need to be
>> entered into the ClassTParams map before proceeding to members.
>>
>> When we reload that class in specialize, we do not re-load Tuppel
>> first, but instead immediately load Tuppel$1, the anonymous inner
>> class in Tuppel.get. So the ClassTParams map is empty.
>>
>> We do not re-load Tuppel's type because it has already been computed.
>> On the other hand, the anonymous inner class Tuppel$1 has not been
>> completed, in fact it does not even show up in the interface.
>
> That doesn't seem right. The inner classes should have been entered by
> 'enterOwnInnerClasses' in the classfile parser. Maybe that's the point where
> class type parameters should be attached to their lazy types, so that the
> classfile parser may get to them later, when the type is forced.
>

I believe if we would use the lazy type everything would be fine. But
we don't. Instead we prompt a full
reload of the class file (in fact we even prompt a full reload of all
enclosiung package loaders). Only this time we do not read the
enclosing class first.

I'm not sure I understand what you mean, but Adriaan has pointed out another shortcoming in the classfile parser. What happens right now is that the synthetic constructor (added by javac to allow subclassing the outer class) takes as parameter an instance of the anonymous class Tuppel$1, which can not be resolved (it is a local class, so it does not appear in the interface of Tuppel). 
Changing the test program slightly
object T {  new Tuppel()}
makes it fail in scalac 2.7.7 as well. We can fix specialization to not look at these types, and that's easy, but I'm not sure how deep is the problem in the parser (but the problem would be somehow minimized, since the above program is not well-typed anyway).
iulian 

Cheers

On Tue, Jul 6, 2010 at 9:22 AM, Paul Phillips <paulp@improving.org> wrote:

> On Tue, Jul 6, 2010 at 2:55 PM, iulian dragos <iulian.dragos@epfl.ch> wrote:
> > So the question is what to do: have a new RC, or go on with the release?

It's weird being the guy saying this, but we should release no matter
what we find now.  I mean that quite literally and unconditionally.  We
can enjoy this trickle of troublesome issues indefinitely, and then when
we finally release and see what is uncovered immediately afterward, we
will feel pretty silly for dragging it out like we were converging on
some platonic ideal.

Speaking for myself and for the Lift community, I say "Ship It" to RC7.

Yes, there is substantial technical debt in Scala and over time, we need to reduce that debt, but I don't think 2.8 can be delayed for the 3-6 months it will take to get that technical debt down to the "background noise" level.

Instead, how about planning on quarterly timed releases that each result in a net-decrease in defects and some new features.

 

RC7 basically works, it has built some big projects, and it is at a
plateau in terms of improvement.  We must ship something so we can go
after the next plateau rather than pushing a sisyphean changelog up a
few more commits only to see it steamroll us once we let go.

(Sorry to drag Plato and Sisyphus into it, but I'm only trying to keep
up with Mr. "Skylla and Charybdis" over here.)

http://github.com/paulp/scala/commit/4629f3c79b32ea8fe7ac1fc4fc86b676d384556c#L0R1841

--
Paul Phillips      | Eschew mastication.
Everyman           |
Empiricist         |
ha! spill, pupil   |----------* http://www.improving.org/paulp/ *----------

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

Adriaan is right; it's the enclosing method that needs to be looked up
for type params, the ClassTParams has nothing to do with it.
Normally, the classfile parser will never load a class like Tupel$1.
There's no need because it is strictly local. Another thing that's
strange here is that the anonymous class (and all classes and packages
around it) gets loaded at phase "parser". That looks too early.
Finally, I noted that the constructor of Tuplel has to be private;
otherwise no error occurs. Does anyone know why that makes a
difference?

On what to do: If in fact the error is only caused by a generic Java
method containing an anonymous class whose outer class has a private
constructor then I'd tend to not do a new RC for that, in particular
because it seems the problem can be worked around by
-no-specialization.

Here's something we could conceivably do to make it easier to deal
with this problem and others like it:
Introduce a field in Global named "specializing", which is set at the
beginning of specialization and reset (in a finally) at the end. In
the error method, check for "specializing", and if it is true, follow
the error message by a clause
"You can circumvent this problem by compiling with -no-specialization".

This would suggest the workaround clearly for anyone who runs into
this problem, or any other problem that might still lurk in
specialization (and I should say, these things are not really the
fault of specialization itself, but rather the way specialization
exercises hitherto untaken paths in other parts of the compiler).

Strictly speaking we should not do this without doing a new RC. But I
would be inclined to bend the rules.
Or are we exaggerating and the problem is small enough we can simply
do nothing? I'd value your opinions on this.

Cheers

On Tue, Jul 6, 2010 at 10:45 PM, martin odersky <martin.odersky@epfl.ch> wrote:

Adriaan is right; it's the enclosing method that needs to be looked up
for type params, the ClassTParams has nothing to do with it.
Normally, the classfile parser will never load a class like Tupel$1.
There's no need because it is strictly local. Another thing that's
strange here is that the anonymous class (and all classes and packages
around it) gets loaded at phase "parser". That looks too early.
Finally, I noted that the constructor of Tuplel has to be private;
otherwise no error occurs. Does anyone know why that makes a
difference?

Yes, it was in my previous mail. Because javac generates and additional, package private, constructor:
Tuple(x$1: Tuppel.Tuple$1) {  this() }
the sole purpose of this package private constructor is to allow the local class, Tuppel$1, to be constructed (it extends the outer class). I have no idea why the 'accessor' constructor takes a parameter (it never uses it) -- maybe as a security measure, that the constructor is never called from other places.
Specialization looks at all members of a class, including this constructor, and tries to complete it's type (a method type, using the 'wrong' class, which should never be part of a type at all, being local). And then it crashes.
As I mentioned earlier, the same effect can be achieved by trying to instantiate the outer class (nevermind that the constructor is private) -- crashes even in 2.7.7. 

On what to do: If in fact the error is only caused by a generic Java
method containing an anonymous class whose outer class has a private
constructor then I'd tend to not do a new RC for that, in particular
because it seems the problem can be worked around by
-no-specialization.

sounds specific enough, I agree. 

Here's something we could conceivably do to make it easier to deal
with this problem and others like it:
Introduce a field in Global named "specializing", which is set at the
beginning of specialization and reset (in a finally) at the end. In
the error method, check for "specializing", and if it is true, follow
the error message by a clause
"You can circumvent this problem by compiling with -no-specialization".

sounds reasonable. 

This would suggest the workaround clearly for anyone who runs into
this problem, or any other problem that might still lurk in
specialization (and I should say, these things are not really the
fault of specialization itself, but rather the way specialization
exercises hitherto untaken paths in other parts of the compiler).

Strictly speaking we should not do this without doing a new RC. But I
would be inclined to bend the rules.
Or are we exaggerating and the problem is small enough we can simply
do nothing? I'd value your opinions on this.

I guess it's reasonable to release with this code, without another RC, because it's only going to make a difference for errors during specialization (clearly bugs).
iulian 

Cheers

Paul Phillips wrote:
> RC7 basically works, it has built some big projects, and it is at a

Ok, I use scala in the big project. This bug is a blocker - I just cant
proceed with it at all - and I dont know what to do and I'm scared now.
Do you understand if there will be release of scala with this kind of
bug - than industry will not trust scala at all. Industry will not be
satisfied that "it's basically works". "Basically works" is a definition
of alfa stage not the release.

On Tue, Jul 06, 2010 at 10:45:31PM +0200, martin odersky wrote:
> Strictly speaking we should not do this without doing a new RC. But I
> would be inclined to bend the rules. Or are we exaggerating and the
> problem is small enough we can simply do nothing?

I think it's small enough to do nothing, but that's a weakly held
opinion. A strongly held opinion is that if the options are 1) ship rc7
2) rebuild rc8 and restart the clock and 3) ship a final which is not
identical to rc7, then option 3) is in my view a grave error. There are
hundreds of bugs either way, can this one be so important as to risk
completely borking the release? That risk is not negligible. I thought
that lesson was already learned the hard way somewhere in the 2.7
series, 2.7.1 maybe? Maybe iulian knows what I'm thinking of here.

Anyway, I'm pleading here, it can't possibly be that important, and if
it is, then let's ship 2.8.1 in six weeks. People have waited this
long, they can wait a little longer. Many of them will anyway.

Vladimir Kirichenko wrote:
> Paul Phillips wrote:
>> RC7 basically works, it has built some big projects, and it is at a

Ok, -no-spec is a lifesaver (thanks god!), but being afraid of some
correct class not being parsed is something very scary for me to use in
production environment as a release that supposed to be production ready.

On Wed, Jul 07, 2010 at 12:13:47AM +0300, Vladimir Kirichenko wrote:
> Ok, I use scala in the big project. This bug is a blocker - I just
> cant proceed with it at all - and I dont know what to do and I'm
> scared now. Do you understand if there will be release of scala with
> this kind of bug - than industry will not trust scala at all. Industry
> will not be satisfied that "it's basically works". "Basically works"
> is a definition of alfa stage not the release.

This is comically out of touch with reality. As always everyone thinks
their bug is the absolute show stopper deal breaker "my god the fate of
the world rests on this" bug. I propose that I could take a programmer
of your choosing to trac and ask them to pick the 50 most critical bugs,
and yours won't make the list.

I'm sorry if industry is waiting on this one decision to decide for all
time how trustworthy scala is. Why don't you sit down with trac for a
few hours and see if you can repeat that with a straight face.

Paul Phillips wrote:
> This is comically out of touch with reality. As always everyone thinks
> their bug is the absolute show stopper deal breaker "my god the fate of
> the world rests on this" bug. I propose that I could take a programmer
> of your choosing to trac and ask them to pick the 50 most critical bugs,
> and yours won't make the list.

So being afraid that in some point correct class file will be rejected
by scala compiler is not an important bug?

> I'm sorry if industry is waiting on this one decision to decide for all
> time how trustworthy scala is. Why don't you sit down with trac for a
> few hours and see if you can repeat that with a straight face.

There are different kind of bugs. Ones that could be solved using
workarounds by programmers and ones those could not. Compiler crashes
are sorta bugs that could hardly be solved. As for me - it's the most
scariest thing about the compiler - that it wont compile.

On Tue, Jul 6, 2010 at 11:18 PM, Paul Phillips wrote:
> On Tue, Jul 06, 2010 at 10:45:31PM +0200, martin odersky wrote:
>> Strictly speaking we should not do this without doing a new RC. But I
>> would be inclined to bend the rules. Or are we exaggerating and the
>> problem is small enough we can simply do nothing?
>
> I think it's small enough to do nothing, but that's a weakly held
> opinion.  A strongly held opinion is that if the options are 1) ship rc7
> 2) rebuild rc8 and restart the clock and 3) ship a final which is not
> identical to rc7, then option 3) is in my view a grave error.  There are
> hundreds of bugs either way, can this one be so important as to risk
> completely borking the release? That risk is not negligible.  I thought
> that lesson was already learned the hard way somewhere in the 2.7
> series, 2.7.1 maybe? Maybe iulian knows what I'm thinking of here.
>
> Anyway, I'm pleading here, it can't possibly be that important, and if
> it is, then let's ship 2.8.1 in six weeks.  People have waited this
> long, they can wait a little longer.  Many of them will anyway.

I can only support all of this. 3) should be no real option. Shipping
a bug-fixing update in the near future (< 6 months) if it should be
necessary should be the way to go.

This particular bug has two work-arounds: don't use private
constructors if you control the Java code if this bug occurs or
disable specialization if you don't control the code. And: Even if it
is unrealistic to expect that parties with big code bases participate
in every RC, this bug has been around since May (RC3), if this was so
important or widespread it should have been detected in an earlier RC.

But then, I'm not holding any stakes in the success of this particular
Scala version aside from wishing that it will become wildly
successful.

On Wed, Jul 07, 2010 at 12:33:19AM +0300, Vladimir Kirichenko wrote:
> So being afraid that in some point correct class file will be rejected
> by scala compiler is not an important bug?

I'll be waiting here for you to spend that time in trac, after which
time I hope you'll be less interested in hurting the straw people. If
you're not willing to do it, then don't pretend you have any idea which
of the 500+ open tickets are most important. The basis of your opinion
is "it's inconvenient to me" plus post-hoc rationalization. Nothing
wrong with that: only with selling it as more than it is.

Shoot, your bug isn't even the most important bug in the category of
bugs specifically described by your quote.

>>>>> "Paul" == Paul Phillips writes:

Paul> There are hundreds of bugs either way, can this one be so
Paul> important as to risk completely borking the release? That risk is
Paul> not negligible.

+1

Paul> I thought that lesson was already learned the hard way somewhere
Paul> in the 2.7 series

+1

I'm at a loss for anything else to say. Unless it's "please, for the
love of god..."

Compiler crashes are annoying, but it's easy to tell you're affected
while there's still a developer in the loop. Code generation bugs and
runtime bugs are much more serious.

- Nathan

On Wed, 2010-07-07 at 00:33 +0300, Vladimir Kirichenko wrote:
> Paul Phillips wrote:
> > This is comically out of touch with reality. As always everyone thinks
> > their bug is the absolute show stopper deal breaker "my god the fate of
> > the world rests on this" bug. I propose that I could take a programmer
> > of your choosing to trac and ask them to pick the 50 most critical bugs,
> > and yours won't make the list.
>
> So being afraid that in some point correct class file will be rejected
> by scala compiler is not an important bug?
>
> > I'm sorry if industry is waiting on this one decision to decide for all
> > time how trustworthy scala is. Why don't you sit down with trac for a
> > few hours and see if you can repeat that with a straight face.
>
> There are different kind of bugs. Ones that could be solved using
> workarounds by programmers and ones those could not. Compiler crashes
> are sorta bugs that could hardly be solved. As for me - it's the most
> scariest thing about the compiler - that it wont compile.
>

On Tue, Jul 6, 2010 at 3:02 PM, James Iry <jamesiry@gmail.com> wrote:

Paul Phillips is clearly the voice of sanity in here.  That's remarkable because he's totally batshit insane.

+n someSome {n > 57}
 

On Tue, Jul 6, 2010 at 2:18 PM, Paul Phillips <paulp@improving.org> wrote:

On Tue, Jul 06, 2010 at 10:45:31PM +0200, martin odersky wrote:
> Strictly speaking we should not do this without doing a new RC. But I
> would be inclined to bend the rules. Or are we exaggerating and the
> problem is small enough we can simply do nothing?

I think it's small enough to do nothing, but that's a weakly held
opinion.  A strongly held opinion is that if the options are 1) ship rc7
2) rebuild rc8 and restart the clock and 3) ship a final which is not
identical to rc7, then option 3) is in my view a grave error.  There are
hundreds of bugs either way, can this one be so important as to risk
completely borking the release? That risk is not negligible.  I thought
that lesson was already learned the hard way somewhere in the 2.7
series, 2.7.1 maybe? Maybe iulian knows what I'm thinking of here.

Anyway, I'm pleading here, it can't possibly be that important, and if
it is, then let's ship 2.8.1 in six weeks.  People have waited this
long, they can wait a little longer.  Many of them will anyway.

--
Paul Phillips      | A Sunday school is a prison in which children do
Everyman           | penance for the evil conscience of their parents.
Empiricist         |     -- H. L. Mencken
pal, i pill push   |----------* http://www.improving.org/paulp/ *----------

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

Paul Phillips is clearly the voice of sanity in here.  That's remarkable because he's totally batshit insane.

On Tue, Jul 6, 2010 at 2:18 PM, Paul Phillips <paulp@improving.org> wrote:

On Tue, Jul 06, 2010 at 10:45:31PM +0200, martin odersky wrote:
> Strictly speaking we should not do this without doing a new RC. But I
> would be inclined to bend the rules. Or are we exaggerating and the
> problem is small enough we can simply do nothing?

I think it's small enough to do nothing, but that's a weakly held
opinion.  A strongly held opinion is that if the options are 1) ship rc7
2) rebuild rc8 and restart the clock and 3) ship a final which is not
identical to rc7, then option 3) is in my view a grave error.  There are
hundreds of bugs either way, can this one be so important as to risk
completely borking the release? That risk is not negligible.  I thought
that lesson was already learned the hard way somewhere in the 2.7
series, 2.7.1 maybe? Maybe iulian knows what I'm thinking of here.

Anyway, I'm pleading here, it can't possibly be that important, and if
it is, then let's ship 2.8.1 in six weeks.  People have waited this
long, they can wait a little longer.  Many of them will anyway.

--
Paul Phillips      | A Sunday school is a prison in which children do
Everyman           | penance for the evil conscience of their parents.
Empiricist         |     -- H. L. Mencken
pal, i pill push   |----------* http://www.improving.org/paulp/ *----------

My recommendation to anyone using Scala in a mission critical way on large projects would be:1. Be intimately aware of the bugs in Trac2. Test all RCs thoroughly, especially early ones3. Periodically test against trunk 4. Be prepared to use your own build of trunk, because it may contain a critical fix you need5. Have someone on staff who has both the time and skills to submit patches
Scala is an astoundingly complex piece of software.  The fact that it is as stable and feature rich as it is with little or no commercial support is something close to a miracle.
If you can't do the above, I suggest you go fishing for someone willing to sell you a support contract for Scala.  I'm sure there are a number of people hanging out on this list who are just waiting for the opportunity to start such as business.  There are many companies that pay huge amounts of money for Java even though Java is free because when they encounter problems they want them fixed pronto.
All I can say is: TANSTAAFL
My more practical suggestion is to wait for a build of Scala were the bug is fixed, sometime after 2.8.0 but before 2.8.1, and use that.  With the exception of times immediately following commits of major new features (e.g. new collections and specialization) I think trunk has been so much more stable than the production releases it isn't even funny.
On Tue, Jul 6, 2010 at 5:33 PM, Vladimir Kirichenko <vladimir.kirichenko@gmail.com> wrote:

Paul Phillips wrote:
> This is comically out of touch with reality.  As always everyone thinks
> their bug is the absolute show stopper deal breaker "my god the fate of
> the world rests on this" bug.  I propose that I could take a programmer
> of your choosing to trac and ask them to pick the 50 most critical bugs,
> and yours won't make the list.

So being afraid that in some point correct class file will be rejected
by scala compiler is not an important bug?

> I'm sorry if industry is waiting on this one decision to decide for all
> time how trustworthy scala is.  Why don't you sit down with trac for a
> few hours and see if you can repeat that with a straight face.

There are different kind of bugs. Ones that could be solved using
workarounds by programmers and ones those could not. Compiler crashes
are sorta bugs that could hardly be solved. As for me - it's the most
scariest thing about the compiler - that it wont compile.

--
Best Regards,
Vladimir Kirichenko

--
http://erikengbrecht.blogspot.com/

On Wed, Jul 7, 2010 at 3:40 AM, Erik Engbrecht wrote:
> My more practical suggestion is to wait for a build of Scala were the bug is
> fixed, sometime after 2.8.0 but before 2.8.1, and use that.

I think one of the workarounds suggested elsewhere is a better idea.
Given the issues with binary compatibility, using a build from trunk
is not something I would advise unless you really know what you're
doing.

Best,
Ismael

On Tue, Jul 6, 2010 at 11:18 PM, Paul Phillips <paulp@improving.org> wrote:

On Tue, Jul 06, 2010 at 10:45:31PM +0200, martin odersky wrote:
> Strictly speaking we should not do this without doing a new RC. But I
> would be inclined to bend the rules. Or are we exaggerating and the
> problem is small enough we can simply do nothing?

I think it's small enough to do nothing, but that's a weakly held
opinion.  A strongly held opinion is that if the options are 1) ship rc7
2) rebuild rc8 and restart the clock and 3) ship a final which is not
identical to rc7, then option 3) is in my view a grave error.  There are
hundreds of bugs either way, can this one be so important as to risk
completely borking the release? That risk is not negligible.  I thought
that lesson was already learned the hard way somewhere in the 2.7
series, 2.7.1 maybe? Maybe iulian knows what I'm thinking of here.

I know, and I take that back. But then we should be consistent: if there is no final without an RC, we should continue to consider regressions as show-stoppers. Otherwise we go back to voting what warrants a fix and what does not.
iulian 

Anyway, I'm pleading here, it can't possibly be that important, and if
it is, then let's ship 2.8.1 in six weeks.  People have waited this
long, they can wait a little longer.  Many of them will anyway.

--
Paul Phillips      | A Sunday school is a prison in which children do
Everyman           | penance for the evil conscience of their parents.
Empiricist         |     -- H. L. Mencken
pal, i pill push   |----------* http://www.improving.org/paulp/ *----------

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Tue, Jul 6, 2010 at 11:00 PM, iulian dragos wrote:
>
>
> On Tue, Jul 6, 2010 at 10:45 PM, martin odersky
> wrote:
>>
>> Adriaan is right; it's the enclosing method that needs to be looked up
>> for type params, the ClassTParams has nothing to do with it.
>> Normally, the classfile parser will never load a class like Tupel$1.
>> There's no need because it is strictly local. Another thing that's
>> strange here is that the anonymous class (and all classes and packages
>> around it) gets loaded at phase "parser". That looks too early.
>> Finally, I noted that the constructor of Tuplel has to be private;
>> otherwise no error occurs. Does anyone know why that makes a
>> difference?
>
> Yes, it was in my previous mail. Because javac generates and additional,
> package private, constructor:
> Tuple(x$1: Tuppel.Tuple$1) {
>   this()
> }
> the sole purpose of this package private constructor is to allow the local
> class, Tuppel$1, to be constructed (it extends the outer class). I have no
> idea why the 'accessor' constructor takes a parameter (it never uses it) --
> maybe as a security measure, that the constructor is never called from other
> places.
> Specialization looks at all members of a class, including this constructor,
> and tries to complete it's type (a method type, using the 'wrong' class,
> which should never be part of a type at all, being local). And then it
> crashes.

OK, I see now. That looks like a pretty narrow corner case. I can
understand Vladimir is bothered by this -- anybody in whose project
this shows up would be. But delaying 2.8 even further would
inconvenience a lot of people. So, my preliminary analysis is:

- the bug is a regression wrt 2.7.7 and on that ground has to be
taken very seriously.

On the other hand

- the bug is sufficiently narrow so that not many projects will be
affected by it.
- the bug is a compiler error not a run-time crash or wrong behavior,
so in that sense less critical,
- there is a work-around in the form of -no-specialization.

Therefore I'd say it's not a blocker, and we should only decide
whether we want to put in place a better diagnostic which points to
-no-specialization (without putting out a new RC). But of course we'll
fix it with high priority in trunk.

Paul uses sometimes a bit strong language, but basically I agree with
him. There is no way to attain an ideal of bug-freeness. Even current
Java has bugs that are known and unfixed for a long time.

Cheers

On 6 Ιουλ 2010, at 20:37, David Pollak <feeder.of.the.bears@gmail.com> wrote:

On Tue, Jul 6, 2010 at 9:22 AM, Paul Phillips <paulp@improving.org> wrote:

> On Tue, Jul 6, 2010 at 2:55 PM, iulian dragos <iulian.dragos@epfl.ch> wrote:
> > So the question is what to do: have a new RC, or go on with the release?

It's weird being the guy saying this, but we should release no matter
what we find now.  I mean that quite literally and unconditionally.  We
can enjoy this trickle of troublesome issues indefinitely, and then when
we finally release and see what is uncovered immediately afterward, we
will feel pretty silly for dragging it out like we were converging on
some platonic ideal.

Speaking for myself and for the Lift community, I say "Ship It" to RC7.

Yes, there is substantial technical debt in Scala and over time, we need to reduce that debt, but I don't think 2.8 can be delayed for the 3-6 months it will take to get that technical debt down to the "background noise" level.

Instead, how about planning on quarterly timed releases that each result in a net-decrease in defects and some new features.

Let's ship it. But, maybe, we should have a discussion some time soon on how to handle release cycles that produce an 'enterprise-ready' (or 'stable') version path with many fixes and a 'dev' version path, responsible to provide new features.

 

RC7 basically works, it has built some big projects, and it is at a
plateau in terms of improvement.  We must ship something so we can go
after the next plateau rather than pushing a sisyphean changelog up a
few more commits only to see it steamroll us once we let go.

(Sorry to drag Plato and Sisyphus into it, but I'm only trying to keep
up with Mr. "Skylla and Charybdis" over here.)

http://github.com/paulp/scala/commit/4629f3c79b32ea8fe7ac1fc4fc86b676d384556c#L0R1841

--
Paul Phillips      | Eschew mastication.
Everyman           |
Empiricist         |
ha! spill, pupil   |----------* http://www.improving.org/paulp/ *----------

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

martin odersky wrote:
> OK, I see now. That looks like a pretty narrow corner case. I can
> understand Vladimir is bothered by this -- anybody in whose project
> this shows up would be. But delaying 2.8 even further would
> inconvenience a lot of people. So, my preliminary analysis is:
>
> - the bug is a regression wrt 2.7.7 and on that ground has to be
> taken very seriously.
>
> On the other hand
>
> - the bug is sufficiently narrow so that not many projects will be
> affected by it.
> - the bug is a compiler error not a run-time crash or wrong behavior,
> so in that sense less critical,
> - there is a work-around in the form of -no-specialization.

Yes. Since there is workaround I withdraw the "criticalness" of the bug.
I didn't care for this particular bug (may be there was wrong impression
about it) - my primary concern was about calling something
production-stable (and a lot of people will expect final version to be)
before knowing for sure why a bug like that appears and whether it
backstab anywhere again.

On 07/06/2010 02:18 PM, Paul Phillips wrote:
> On Tue, Jul 06, 2010 at 10:45:31PM +0200, martin odersky wrote:
>> Strictly speaking we should not do this without doing a new RC. But I
>> would be inclined to bend the rules. Or are we exaggerating and the
>> problem is small enough we can simply do nothing?
>
> I think it's small enough to do nothing, but that's a weakly held
> opinion. A strongly held opinion is that if the options are 1) ship rc7
> 2) rebuild rc8 and restart the clock and 3) ship a final which is not
> identical to rc7, then option 3) is in my view a grave error. There are
> hundreds of bugs either way, can this one be so important as to risk
> completely borking the release? That risk is not negligible. I thought
> that lesson was already learned the hard way somewhere in the 2.7
> series, 2.7.1 maybe? Maybe iulian knows what I'm thinking of here.
>
> Anyway, I'm pleading here, it can't possibly be that important, and if
> it is, then let's ship 2.8.1 in six weeks. People have waited this
> long, they can wait a little longer. Many of them will anyway.

The pain in getting 2.8 out reminds me of the pain we went through with
getting Subversion 1.5 out, which added merge-tracking as its defining
feature and took 1.75 years for us to get that release out, which is
double the normal 3/4 of a year period.

Hyrum Wright wrote a paper on our experience and how the release process
was modified. There may be something nuggets to gain from the read for
Scala:

http://www.hyrumwright.org/papers/floss2009.pdf

Regards,
Blair

On Wed, Jul 07, 2010 at 09:41:32AM -0700, Blair Zajac wrote:
> http://www.hyrumwright.org/papers/floss2009.pdf

"Subversion 1.5: A Case Study in Open Source Release Mismanagement"

Classic title! I look forward to reading it.

On Wed, Jul 07, 2010 at 10:56:17AM +0200, martin odersky wrote:
> Paul uses sometimes a bit strong language, but basically I agree with
> him. There is no way to attain an ideal of bug-freeness. Even current
> Java has bugs that are known and unfixed for a long time.

What can I say: I'm an American! We are not blessed with the European
talent for understatement, but we make up for it with our overconfidence
and selective memories.

Is that what it is?
All this time I've been thinking I had a touch of Aspergers, now I've got to find a friendly therapist willing to give a diagnosis of being a bit American.
Still, it sounds debilitating... I wonder if americanism classes as a disability when it comes to getting better parking spots :)

On 8 July 2010 04:55, Paul Phillips <paulp@improving.org> wrote:

On Wed, Jul 07, 2010 at 10:56:17AM +0200, martin odersky wrote:
> Paul uses sometimes a bit strong language, but basically I agree with
> him. There is no way to attain an ideal of bug-freeness. Even current
> Java has bugs that are known and unfixed for a long time.

What can I say: I'm an American! We are not blessed with the European
talent for understatement, but we make up for it with our overconfidence
and selective memories.

--
Paul Phillips      | Giving every man a vote has no more made men wise
Imperfectionist    | and free than Christianity has made them good.
Empiricist         |     -- H. L. Mencken
pal, i pill push   |----------* http://www.improving.org/paulp/ *----------

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

On 07/08/2010 12:36 AM, Kevin Wright wrote:
> Is that what it is?
>
> All this time I've been thinking I had a touch of Aspergers, now I've
> got to find a friendly therapist willing to give a diagnosis of being a
> bit American.
>
> Still, it sounds debilitating... I wonder if americanism classes as a
> disability when it comes to getting better parking spots :)

Sure, they don't say anything when you park your Hummer on top of that
crappy little Peugeot 106 that's sitting inconveniently in the best spot ;)

On Thu, Jul 8, 2010 at 8:11 AM, Derek Chen-Becker <java@chen-becker.org> wrote:

On 07/08/2010 12:36 AM, Kevin Wright wrote:
> Is that what it is?
>
> All this time I've been thinking I had a touch of Aspergers, now I've
> got to find a friendly therapist willing to give a diagnosis of being a
> bit American.
>
> Still, it sounds debilitating... I wonder if americanism classes as a
> disability when it comes to getting better parking spots :)

Sure, they don't say anything when you park your Hummer on top of that
crappy little Peugeot 106 that's sitting inconveniently in the best spot ;)

I think you mean "spots".  Us real Amuricans always use 2 parking spaces because 1 is just not enough. ;-)  


--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics