optimise/partest issue summary

Paul Phillips wrote:
> OK there are two problems, one with partest, one apparently with
> -optimise.
>
> The partest problem is very annoying at this moment in time but
> shouldn't be hard to fix. When the compiler crashes at a sufficiently
> unexpected point, it looks like it's not caught, the worker vanishes,
> and everyone else stands around looking at one another.

OK, I fixed this by handling those AssertionErrors. The check-in test go
through now, but unfortunately it is hard to see which of the 2 tests
failed. In a previous version of partest, it would print a summary of
the tests that failed at the end, which I think is very helpful, because
it means people don't have to run the entire suite on their machine to
find the failing tests. (unfortunately, after the partest refactoring I
can't quickly find the place any more where one could re-add such a
summary.)

Cheers,
Philipp

That would be great. In the meantime, what is the recommended way of finding out what test caused the crash? I see no 'failed' markers anywhere, but two stack traces. I ended up using a bash script to run all files sequentially, looking for a crash.. needless to say that is annoying, time consuming, and again time consuming....
On Mon, Apr 12, 2010 at 4:38 PM, Philipp Haller <philipp.haller@epfl.ch> wrote:

Paul Phillips wrote:

OK there are two problems, one with partest, one apparently with -optimise.

The partest problem is very annoying at this moment in time but shouldn't be hard to fix.  When the compiler crashes at a sufficiently unexpected point, it looks like it's not caught, the worker vanishes, and everyone else stands around looking at one another.

OK, I fixed this by handling those AssertionErrors. The check-in test go through now, but unfortunately it is hard to see which of the 2 tests failed. In a previous version of partest, it would print a summary of the tests that failed at the end, which I think is very helpful, because it means people don't have to run the entire suite on their machine to find the failing tests. (unfortunately, after the partest refactoring I can't quickly find the place any more where one could re-add such a summary.)

Cheers,
Philipp

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

Thanks for finding the partest spot to catch more. I am expanding the
failure output right now.

On Mon, Apr 12, 2010 at 05:23:58PM +0200, Iulian Dragos wrote:
> That would be great. In the meantime, what is the recommended way of
> finding out what test caused the crash? I see no 'failed' markers
> anywhere, but two stack traces. I ended up using a bash script to run
> all files sequentially, looking for a crash.. needless to say that is
> annoying, time consuming, and again time consuming....

It's supposed to just tell you, it's the uncaught AssertionError which
was preventing this. It's very much on my list to generate a proper
report from a test run (nobody runs the test suite more than I do, I
promise) but hadn't gotten there yet.

On Mon, Apr 12, 2010 at 5:23 PM, Iulian Dragos <iulian.dragos@epfl.ch> wrote:

That would be great. In the meantime, what is the recommended way of finding out what test caused the crash? I see no 'failed' markers anywhere, but two stack traces. I ended up using a bash script to run all files sequentially, looking for a crash.. needless to say that is annoying, time consuming, and again time consuming....

I noticed that log files for the failed tests remain in the test directory, so that's ok for now..  

On Mon, Apr 12, 2010 at 4:38 PM, Philipp Haller <philipp.haller@epfl.ch> wrote:

Paul Phillips wrote:

OK there are two problems, one with partest, one apparently with -optimise.

The partest problem is very annoying at this moment in time but shouldn't be hard to fix.  When the compiler crashes at a sufficiently unexpected point, it looks like it's not caught, the worker vanishes, and everyone else stands around looking at one another.

OK, I fixed this by handling those AssertionErrors. The check-in test go through now, but unfortunately it is hard to see which of the 2 tests failed. In a previous version of partest, it would print a summary of the tests that failed at the end, which I think is very helpful, because it means people don't have to run the entire suite on their machine to find the failing tests. (unfortunately, after the partest refactoring I can't quickly find the place any more where one could re-add such a summary.)

Cheers,
Philipp

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Mon, Apr 12, 2010 at 5:41 PM, Paul Phillips <paulp@improving.org> wrote:

On Mon, Apr 12, 2010 at 05:23:58PM +0200, Iulian Dragos wrote:
> That would be great. In the meantime, what is the recommended way of
> finding out what test caused the crash? I see no 'failed' markers
> anywhere, but two stack traces. I ended up using a bash script to run
> all files sequentially, looking for a crash.. needless to say that is
> annoying, time consuming, and again time consuming....

It's supposed to just tell you, it's the uncaught AssertionError which
was preventing this.  It's very much on my list to generate a proper
report from a test run (nobody runs the test suite more than I do, I
promise) but hadn't gotten there yet.

I had the impression this has always worked, that's why I am so puzzled. I'm not blaming you or anyone else, I am just surprised we haven't hit this problem in what is pretty basic functionality of partest.
iulian 

--
Paul Phillips      | Christ died for our sins.  Dare we make his martyrdom
Stickler           | meaningless by not committing them?
Empiricist         |     -- Jules Feiffer
slap pi uphill!    |----------* http://www.improving.org/paulp/ *----------

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Mon, Apr 12, 2010 at 5:41 PM, Paul Phillips wrote:
> On Mon, Apr 12, 2010 at 05:23:58PM +0200, Iulian Dragos wrote:
>> That would be great. In the meantime, what is the recommended way of
>> finding out what test caused the crash? I see no 'failed' markers
>> anywhere, but two stack traces. I ended up using a bash script to run
>> all files sequentially, looking for a crash.. needless to say that is
>> annoying, time consuming, and again time consuming....
>
> It's supposed to just tell you, it's the uncaught AssertionError which
> was preventing this.  It's very much on my list to generate a proper
> report from a test run (nobody runs the test suite more than I do, I
> promise) but hadn't gotten there yet.

Some time ago, I started working on a patch to generate (very basic)
JUnit report xmls from test runs. I sent it to Phillip but never took
the time to finish the work on it before you started to change things.
I didn't check if it still applies (very probably not) but when you
are at generating reports you could use this as a stub for generating
JUnit reports which are actually understood and processable by Hudson:

http://github.com/jrudolph/scala/commit/c6eb47aef43cd82f13acac643fe698fa...

(Hudson seems to be a bit peculiar about the details of the xml format)

>
> --
> Paul Phillips      | Christ died for our sins.  Dare we make his martyrdom
> Stickler           | meaningless by not committing them?
> Empiricist         |     -- Jules Feiffer
> slap pi uphill!    |----------* http://www.improving.org/paulp/ *----------
>

On Mon, Apr 12, 2010 at 05:50:15PM +0200, Iulian Dragos wrote:
> I had the impression this has always worked, that's why I am so
> puzzled. I'm not blaming you or anyone else, I am just surprised we
> haven't hit this problem in what is pretty basic functionality of
> partest.

More than likely I broke it. There was a lot of exception handling
going on in partest. The behavior I was and am targetting which has
caused me much grief is when partest just stops and you don't know why.
So I tried a ton of different things trying to make sure that nothing
disappeared without saying something informative, but it isn't all the
way there yet. I fully realize it's very inconvenient right now to have
any new excitement in partest.

Only now does the real issue crystallize... on the one hand you have
these crashes in various spots which throw an angry exception. But if
you wrap and catch too much, you swallow the InterruptedExceptions
without which there will be no timeout. So now having realized that,
I'm hoping the commit I just sent gives us better outcomes. Try
something like:

echo "kalsdjfals" > test/files/script/a.scala
ant -Dpartest.script=true -Dpartest.test-timeout=5 test.suite

And you should see 1 failure and 4 timeouts, which gives me some hope
the essential issue is solved.

This of course is why I've always wanted a simple way of saying "catch
everything except that which would probably break stuff" and that was
what I had in mind for Exception, and why I had an Exception patch this
weekend, and what I also reverted because even if it wasn't causing
deadlocks it has broader implications than I can deal with right now.

On Mon, Apr 12, 2010 at 05:23:58PM +0200, Iulian Dragos wrote:
> That would be great. In the meantime, what is the recommended way of
> finding out what test caused the crash? I see no 'failed' markers
> anywhere, but two stack traces. I ended up using a bash script to run
> all files sequentially, looking for a crash.. needless to say that is
> annoying, time consuming, and again time consuming....

OK, hopefully this output (from latest scala-reports) is more to your
liking:

[partest] 1 of 1216 tests failed (elapsed time: 00:09:17)
[partest] Summary of failures:
[partest]
[ FAILED ]
[partest] error: java.lang.AssertionError: assertion failed: could not find
toMap: (x$1: scala.collection.TraversableOnce,x$2:
Predef$<:<)scala.collection.immutable.Mapinside:
[partest] List(constructor TraversableOnce$class)
[partest] one error found
[partest]