- About Scala
- Documentation
- Code Examples
- Software
- Scala Developers
protected lessons
Sun, 2010-10-17, 23:27
Hi Paul,
The protected example (#3871) showed very nicely the problems we are facing with the compiler. First, we need more experimentation and more tests. It was only your experiments that unearthed the bug. The code was clearly and utterly nonsensical before.
But then it gets difficult. I believe a purely test-driven approach (i.e. put change in, watch if something fails, if one more test case passes accept the change) is doomed when dealing with the compiler. There are simply too many scenarios. To give some perspective: When I was writing javac, Sun had a test suite (constructed by a Russian team over several years) of some 15000 tests. I am sure it is more like 100000 today. Those 15000 tests caught some sorts of errors but not the interesting ones -- those showed up in the build of JDK itself. The upshot is that even with 100K tests we cannot hope to get a decent test coverage that matches the bugs our users are finding. Conversely, passing all the tests we have says little when it comes to changes in the type checker.
So what recourse do we have? We need more tests to uncover bugs. But we cannot use the same tests (alone) to give us confidence that a fix is correct. All code in the compiler frontend should have a recognizable connection to the spec and the two have to evolve in parallel. Because tests are so weak, the correctness argument must lean more on spec correspondence than anything else. If we ignore the spec, we will just add patches over patches, and will lose the clear line of what's in the compiler (yes, this line is already sometimes hard to discern, so the ambition should be to make it clearer!)
In any case, you had the right intuition not checking in the change. I just wanted to write down my observations because they illustrate an important, and somewhat non-intuitive, aspect of compiler writing, I think.
Cheers
-- Martin
The protected example (#3871) showed very nicely the problems we are facing with the compiler. First, we need more experimentation and more tests. It was only your experiments that unearthed the bug. The code was clearly and utterly nonsensical before.
But then it gets difficult. I believe a purely test-driven approach (i.e. put change in, watch if something fails, if one more test case passes accept the change) is doomed when dealing with the compiler. There are simply too many scenarios. To give some perspective: When I was writing javac, Sun had a test suite (constructed by a Russian team over several years) of some 15000 tests. I am sure it is more like 100000 today. Those 15000 tests caught some sorts of errors but not the interesting ones -- those showed up in the build of JDK itself. The upshot is that even with 100K tests we cannot hope to get a decent test coverage that matches the bugs our users are finding. Conversely, passing all the tests we have says little when it comes to changes in the type checker.
So what recourse do we have? We need more tests to uncover bugs. But we cannot use the same tests (alone) to give us confidence that a fix is correct. All code in the compiler frontend should have a recognizable connection to the spec and the two have to evolve in parallel. Because tests are so weak, the correctness argument must lean more on spec correspondence than anything else. If we ignore the spec, we will just add patches over patches, and will lose the clear line of what's in the compiler (yes, this line is already sometimes hard to discern, so the ambition should be to make it clearer!)
In any case, you had the right intuition not checking in the change. I just wanted to write down my observations because they illustrate an important, and somewhat non-intuitive, aspect of compiler writing, I think.
Cheers
-- Martin
Treating the compiler as a natural phenomenon which one only discovers
empirically is not a route which I pursue with any enthusiasm. Trust me
on this! We have no disagreement regarding the ideal way to do things,
but sometimes I can't for lack of information or understanding. But I
usually know what I don't know, so the threat is contained. All I meant
with my comment in that ticket is that if I construct a black box
solution which empirically solves the problem, if it's not an acceptable
solution then if it's anything short of hopeless I'd appreciate some
details so I can salvage something from the effort.
Or, equally as good is for you to dive in, sprinkling in some comments
along the way, as in the outcome we got. Thanks for fixing it.