Simple way to verify class files generated by scalac

Scalac has an -Xverify switch to check bytecode it generates. Not sure if that's what your're after...
- Tiark

On Jul 14, 2011, at 11:31 AM, Grzegorz Kossakowski wrote:

> Hi,
>
> Detection of broken class files (in this case: broken signatures) was scratching my itch. Any problem that crashes JDT is a blocker for Scala+GWT project.
>
> I observed that just importing Scala type that has broken class file is enough to crash JDT as it seems to do whole parsing eagerly. I came to conclusion that if I could generate java files that important Scala types from the library I could detect a fair number of errors. I decided to give give it a try and come up with badsigs.sh bash script. I must warn you that my Bash scripting capabilities are very, very limited.
>
> Anyway, I detected a few errors in the library from the 2.9.0.1 release and in latest build from trunk. See ecj-*.log files attached.
>
> The immediate conclusion that you can derive from those logs is that parallel collections are completely unusable from Eclipse at the moment.
>
> I'm wondering if there would be interest to integrate more polished version of the script with some tools that detect problems concerning Scala nightly builds.
>
> Also, If someone could point me to some good resource on Ruby/Python shell scripting I'd be very willing to learn a bit. I find bash to be fairly annoying to work with for anything more complex.
>
> I should point out that my little script doesn't put any guarantees on coverage of class files checking but this seems to be better than nothing.
>
> Thoughts?
>

On 14 July 2011 11:39, Tiark Rompf <tiark.rompf@epfl.ch> wrote:

Scalac has an -Xverify switch to check bytecode it generates. Not sure if that's what your're after...

-Xverify detects none of the problems I report here, unfortunately.
--
Grzegorz Kossakowski

On 7/14/11 2:31 AM, Grzegorz Kossakowski wrote:
> I observed that just importing Scala type that has broken class file is
> enough to crash JDT as it seems to do whole parsing eagerly. I came to
> conclusion that if I could generate java files that important Scala
> types from the library I could detect a fair number of errors. I decided
> to give give it a try and come up with badsigs.sh bash script. I must
> warn you that my Bash scripting capabilities are very, very limited.

One of the first things the script does is blindly "rm -rf src" in
whatever directory you happen to be in, which is pretty reckless.

> Also, If someone could point me to some good resource on
> Ruby/Python shell scripting I'd be very willing to learn a bit. I find
> bash to be fairly annoying to work with for anything more complex.

It's worth gaining some facility with bash. Everyone finds it annoying,
but I've found it very useful, and then maybe there would be a second
person who might touch the scala runner.

On 14 July 2011 13:29, Paul Phillips <paulp@improving.org> wrote:

On 7/14/11 2:31 AM, Grzegorz Kossakowski wrote:
> I observed that just importing Scala type that has broken class file is
> enough to crash JDT as it seems to do whole parsing eagerly. I came to
> conclusion that if I could generate java files that important Scala
> types from the library I could detect a fair number of errors. I decided
> to give give it a try and come up with badsigs.sh bash script. I must
> warn you that my Bash scripting capabilities are very, very limited.

One of the first things the script does is blindly "rm -rf src" in
whatever directory you happen to be in, which is pretty reckless.

Good point. I'll rework it a bit. 

> Also, If someone could point me to some good resource on
> Ruby/Python shell scripting I'd be very willing to learn a bit. I find
> bash to be fairly annoying to work with for anything more complex.

It's worth gaining some facility with bash.  Everyone finds it annoying,
but I've found it very useful, and then maybe there would be a second
person who might touch the scala runner.

I tend to touch things that directly affect me (like broken signatures). That's the only effective way to stay focused.
I've never had any problems with scala runner.

--
Grzegorz Kossakowski

Grzegorz, 
A standalone tool to verify bytecode is described at:   http://blogs.oracle.com/dns/entry/yet_another_offline_java_bytecode 
For custom checks, one possibility is extending a tool by Iulian:   http://github.com/dragos/dupcheck

Miguel 
http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

A previous stab at this problem: http://www.scala-lang.org/node/8150 (the replies to that post contain additional tips and how-tos). 
Miguel http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

On 14 July 2011 13:54, Miguel Garcia <mgarcia512@yahoo.com> wrote:

Grzegorz, 
A standalone tool to verify bytecode is described at:   http://blogs.oracle.com/dns/entry/yet_another_offline_java_bytecode 

Am I mistaken or they don't tackle issues with information that VM ignores (i.e. signatures)? 

For custom checks, one possibility is extending a tool by Iulian:   http://github.com/dragos/dupcheck

This looks promising. I was trying to use ASM to implement some checks but got tired working with Visitor pattern. I have enough of them in GWT.
However, I must point out that I don't have an ambition to create a comprehensive tool that verifies class files. Actually, my only problem is that compiler used in Eclipse crashes while parsing Scala-generated class files and I need to address this problem in order to move with my project.
That particular class of issues should be in interest of Scala IDE folks too because if Scala's standard library crashes JDT then there's no hope for pleasant experience for Scala IDE users.
--
Grzegorz Kossakowski

On 14 July 2011 11:31, Grzegorz Kossakowski <grzegorz.kossakowski@gmail.com> wrote:

Hi,
Detection of broken class files (in this case: broken signatures) was scratching my itch. Any problem that crashes JDT is a blocker for Scala+GWT project.

Due to various problems I decided to write a little Scala program that really parsers class files and it uses that data to generate Java files that import Scala types. 
Also, I'm running Eclipse compiler from that Scala program so I'm reusing the same JVM instance all the time which speeds up things a lot.
The whole project is https://github.com/gkossakowski/badsigs
Log for scala library from latest trunk is attached (22 errors found). Errors detected include:

• broken Signatures in various ways
• broken InnerClasses tables which manifests in errors of form The import ... cannot be resolved

--
Grzegorz Kossakowski

On 7/19/11 7:16 AM, Grzegorz Kossakowski wrote:
> The whole project is https://github.com/gkossakowski/badsigs
>
> Log for scala library from latest trunk is attached (22 errors found).

Hey, I'm completely buried right now but very happy to see someone else taking an interest in this (even if it's not exactly "interest" as much as "why is this brokenness impeding me.") Looking at your repo I have identified at least a couple specific issues:

- a case class nested inside a parameterized class receives a synthetic companion object, which references the enclosing type parameter but is not positioned as an inner class. Definitely broken. The general issue of what do about class Bar, nested object Foo and "Bar$Foo" vs. "Bar$Foo$" has already been the subject of much wailing and gnashing of teeth, and that is before the type parameters arrive on the scene. (See many closed and open bugs in jira.)

- classes defined inside package objects are broken. This was known (to me anyway) and is also visible around jira. I am not sure if this specifically is material, but it is likely to be: package-object-nested classes end up subverting the name of the package object, receiving names like

package$Repetition$ParIterator$.class
package$Repetition$ParIterator.class

The reason it's easy to imagine this breaking things is the common appearance of things like nme.PACKAGEkw:

final def isPackageObject = isModule && name == nme.PACKAGEkw && owner.isPackageClass
final def isPackageObjectClass = isModuleClass && name.toTermName == nme.PACKAGEkw && owner.isPackageClass
final def definedInPackage = owner.isPackageClass || owner.isPackageObjectClass
final def skipPackageObject: Symbol = if (isPackageObjectClass) owner else this

The issues must be fixed, but we should not in the standard library at this moment be putting classes inside package objects.

- Boy, RedBlack.Empty never misses one of these go-arounds. That issue is coming into better focus for me now. It is a nested case object, and only this is created:

RedBlack$Empty$

"RedBlack$Empty" is not created at all. Yet this appears in RedBlack's constant pool:

const #81 = Asciz ()Lscala/collection/immutable/RedBlack.Empty;;

I think this is a variation of the first issue, nested case things.

https://issues.scala-lang.org/browse/SI-4819On 19 July 2011 19:19, Paul Phillips <paulp@improving.org> wrote:

On 7/19/11 7:16 AM, Grzegorz Kossakowski wrote:
> The whole project is https://github.com/gkossakowski/badsigs
>
> Log for scala library from latest trunk is attached (22 errors found).

Hey, I'm completely buried right now but very happy to see someone else taking an interest in this (even if it's not exactly "interest" as much as "why is this brokenness impeding me.") Looking at your repo I have identified at least a couple specific issues:

Hi Paul,
I improved my tool a bit so you can do things like this:
mac-grek:badsigs grek$ cat Test.scalapackage test
abstract class Test[A] {    case class Y[B](b: B)  }mac-grek:badsigs grek$ rm -rf classes/* && scalac -d classes/ Test.scala && ~/scala/badsigs/badsigs.sh `pwd`/classes Cleaning up /Users/grek/scala/badsigs/badsigs_working_dirChecking /Users/grek/tmp/badsigs/classes:Running Main app (will generate Java files and run ecj)----------1. ERROR in /Users/grek/scala/badsigs/badsigs_working_dir/src/C0.java (at line 0) import test.Test$Y$; ^Inconsistent classfile encountered: The undefined type parameter A is referenced from within Test$Y$ ----------1 problem (1 error)
Found 1 errors
This makes it's very easy to test changes, come up with minimal test-cases unrelated to Scala library, etc. Changes has been pushed to GitHub.  

- a case class nested inside a parameterized class receives a synthetic companion object, which references the enclosing type parameter but is not positioned as an inner class.  Definitely broken.  The general issue of what do about class Bar, nested object Foo and "Bar$Foo" vs. "Bar$Foo$" has already been the subject of much wailing and gnashing of teeth, and that is before the type parameters arrive on the scene.  (See many closed and open bugs in jira.)

I filed a new ticket (SI-4819) with my test-case presented above included. Once we get fix for it we can revisit old tickets, too.
 

- classes defined inside package objects are broken.  This was known (to me anyway) and is also visible around jira.  I am not sure if this specifically is material, but it is likely to be: package-object-nested classes end up subverting the name of the package object, receiving names like

 package$Repetition$ParIterator$.class
 package$Repetition$ParIterator.class

The reason it's easy to imagine this breaking things is the common appearance of things like nme.PACKAGEkw:

   final def isPackageObject = isModule && name == nme.PACKAGEkw && owner.isPackageClass
   final def isPackageObjectClass = isModuleClass && name.toTermName == nme.PACKAGEkw && owner.isPackageClass
   final def definedInPackage  = owner.isPackageClass || owner.isPackageObjectClass
   final def skipPackageObject: Symbol = if (isPackageObjectClass) owner else this

The issues must be fixed, but we should not in the standard library at this moment be putting classes inside package objects.

I think I don't understand all subtle issues with package objects so I'll skip this issue until we get rest fixed.  

- Boy, RedBlack.Empty never misses one of these go-arounds.  That issue is coming into better focus for me now.  It is a nested case object, and only this is created:

 RedBlack$Empty$

"RedBlack$Empty" is not created at all.  Yet this appears in RedBlack's constant pool:

const #81 = Asciz       ()Lscala/collection/immutable/RedBlack<TA;>.Empty;;

I think this is a variation of the first issue, nested case things.

It's true that case objects are problematic here. I created a ticket (SI-4820) to track this particular issue so we are not distracted by complexity of RedBlack implementation.
I'll try to dig a bit more into those signatures and try to find out what's the real problem here.

--
Grzegorz Kossakowski

+1 to any work done here.   This was holding up a few key integration points I had to work around at work.

I hope to create a 'regession' suite for scala that will let us keep track of failing bugs like this without holding up general development.   What you have could be pulled in directly...

On Jul 20, 2011 8:16 AM, "Grzegorz Kossakowski" <grzegorz.kossakowski@gmail.com> wrote:
> https://issues.scala-lang.org/browse/SI-4819On 19 July 2011 19:19, Paul
> Phillips <paulp@improving.org> wrote:
>
>> On 7/19/11 7:16 AM, Grzegorz Kossakowski wrote:
>> > The whole project is https://github.com/gkossakowski/badsigs
>> >
>> > Log for scala library from latest trunk is attached (22 errors found).
>>
>> Hey, I'm completely buried right now but very happy to see someone else
>> taking an interest in this (even if it's not exactly "interest" as much as
>> "why is this brokenness impeding me.") Looking at your repo I have
>> identified at least a couple specific issues:
>>
>
> Hi Paul,
>
> I improved my tool a bit so you can do things like this:
>
> mac-grek:badsigs grek$ cat Test.scala
> package test
>
> abstract class Test[A] {
>
> case class Y[B](b: B)
>
> }
> mac-grek:badsigs grek$ rm -rf classes/* && scalac -d classes/ Test.scala &&
> ~/scala/badsigs/badsigs.sh `pwd`/classes
> Cleaning up /Users/grek/scala/badsigs/badsigs_working_dir
> Checking /Users/grek/tmp/badsigs/classes:
> Running Main app (will generate Java files and run ecj)
> ----------
> 1. ERROR in /Users/grek/scala/badsigs/badsigs_working_dir/src/C0.java (at
> line 0)
> import test.Test$Y$;
> ^
> Inconsistent classfile encountered: The undefined type parameter A is
> referenced from within Test$Y$
> ----------
> 1 problem (1 error)
>
> Found 1 errors
>
> This makes it's very easy to test changes, come up with minimal test-cases
> unrelated to Scala library, etc. Changes has been pushed to GitHub.
>
>
>> - a case class nested inside a parameterized class receives a synthetic
>> companion object, which references the enclosing type parameter but is not
>> positioned as an inner class. Definitely broken. The general issue of what
>> do about class Bar, nested object Foo and "Bar$Foo" vs. "Bar$Foo$" has
>> already been the subject of much wailing and gnashing of teeth, and that is
>> before the type parameters arrive on the scene. (See many closed and open
>> bugs in jira.)
>>
>
>
> I filed a new ticket (SI-4819 <https://issues.scala-lang.org/browse/SI-4819>)
> with my test-case presented above included. Once we get fix for it we can
> revisit old tickets, too.
>
>
>
>> - classes defined inside package objects are broken. This was known (to me
>> anyway) and is also visible around jira. I am not sure if this specifically
>> is material, but it is likely to be: package-object-nested classes end up
>> subverting the name of the package object, receiving names like
>>
>> package$Repetition$ParIterator$.class
>> package$Repetition$ParIterator.class
>>
>> The reason it's easy to imagine this breaking things is the common
>> appearance of things like nme.PACKAGEkw:
>>
>> final def isPackageObject = isModule && name == nme.PACKAGEkw &&
>> owner.isPackageClass
>> final def isPackageObjectClass = isModuleClass && name.toTermName ==
>> nme.PACKAGEkw && owner.isPackageClass
>> final def definedInPackage = owner.isPackageClass ||
>> owner.isPackageObjectClass
>> final def skipPackageObject: Symbol = if (isPackageObjectClass) owner
>> else this
>>
>> The issues must be fixed, but we should not in the standard library at this
>> moment be putting classes inside package objects.
>>
>
> I think I don't understand all subtle issues with package objects so I'll
> skip this issue until we get rest fixed.
>
>
>> - Boy, RedBlack.Empty never misses one of these go-arounds. That issue is
>> coming into better focus for me now. It is a nested case object, and only
>> this is created:
>>
>> RedBlack$Empty$
>>
>> "RedBlack$Empty" is not created at all. Yet this appears in RedBlack's
>> constant pool:
>>
>> const #81 = Asciz ()Lscala/collection/immutable/RedBlack<TA;>.Empty;;
>>
>> I think this is a variation of the first issue, nested case things.
>>
>
> It's true that case objects are problematic here. I created a ticket (
> SI-4820 <https://issues.scala-lang.org/browse/SI-4820>) to track this
> particular issue so we are not distracted by complexity of RedBlack
> implementation.
>
> I'll try to dig a bit more into those signatures and try to find out what's
> the real problem here.
>
> --
> Grzegorz Kossakowski

On 20 July 2011 16:46, Josh Suereth <joshua.suereth@gmail.com> wrote:

+1 to any work done here.   This was holding up a few key integration points I had to work around at work.

I hope to create a 'regession' suite for scala that will let us keep track of failing bugs like this without holding up general development.   What you have could be pulled in directly...

We're working on it:https://scala-webapps.epfl.ch/jenkins/view/Misc/job/scala-signatures/
--
Grzegorz Kossakowski

+ 12.  Can't wait to join the team on monday.

On Wed, Jul 20, 2011 at 10:51 AM, Grzegorz Kossakowski <grzegorz.kossakowski@gmail.com> wrote:

On 20 July 2011 16:46, Josh Suereth <joshua.suereth@gmail.com> wrote:

+1 to any work done here.   This was holding up a few key integration points I had to work around at work.

I hope to create a 'regession' suite for scala that will let us keep track of failing bugs like this without holding up general development.   What you have could be pulled in directly...

We're working on it:https://scala-webapps.epfl.ch/jenkins/view/Misc/job/scala-signatures/

:-)

On Jul 20, 2011 7:42 PM, "Josh Suereth" <joshua.suereth@gmail.com> wrote:
> + 12. Can't wait to join the team on monday.
>
> On Wed, Jul 20, 2011 at 10:51 AM, Grzegorz Kossakowski <
> grzegorz.kossakowski@gmail.com> wrote:
>
>> On 20 July 2011 16:46, Josh Suereth <joshua.suereth@gmail.com> wrote:
>>
>>> +1 to any work done here. This was holding up a few key integration
>>> points I had to work around at work.
>>>
>>> I hope to create a 'regession' suite for scala that will let us keep track
>>> of failing bugs like this without holding up general development. What you
>>> have could be pulled in directly...
>>>
>> We're working on it:
>> https://scala-webapps.epfl.ch/jenkins/view/Misc/job/scala-signatures/
>>
>> --
>> Grzegorz Kossakowski
>>
>>