emitting classfiles with version number >= 50.0

On Tue, Feb 7, 2012 at 10:54 AM, Miguel Garcia wrote:
> I'm preparing a topic offer for a student project, to enable GenJVM to emit
> classfiles with version number >= 50.0
>
> For that I'm considering ASM [1] because in addition to being fast, allow
> multi-threading (different threads emitting different classes), it can also
> compute stack map frames on the fly [2] [3].
>
> Before moving forward with the topic offer, a question to the list: do you
> know of any previous attempts at the idea?

E.g.
https://issues.scala-lang.org/browse/SI-611

This one contains another issue that could be solved by switching to ASM:

https://issues.scala-lang.org/browse/SI-3745

On Tue, Feb 7, 2012 at 10:54 AM, Miguel Garcia wrote:
>
>
> I'm preparing a topic offer for a student project, to enable GenJVM to emit
> classfiles with version number >= 50.0
>
> For that I'm considering ASM [1] because in addition to being fast, allow
> multi-threading (different threads emitting different classes), it can also
> compute stack map frames on the fly [2] [3].
>
> Before moving forward with the topic offer, a question to the list: do you
> know of any previous attempts at the idea?
>
What's version >= 50? Is that the new locals format? Then Iulian would
know more about it.

A classfile version 50 and up loads faster because the JVM need only typecheck the typeframe information encoded (in a compressed format) as attributes, rather than perform type inference as usual [1].

Ragarding jump offsets [2], ASM optimizes them.


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/


[1] Sec. 4.10.1, Verification by Type Checking
    download.oracle.com/otn-pub/jcp/jvm-se7-mr3-approved-oth-JSpec/jvm-javase7-mr3-spec.pdf

[2] https://issues.scala-lang.org/browse/SI-3745

Hi,

As far as I know this would only apply to class not loaded via -bootcp so the standard library doesn't load faster, because it isn't checked right now, but only to third-party libraries right?

Can ASM generate all class fiel formats or only the newest version? What would be the strategy to move forward? fjbg to emit version 49.0 and ASM to emit >= 50.0?

It might make sense to not only try to benefit from the new StackMapAttribute but also from emitting invokedynamic instructions instead of using reflection for structural types (and probably closures/lambdas instead of generating class files (depends on how much of the logic depends on the VM/how much on "special" classes only available on 8.))

Bye,

Simon

On Tue, Feb 7, 2012 at 10:48 AM, Simon Ochsenreither <simon.ochsenreither@googlemail.com> wrote:

As far as I know this would only apply to class not loaded via -bootcp so the standard library doesn't load faster, because it isn't checked right now, but only to third-party libraries right?

If you use the Scala launcher, yes. If you use the Java launcher or a container like Tomcat, then -bootcp is not used, so the standard library would load faster. I would say that this is a common scenario.
Best,Ismael

On Tue, Feb 7, 2012 at 10:54, Miguel Garcia <mgarcia512@yahoo.com> wrote:

I'm preparing a topic offer for a student project, to enable GenJVM to emit classfiles with version number >= 50.0

For that I'm considering ASM [1] because in addition to being fast, allow multi-threading (different threads emitting different classes), it can also compute stack map frames on the fly [2] [3].

Before moving forward with the topic offer, a question to the list: do you know of any previous attempts at the idea?

FWIW, I've spent a couple of hours trying to rewrite GenJVM to use ASM, but I came to the conclusion that it would be a better approach to rewrite FJBG to use ASM as FJBG has a bunch of high level constructs that ASM doesn't. I haven't touched it since, though.
 

Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/


[1] http://asm.ow2.org/

[2] Sec. 3.1.5, Stack map frames, http://download.forge.objectweb.org/asm/asm4-guide.pdf

[3] an explanation of least-common-ancestors and interfaces,
  https://groups.google.com/d/msg/jvm-languages/IT6Nn9eX5VM/amiHZk_sNUwJ

On Tue, Feb 7, 2012 at 12:03 PM, Szabolcs Berecz
wrote:
> FWIW, I've spent a couple of hours trying to rewrite GenJVM to use ASM, but
> I came to the conclusion that it would be a better approach to rewrite FJBG
> to use ASM as FJBG has a bunch of high level constructs that ASM doesn't. I
> haven't touched it since, though.

Did you use the event API or the tree API of ASM? What was missing?

On Tue, Feb 7, 2012 at 12:06, Johannes Rudolph <johannes.rudolph@googlemail.com> wrote:

On Tue, Feb 7, 2012 at 12:03 PM, Szabolcs Berecz
<szabolcs.berecz@gmail.com> wrote:
> FWIW, I've spent a couple of hours trying to rewrite GenJVM to use ASM, but
> I came to the conclusion that it would be a better approach to rewrite FJBG
> to use ASM as FJBG has a bunch of high level constructs that ASM doesn't. I
> haven't touched it since, though.

Did you use the event API or the tree API of ASM? What was missing?

I didn't get that far, I was just getting used to GenJVM and fjbg when I noticed that it has these nice wrappers around jvm instructions which ASM doesn't. ASM is missing most of what is in ch.epfl.lamp.fjbg.JCode and JExtendedCode, and maybe others too. So, these would need to be rewritten on top of ASM. I figured, it would be easier to replace the core of fjbg and when everything works correctly, proceed with cleaning up the layer between ASM and GenJVM (if necessary).

Of course, I didn't get far, so don't take my word for it.
 

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

> Can ASM generate all class fiel formats or only the newest version?

All of them. It's a bytecode manipulation library.

> What would be the strategy to move forward? fjbg to emit version 49.0 and ASM to emit >= 50.0?

There's nothing fjbg can do that ASM can't.

> It might make sense to not only try to benefit from the new StackMapAttribute but also from
> emitting invokedynamic instructions instead of using reflection for structural types
> (and probably closures/lambdas instead of generating class files (depends on how much of the logic
> depends on the VM/how much on "special" classes only available on 8.))

Those issues are orthogonal to using ASM in GenJVM. ASM would be certainly no obstacle to their realization, on the contrary.

Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

On Tue, Feb 7, 2012 at 1:54 AM, Miguel Garcia wrote:
> Before moving forward with the topic offer, a question to the list: do you
> know of any previous attempts at the idea?

I've done some of it before, long ago (shortly after I wrote ssbg,
maybe.) It is in fact what drove me toward writing awesome, the name
of which is derived from asm: AweSoMe.

https://github.com/paulp/awesome/tree/older-version

(older-version is a tag I just pushed - it was in the history, but
after that I deleted most of the asm code.)

Dealing with ASM directly was so unappealing I couldn't bear to
integrate it into scalac without an intermediate scala-centric layer.
However I got hung up working on the bytecode parsing side and only
barely got started on the creation side, and I've never been able to
come back to it. There's a lot of interesting (to me) work in there
though.

I could write the below a lot better now with manifests (and other
ways, it wasn't the endpoint.)

// Generates the jvm corresponding to the following Java class:
//
// public class Example {
// public static void main (String[] args) {
// System.out.println("Hello world!");
// }
// }
class HelloAwesome extends ClassLoader with ExampleCommon {
def className = "HelloAwesome"
val code = OpenClass() { cw =>
cw initialize className
cw.defaultConstructor()

cw mainMethod { implicit mw =>
// Or you could say "java/lang/System" getStatic [...]
(classOf[System] getStatic "out" invokeVirtual "println")("Hello world!")
(classOf[System] invokeStatic "setProperty")("propfoo", "valuebar")
}

cw.instanceMethod("foo", (x: String) => x.length) { implicit mw =>
mw.visitVarInsn(ALOAD, 1)
("java/lang/String" invokeVirtual "length")()
mw visitInsn IRETURN
}
}
}

> Dealing with ASM directly was so unappealing I couldn't bear to
> integrate it into scalac without an intermediate scala-centric layer.

Early optimization :)

I'll post updates once the student project gets started. A capability I'd emphasize is using ASM to build classfiles concurrently (preparing stack maps is compute intensive so that will mesh well task parellelism). No daydreaming here: a proof of concept is already available for "parallel dead-code elimination" [1]


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/



[1] https://github.com/magarciaEPFL/scala/tree/parallelDCE

On Tue, Feb 7, 2012 at 10:57 AM, martin odersky <martin.odersky@epfl.ch> wrote:

On Tue, Feb 7, 2012 at 10:54 AM, Miguel Garcia <mgarcia512@yahoo.com> wrote:
>
>
> I'm preparing a topic offer for a student project, to enable GenJVM to emit
> classfiles with version number >= 50.0
>
> For that I'm considering ASM [1] because in addition to being fast, allow
> multi-threading (different threads emitting different classes), it can also
> compute stack map frames on the fly [2] [3].
>
> Before moving forward with the topic offer, a question to the list: do you
> know of any previous attempts at the idea?
>
What's version >= 50? Is that the new locals format? Then Iulian would
know more about it.

We had a semester project student working on it. Didn't get too far. I rewrote all of his code and it's in one of my branches in my scala fork. I can pack it up and send it your way.
ASM most probably won't be able to generate the StackMap attributes on Scala bytecode, because there are some subtyping relationships we rely on that are not explicit in the bytecode. For that we need more casts in erasure. If you get my branch, you'll see it in no time when boostrapping the compiler. It has to do with traits, and least upper bounds. Something involving
class Base {}
trait A extends Base {..}
and the fact that 'trait A' is an interface (that does *not* extends Base). It's more involved than that, but I'd need to look up the code. Just be warned it's not smooth sailing.
If you have a proof of concept, it'd be nice to check if ASM can add the stack maps on existing Scala bytecode, like the compiler itself (I guess it has that option). If that works, then we're safe.
Unrelated: why is ASM necessary for parallel code generation? Does FJBG rely on statics? I thought it could do it.
iulian 

I was aware about that problem, haven't worked out any solution yet, but I trust we can place whatever logic is needed as an override of ASM's getCommonSuperclass() which is the hook ASM provides for that purpose. I'll definitely take a deeper look before moving forward with ASM.

For those interested, here's what you need to get started solving that problem:

$ JAVA_OPTS="-XX:-FailOverToOldVerifier" . . .

abstract class Base {
  def foo = 10
}

trait T extends Base {
  def bar = 20
}

case class D1 extends T {}
case class D2 extends T {}

object Test extends Application {
  val c: Boolean = true
  def moo(x: Base) {}
  moo(if (c) new D1 else new D2)
}

> Unrelated: why is ASM necessary for parallel code generation? Does FJBG rely on statics? I thought it could do it.

Right, fjbg could also be used in a multithreaded way by devoting dedicated instances to each thread. ASM still appears justified for future-proofing purposes, otherwise we'll have keep up with future classfiles formats (and we've already fallen behind).


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

On Wed, Feb 8, 2012 at 9:09 AM, Miguel Garcia wrote:
> ASM still appears justified for future-proofing purposes, otherwise we'll
> have keep up with future classfiles formats (and we've already fallen
> behind).

Yes, ASM is self-justifying in that regard, and not only in terms of
classfile formats, but best practices in the navigation of tricky
areas like method handles.

I have no problem writing our own versions of things when the
situation warrants it (hey, we still have a private fork of jline,
though this is no longer a virtue) but given the huge amount of
necessarily scala-specific code we must write and maintain, we should
be attempting to leverage commodity software wherever we can. In this
respect, using ASM (customized if need be) should be automatic, at
least once the integration work has been performed.