JavaRebel + Scala interpreter

On Tue, Feb 3, 2009 at 7:54 AM, Joni Freeman <joni.freeman@ri.fi> wrote:

It seems that Scala interpreter can't be used with JavaRebel to its full
potent. JavaRebel can only pickup changes to existing methods reducing
it to same level as JVM's own hot swap. Any ideas what could be the
problem?

Note, this was first brought up in JavaRebel forums:
http://www.zeroturnaround.com/forum/topic.php?id=175

Motivation: JavaRebel and continuous compilation (auto compilation
triggered by file changes) would offer very dynamic interpreter
session.

Unfortunately it's a limitation of the Scala compiler. Because of the way it caches classes even though the new classes are loaded the compiler can't see this fact.

I believe Martin has said in the past (in a different but simpler context - to do with changing the interpreter classpath at run time) that changing the way this works is potentially very difficult.

Are you properly pointing JavaRebel at your compilation target
directory, or just using it as a -javaagent? It works both ways for
me, but it is far more effective when it knows which class die to
monitor.

Daniel

On Feb 3, 2009, at 1:54 AM, Joni Freeman wrote:

> It seems that Scala interpreter can't be used with JavaRebel to its
> full
> potent. JavaRebel can only pickup changes to existing methods reducing
> it to same level as JVM's own hot swap. Any ideas what could be the
> problem?
>
> Note, this was first brought up in JavaRebel forums:
> http://www.zeroturnaround.com/forum/topic.php?id=175
>
> Motivation: JavaRebel and continuous compilation (auto compilation
> triggered by file changes) would offer very dynamic interpreter
> session.
>
> Cheers, Joni
>
>

A limitation of re compiler?  Does that actually apply here since JavaRebel is working at runtime?
I use JavaRebel with the Scala interpreter all the time and it works basically without a hitch.  Occaisionally, something will get "stuck" and JavaRebel will lose track of exactly what has changed, resulting in a non-runnable state.  However, I haven't experienced this problem when pointing JavaRebel at the compile dir, so I'm assuming that this isn't normal behavior.

Daniel
On Feb 3, 2009, at 4:11 AM, David MacIver <david.maciver@gmail.com> wrote:

On Tue, Feb 3, 2009 at 7:54 AM, Joni Freeman < (joni [dot] freeman [at] ri [dot] fi> wrote:

It seems that Scala interpreter can't be used with JavaRebel to its full
potent. JavaRebel can only pickup changes to existing methods reducing
it to same level as JVM's own hot swap. Any ideas what could be the
problem?

Note, this was first brought up in JavaRebel forums:
http://www.zeroturnaround.com/forum/topic.php?id=175

Motivation: JavaRebel and continuous compilation (auto compilation
triggered by file changes) would offer very dynamic interpreter
session.

Unfortunately it's a limitation of the Scala compiler. Because of the way it caches classes even though the new classes are loaded the compiler can't see this fact.

I believe Martin has said in the past (in a different but simpler context - to do with changing the interpreter classpath at run time) that changing the way this works is potentially very difficult.

On Tue, Feb 03, 2009 at 08:39:59AM -0600, Daniel Spiewak wrote:
> Are you properly pointing JavaRebel at your compilation target directory,
> or just using it as a -javaagent? It works both ways for me, but it is
> far more effective when it knows which class die to monitor.

I think people need to be clearer on what it means to "work" or "not work". Here is an excerpt about the interpreter:

* In more detail, a single compiler instance is used
* to accumulate all successfully compiled or interpreted Scala code. To
* "interpret" a line of code, the compiler generates a fresh object that
* includes the line of code and which has public member(s) to export
* all variables defined by that code. To extract the result of an
* interpreted line to show the user, a second "result object" is created
* which imports the variables exported by the above object and then
* exports a single member named "result". To accomodate user expressions
* that read from variables or methods defined in previous statements, "import"
* statements are used.

I am not sure if javarebel even could fully "work" as might be expected, given that mechanism.

I can see how JavaRebel might gave trouble with reloading
"interpreter"-level classes, but we're talking about classes defined
and compiled *outside* the interpreter and just placed on its
classpath. In that sense, it shouldn't be any different from a
regular Java application.

By "works" I mean that JavaRebel reloads classes which have been
recompiled, including new methods, fields, inner classes (closures,
etc) and so on. These changes are visible within the interpreter
against both new and old instances of the reloaded classes.

Daniel

On Feb 3, 2009, at 9:08 AM, Paul Phillips wrote:

> On Tue, Feb 03, 2009 at 08:39:59AM -0600, Daniel Spiewak wrote:
>> Are you properly pointing JavaRebel at your compilation target
>> directory,
>> or just using it as a -javaagent? It works both ways for me, but
>> it is
>> far more effective when it knows which class die to monitor.
>
> I think people need to be clearer on what it means to "work" or "not
> work". Here is an excerpt about the interpreter:
>
> * In more detail, a single compiler instance is used
> * to accumulate all successfully compiled or interpreted Scala
> code. To
> * "interpret" a line of code, the compiler generates a fresh
> object that
> * includes the line of code and which has public member(s) to
> export
> * all variables defined by that code. To extract the result of an
> * interpreted line to show the user, a second "result object" is
> created
> * which imports the variables exported by the above object and then
> * exports a single member named "result". To accomodate user
> expressions
> * that read from variables or methods defined in previous
> statements, "import"
> * statements are used.
>
> I am not sure if javarebel even could fully "work" as might be
> expected, given that mechanism.
>

Hi Daniel,

I'm thrilled to hear that this is supposed to work. I pointed JavaRebel
at the target directory but no luck. This is how I started the Scala
interpreter:

export JAVA_OPTS="-javaagent:/usr/local/java/javarebel-1.2/javarebel.jar
-noverify"
scala -Drebel.dirs=target/classes -cp target/classes

Then added a new method (bar) to an existing class (Foo) and recompiled.
I checked that modified class file appeared at target/classes. But, the
added method is still not visible in the interpreter session. I can see
that JavaRebel reloads the class definition though:
JavaRebel: Reloading class 'test.Foo'

scala> new test.Foo().bar
:5: error: value bar is not a member of test.Foo

I tried with JavaRebel 1.3 and 2.0-m1 (scala 2.7.3).

Thanks, Joni

On Tue, 2009-02-03 at 08:39 -0600, Daniel Spiewak wrote:
> Are you properly pointing JavaRebel at your compilation target
> directory, or just using it as a -javaagent? It works both ways for
> me, but it is far more effective when it knows which class die to
> monitor.
>
> Daniel
>
> On Feb 3, 2009, at 1:54 AM, Joni Freeman wrote:
>
> > It seems that Scala interpreter can't be used with JavaRebel to its
> > full
> > potent. JavaRebel can only pickup changes to existing methods reducing
> > it to same level as JVM's own hot swap. Any ideas what could be the
> > problem?
> >
> > Note, this was first brought up in JavaRebel forums:
> > http://www.zeroturnaround.com/forum/topic.php?id=175
> >
> > Motivation: JavaRebel and continuous compilation (auto compilation
> > triggered by file changes) would offer very dynamic interpreter
> > session.
> >
> > Cheers, Joni
> >
> >
>

Could you post an interpreter log demonstrating what you're saying works? I find it very hard to believe given what I understand of the compiler, so I suspect I'm misunderstanding you.

On Tue, Feb 3, 2009 at 2:47 PM, Daniel Spiewak <djspiewak@gmail.com> wrote:

A limitation of re compiler?  Does that actually apply here since JavaRebel is working at runtime?
I use JavaRebel with the Scala interpreter all the time and it works basically without a hitch.  Occaisionally, something will get "stuck" and JavaRebel will lose track of exactly what has changed, resulting in a non-runnable state.  However, I haven't experienced this problem when pointing JavaRebel at the compile dir, so I'm assuming that this isn't normal behavior.

Daniel
On Feb 3, 2009, at 4:11 AM, David MacIver <david.maciver@gmail.com> wrote:

On Tue, Feb 3, 2009 at 7:54 AM, Joni Freeman <joni.freeman@ri.fijoni.freeman@ri.fi> wrote:

It seems that Scala interpreter can't be used with JavaRebel to its full
potent. JavaRebel can only pickup changes to existing methods reducing
it to same level as JVM's own hot swap. Any ideas what could be the
problem?

Note, this was first brought up in JavaRebel forums:
http://www.zeroturnaround.com/forum/topic.php?id=175

Motivation: JavaRebel and continuous compilation (auto compilation
triggered by file changes) would offer very dynamic interpreter
session.

Unfortunately it's a limitation of the Scala compiler. Because of the way it caches classes even though the new classes are loaded the compiler can't see this fact.

I believe Martin has said in the past (in a different but simpler context - to do with changing the interpreter classpath at run time) that changing the way this works is potentially very difficult.

Well, it would seem that I was wrong.  I just attempted to add a method to a class, recompile and view the changes in an already-running interpreter.  It didn't work with any configuration of JavaRebel.  This might be where I thought JavaRebel was getting confused in some of my other tests...  :-S  My appologies for the misinformation.

Turning a complete 180, I would definitely agree with the topic originator that this is a somewhat annoying.  Does anyone know off the top of their head if JavaRebel handles this case successfully with other interactive shells (JIRB, Clojure, etc)?  Assuming it does work in these shells, and at the risk of sounding naive: why can they do it while Scala can't?  Scala is a pretty powerful language, but it's not that fundamentally different from any other object-oriented / functional language.

Daniel

David MacIver wrote:

a1a662200902031037n7b475177n1bd278cc99017c03 [at] mail [dot] gmail [dot] com" type="cite">Could you post an interpreter log demonstrating what you're saying works? I find it very hard to believe given what I understand of the compiler, so I suspect I'm misunderstanding you.

On Tue, Feb 3, 2009 at 2:47 PM, Daniel Spiewak <djspiewak [at] gmail [dot] com" rel="nofollow">djspiewak@gmail.com> wrote:

A limitation of re compiler?  Does that actually apply here since JavaRebel is working at runtime?
I use JavaRebel with the Scala interpreter all the time and it works basically without a hitch.  Occaisionally, something will get "stuck" and JavaRebel will lose track of exactly what has changed, resulting in a non-runnable state.  However, I haven't experienced this problem when pointing JavaRebel at the compile dir, so I'm assuming that this isn't normal behavior.

Daniel
On Feb 3, 2009, at 4:11 AM, David MacIver <david [dot] maciver [at] gmail [dot] com" target="_blank" rel="nofollow">david.maciver@gmail.com> wrote:

On Tue, Feb 3, 2009 at 7:54 AM, Joni Freeman <joni [dot] freeman [at] ri [dot] fi" target="_blank" rel="nofollow">joni.freeman@ri.fi> wrote:

It seems that Scala interpreter can't be used with JavaRebel to its full
potent. JavaRebel can only pickup changes to existing methods reducing
it to same level as JVM's own hot swap. Any ideas what could be the
problem?

Note, this was first brought up in JavaRebel forums:
http://www.zeroturnaround.com/forum/topic.php?id=175

Motivation: JavaRebel and continuous compilation (auto compilation
triggered by file changes) would offer very dynamic interpreter
session.

Unfortunately it's a limitation of the Scala compiler. Because of the way it caches classes even though the new classes are loaded the compiler can't see this fact.

I believe Martin has said in the past (in a different but simpler context - to do with changing the interpreter classpath at run time) that changing the way this works is potentially very difficult.

I don't know if JRuby and Clojure work with javarebel, but JRuby at least supports explicit reloading of files and I assume clojure does too.

JRuby and Clojure are both dynamic languages which are designed to be able to load code at runtime.  Scala is a statically typed language which isn't. So as well as not having built in support for this sort of operations, one needs to worry about e.g. what happens when things that previously type checked no longer work. When used with a static codebase this is typically not a problem because you can usually guarantee that all the things that would be broken have also been recompiled. So it's a much harder problem.

There are definitely non-essential limiting factors in the compiler which make this more difficult than it needs to be and prevent some apparently easy partial fixes from working, but even if those were removed it would be fairly difficult to fix in a completely consistent way.

On Tue, Feb 3, 2009 at 6:51 PM, Daniel Spiewak <djspiewak@gmail.com> wrote:

Well, it would seem that I was wrong.  I just attempted to add a method to a class, recompile and view the changes in an already-running interpreter.  It didn't work with any configuration of JavaRebel.  This might be where I thought JavaRebel was getting confused in some of my other tests...  :-S  My appologies for the misinformation.

Turning a complete 180, I would definitely agree with the topic originator that this is a somewhat annoying.  Does anyone know off the top of their head if JavaRebel handles this case successfully with other interactive shells (JIRB, Clojure, etc)?  Assuming it does work in these shells, and at the risk of sounding naive: why can they do it while Scala can't?  Scala is a pretty powerful language, but it's not that fundamentally different from any other object-oriented / functional language.

Daniel

David MacIver wrote:

Could you post an interpreter log demonstrating what you're saying works? I find it very hard to believe given what I understand of the compiler, so I suspect I'm misunderstanding you.

On Tue, Feb 3, 2009 at 2:47 PM, Daniel Spiewak <djspiewak@gmail.com> wrote:

A limitation of re compiler?  Does that actually apply here since JavaRebel is working at runtime?
I use JavaRebel with the Scala interpreter all the time and it works basically without a hitch.  Occaisionally, something will get "stuck" and JavaRebel will lose track of exactly what has changed, resulting in a non-runnable state.  However, I haven't experienced this problem when pointing JavaRebel at the compile dir, so I'm assuming that this isn't normal behavior.

Daniel
On Feb 3, 2009, at 4:11 AM, David MacIver <david.maciver@gmail.com> wrote:

On Tue, Feb 3, 2009 at 7:54 AM, Joni Freeman <joni.freeman@ri.fi> wrote:

It seems that Scala interpreter can't be used with JavaRebel to its full
potent. JavaRebel can only pickup changes to existing methods reducing
it to same level as JVM's own hot swap. Any ideas what could be the
problem?

Note, this was first brought up in JavaRebel forums:
http://www.zeroturnaround.com/forum/topic.php?id=175

Motivation: JavaRebel and continuous compilation (auto compilation
triggered by file changes) would offer very dynamic interpreter
session.

Unfortunately it's a limitation of the Scala compiler. Because of the way it caches classes even though the new classes are loaded the compiler can't see this fact.

I believe Martin has said in the past (in a different but simpler context - to do with changing the interpreter classpath at run time) that changing the way this works is potentially very difficult.

Well, I'm not asking the Scala interpreter/compiler to do the hard part.  That is, I don't expect Scala to verify that things will still work once the reload happens, JavaRebel will take care of that.  I guess what I don't understand is conceptually why this doesn't work with Scala when it works just fine with Java.  I can deploy an application written in Java using JavaRebel, recompile things to add/remove/spindle methods and everything will reload quite nicely.  In fact, as I understand it, I can do the same thing with a fully-compiled Scala application.  The problem seems to be that the Scala interpreter is doing things in a weird way which obviates whatever tricks JavaRebel uses to hot-swap classes.  Again, there may be some issues of which I am not aware (that's what I'm asking), but I can't imagine that a static type system is one of them.  If it were, then hot-reloading shouldn't work on any JVM language (seeing how bytecode is a statically-typed language).

Daniel

On Tue, Feb 3, 2009 at 1:26 PM, David MacIver <david.maciver@gmail.com> wrote:

I don't know if JRuby and Clojure work with javarebel, but JRuby at least supports explicit reloading of files and I assume clojure does too.

JRuby and Clojure are both dynamic languages which are designed to be able to load code at runtime.  Scala is a statically typed language which isn't. So as well as not having built in support for this sort of operations, one needs to worry about e.g. what happens when things that previously type checked no longer work. When used with a static codebase this is typically not a problem because you can usually guarantee that all the things that would be broken have also been recompiled. So it's a much harder problem.

There are definitely non-essential limiting factors in the compiler which make this more difficult than it needs to be and prevent some apparently easy partial fixes from working, but even if those were removed it would be fairly difficult to fix in a completely consistent way.

On Tue, Feb 3, 2009 at 6:51 PM, Daniel Spiewak <djspiewak@gmail.com> wrote:

Well, it would seem that I was wrong.  I just attempted to add a method to a class, recompile and view the changes in an already-running interpreter.  It didn't work with any configuration of JavaRebel.  This might be where I thought JavaRebel was getting confused in some of my other tests...  :-S  My appologies for the misinformation.

Turning a complete 180, I would definitely agree with the topic originator that this is a somewhat annoying.  Does anyone know off the top of their head if JavaRebel handles this case successfully with other interactive shells (JIRB, Clojure, etc)?  Assuming it does work in these shells, and at the risk of sounding naive: why can they do it while Scala can't?  Scala is a pretty powerful language, but it's not that fundamentally different from any other object-oriented / functional language.

Daniel

David MacIver wrote:

Could you post an interpreter log demonstrating what you're saying works? I find it very hard to believe given what I understand of the compiler, so I suspect I'm misunderstanding you.

On Tue, Feb 3, 2009 at 2:47 PM, Daniel Spiewak <djspiewak@gmail.com> wrote:

A limitation of re compiler?  Does that actually apply here since JavaRebel is working at runtime?
I use JavaRebel with the Scala interpreter all the time and it works basically without a hitch.  Occaisionally, something will get "stuck" and JavaRebel will lose track of exactly what has changed, resulting in a non-runnable state.  However, I haven't experienced this problem when pointing JavaRebel at the compile dir, so I'm assuming that this isn't normal behavior.

Daniel
On Feb 3, 2009, at 4:11 AM, David MacIver <david.maciver@gmail.com> wrote:

On Tue, Feb 3, 2009 at 7:54 AM, Joni Freeman <joni.freeman@ri.fi> wrote:

It seems that Scala interpreter can't be used with JavaRebel to its full
potent. JavaRebel can only pickup changes to existing methods reducing
it to same level as JVM's own hot swap. Any ideas what could be the
problem?

Note, this was first brought up in JavaRebel forums:
http://www.zeroturnaround.com/forum/topic.php?id=175

Motivation: JavaRebel and continuous compilation (auto compilation
triggered by file changes) would offer very dynamic interpreter
session.

Unfortunately it's a limitation of the Scala compiler. Because of the way it caches classes even though the new classes are loaded the compiler can't see this fact.

I believe Martin has said in the past (in a different but simpler context - to do with changing the interpreter classpath at run time) that changing the way this works is potentially very difficult.

Ok. On further thought this is at least somewhat doable. However there's no reasonable world in which it would just magically work like you seem to imagine it should.

In particular, Scala support for Javarebel is (as far as I know, not having used it) 100% in line with Java support. If Java had any similar REPL functionality it would have exactly the same problem. The problem is that code that has been entered into the REPL needs to be recompiled. When you do project level reloading, the entire codebase has been recompiled and thus things have been checked against eachother. When the compilation is interactive, things are a little more problematic.

So, if a javarebel plugin were written which hooked into the interpreter and handled the recompilation of already executed code when the classes on the classpath changes, this might be made to work. There are difficulties with the compiler's caching of class info, and those need to be worked around somehow, but the approach should basically be workable. It's not trivial to implement though.

On Tue, Feb 3, 2009 at 7:32 PM, Daniel Spiewak <djspiewak@gmail.com> wrote:

Well, I'm not asking the Scala interpreter/compiler to do the hard part.  That is, I don't expect Scala to verify that things will still work once the reload happens, JavaRebel will take care of that.  I guess what I don't understand is conceptually why this doesn't work with Scala when it works just fine with Java.  I can deploy an application written in Java using JavaRebel, recompile things to add/remove/spindle methods and everything will reload quite nicely.  In fact, as I understand it, I can do the same thing with a fully-compiled Scala application.  The problem seems to be that the Scala interpreter is doing things in a weird way which obviates whatever tricks JavaRebel uses to hot-swap classes.  Again, there may be some issues of which I am not aware (that's what I'm asking), but I can't imagine that a static type system is one of them.  If it were, then hot-reloading shouldn't work on any JVM language (seeing how bytecode is a statically-typed language).

Java bytecode is a very weakly statically typed langage. In particular calls to methods are generally not type checked - they'll just fail at runtime with a NoSuchMethodError if the method doesn't exist. Further, the JVM specification explicitly says that hot reloading shouldn't work if you try to change the type signature. I don't know what Javarebel does to work around this, but hot reloading of java bytecode is definitely not a trivial issue.