Can't get remote actors to work (server throws a MatchError)

To help track this down, I extracted the scala.actor.remote source
files from my scala 2.7.3 distribution, changed the package name to
'scala.actor.remote_snej', changed the app code to import from this
new package, and compiled everything together. That way I can modify
the classes to help track down the problem. The first thing I did was
to make some of the 'catch' handlers print exception backtraces.

The weird thing is that the app now fails slightly differently, even
though everything should be exactly the same (apart from package
names.) But as before, the exception message implies that there are
two incompatible classes with the same name:
> cannot assign instance of scala.Symbol to field
> scala.actors.remote_snej.Locator.name of type scala.Symbol
...which is nonsensical unless there are two different classes named
scala.Symbol. (Full dump is below.)

This problem, like the one I got yesterday using the regular remote
actor package, seems to stem from the Java object de-serialization.
It's been years since I did any serious Java work, but IIRC this can
occur if the same class is loaded through two different ClassLoaders.
But I have no idea why that would be happening…

Which JVM versions is Scala 2.7.3 compatible with? I've got Java SE
1.5.0_16, the default installation that comes with OS X 10.5.6.
Both sides of the connection are running as the same user on the same
machine, so there's no possibility of them loading different software.

—Jens

PS: Here's the full output from running the RemotePongApp:

Info: Thread[Thread-0,5,main]: corePoolSize = 4, maxPoolSize = 256
Info: Thread[Thread-6,5,main]: waiting for new connection...
Info: created service at Node(10.0.1.105,9000)
Info: Started new Thread[Thread-7,5,main]
Info: Thread[Thread-6,5,main]: waiting for new connection...
Info: Thread[Thread-7,5,main]: caught java.lang.ClassCastException:
cannot assign instance of scala.Symbol to field
scala.actors.remote_snej.Locator.name of type scala.Symbol in instance
of scala.actors.remote_snej.Locator
at java.io.ObjectStreamClass
$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2004)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:
1184)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
1914)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
1832)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
1719)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:
1908)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:
1832)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
1719)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at
scala
.actors.remote_snej.JavaSerializer.deserialize(JavaSerializer.scala:52)
at scala.actors.remote_snej.Serializer.readObject(Serializer.scala:41)
at scala.actors.remote_snej.TcpServiceWorker.run(TcpService.scala:265)
Info: Thread[Thread-7,5,main]: terminated

On Jan 30, 2009, at 11:02 AM, I wrote:

> This problem, like the one I got yesterday using the regular remote
> actor package, seems to stem from the Java object de-serialization.
> It's been years since I did any serious Java work, but IIRC this can
> occur if the same class is loaded through two different ClassLoaders.

Looks like I was right. I've tracked the problem down to the
CustomObjectInputStream class (in JavaSerializer.scala). If I turn the
class into a no-op by commenting out its resolveClass override, the
pingpong sample runs correctly.

The purpose of this class seems to be to give a different Java
ClassLoader first crack at loading classes while unserializing objects
received from the remote Actor. By instrumenting the code, I found
that the only class that gets loaded specially is ... scala.Symbol,
the one that causes the failure. If I check the getClassLoader
property of the various classes being resolved by this method,
scala.Symbol ends up with a different one (a sun.misc.Launcher
$AppClassLoader) than all the others (a java.net.URLClassLoader).

Where did this AppClassLoader come from? it's the system class loader
as returned by ClassLoader.getSystemClassLoader (called at
RemoteActor.scala:50). I have no idea why this isn't the same as the
ClassLoader used to load the Scala classes.

As the docs say, the initial setting of the ClassLoader chain is
platform-dependent. It appears there's something about my platform or
configuration that's incompatible with whatever it is this custom
resolveClass method is trying to do. So is this actually a bug /
compatibility problem in the Scala remote-actors library itself?

—Jens

PS: Someone else feel free to chime in, especially if you understand
the remoteactor package, or ClassLoaders, better than I do. I'm not
intentionally talking to myself here! :)

Jens,

Excellent sleuthing!

i've not run into this problem with Scala actors on my platform, but ran into it in spades using CAL as they were using class loaders to resolve all kinds of resources other than classes.

Did you file a ticket? i'm particularly interested in how the Scala folks respond to this issue as i'm in the process of making some recommendations and need good anecdotes regarding responsiveness.

Best wishes,

--greg

On Fri, Jan 30, 2009 at 12:21 PM, Jens Alfke <jens@mooseyard.com> wrote:

On Jan 30, 2009, at 11:02 AM, I wrote:

This problem, like the one I got yesterday using the regular remote actor package, seems to stem from the Java object de-serialization. It's been years since I did any serious Java work, but IIRC this can occur if the same class is loaded through two different ClassLoaders.

Looks like I was right. I've tracked the problem down to the CustomObjectInputStream class (in JavaSerializer.scala). If I turn the class into a no-op by commenting out its resolveClass override, the pingpong sample runs correctly.

The purpose of this class seems to be to give a different Java ClassLoader first crack at loading classes while unserializing objects received from the remote Actor. By instrumenting the code, I found that the only class that gets loaded specially is ... scala.Symbol, the one that causes the failure. If I check the getClassLoader property of the various classes being resolved by this method, scala.Symbol ends up with a different one (a sun.misc.Launcher$AppClassLoader) than all the others (a java.net.URLClassLoader).

Where did this AppClassLoader come from? it's the system class loader as returned by ClassLoader.getSystemClassLoader (called at RemoteActor.scala:50). I have no idea why this isn't the same as the ClassLoader used to load the Scala classes.

As the docs say, the initial setting of the ClassLoader chain is platform-dependent. It appears there's something about my platform or configuration that's incompatible with whatever it is this custom resolveClass method is trying to do. So is this actually a bug / compatibility problem in the Scala remote-actors library itself?

—Jens

PS: Someone else feel free to chime in, especially if you understand the remoteactor package, or ClassLoaders, better than I do. I'm not intentionally talking to myself here! :)

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105

+1 206.650.3740

http://biosimilarity.blogspot.com

Hopefully it isn't wait until java 7 and the classloader redesign....
It definitely sounds like a bug in how scala is wiring up classloaders (or maybe it expects you to place your classes elsewhere?)

Sent from my iPhone
On Jan 30, 2009, at 3:36 PM, Meredith Gregory <lgreg.meredith@gmail.com> wrote:

Jens,

Excellent sleuthing!

i've not run into this problem with Scala actors on my platform, but ran into it in spades using CAL as they were using class loaders to resolve all kinds of resources other than classes.

Did you file a ticket? i'm particularly interested in how the Scala folks respond to this issue as i'm in the process of making some recommendations and need good anecdotes regarding responsiveness.

Best wishes,

--greg

On Fri, Jan 30, 2009 at 12:21 PM, Jens Alfke < (jens [at] mooseyard [dot] com> wrote:

On Jan 30, 2009, at 11:02 AM, I wrote:

This problem, like the one I got yesterday using the regular remote actor package, seems to stem from the Java object de-serialization. It's been years since I did any serious Java work, but IIRC this can occur if the same class is loaded through two different ClassLoaders.

Looks like I was right. I've tracked the problem down to the CustomObjectInputStream class (in JavaSerializer.scala). If I turn the class into a no-op by commenting out its resolveClass override, the pingpong sample runs correctly.

The purpose of this class seems to be to give a different Java ClassLoader first crack at loading classes while unserializing objects received from the remote Actor. By instrumenting the code, I found that the only class that gets loaded specially is ... scala.Symbol, the one that causes the failure. If I check the getClassLoader property of the various classes being resolved by this method, scala.Symbol ends up with a different one (a sun.misc.Launcher$AppClassLoader) than all the others (a java.net.URLClassLoader).

Where did this AppClassLoader come from? it's the system class loader as returned by ClassLoader.getSystemClassLoader (called at RemoteActor.scala:50). I have no idea why this isn't the same as the ClassLoader used to load the Scala classes.

As the docs say, the initial setting of the ClassLoader chain is platform-dependent. It appears there's something about my platform or configuration that's incompatible with whatever it is this custom resolveClass method is trying to do. So is this actually a bug / compatibility problem in the Scala remote-actors library itself?

—Jens

PS: Someone else feel free to chime in, especially if you understand the remoteactor package, or ClassLoaders, better than I do. I'm not intentionally talking to myself here! :)

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105

+1 206.650.3740

http://biosimilarity.blogspot.com

On Thu, Jan 29, 2009 at 03:05:41PM -0800, Jens Alfke wrote:
> So is something screwy going on with Java serialization such that there are two incompatible Node classes?

By all appearances, yes. The first failures I run into are in processMsg in NetKernel, where cases that should be
guaranteed to match, don't:

On Jan 30, 2009, at 12:36 PM, Meredith Gregory wrote:

Did you file a ticket? i'm particularly interested in how the Scala folks respond to this issue as i'm in the process of making some recommendations and need good anecdotes regarding responsiveness.

FYI, I've just filed ticket #1686. 
In searching through Trac, I found that there was a prior ticket reporting similar problems, but which was closed by the originator as being a misunderstanding (though with no explanation given.) Also, a recently checked-in RemoteActor example sets the classLoader explicitly at initialization time, implying that the author was aware of this issue.
—Jens