Scala 2.9.x's problems with the DalvikVM

Hi Paul,

I finally found time to do another proper test and it seems that at
least the example code from SI-4620 now works with the latest snapshot
(scala-compiler-2.10.0-20120105.025731-245.jar).

I checked the resource consumption with my logging dalvikvm and the
test app now seems to use only 4MB of LinearAlloc memory which is
under the current limit of 5MB. In newer versions of Android (> 4)
this limit has been increased to 8 MB so there should be some buffer
space for the next time.

I assume it was your fix which decreased resource consumption to this
level. So thanks again!

Johannes

On Sat, Dec 17, 2011 at 5:51 PM, Johannes Rudolph
wrote:
> BTW, thanks for the effort here, it's much appreciated. I did one
> superficial test with a SNAPSHOT version last week, and unfortunately
> it still isn't working. I had no time yet to rerun with my customized
> dalvik version to check how much things have improved exactly and
> what's still missing. (Not every day is the right day to get your
> mobile phone out of the continuous reboot loop caused by a broken
> dalvikvm...)
>
> On Mon, Dec 5, 2011 at 5:20 PM, Paul Phillips wrote:
>> On Mon, Dec 5, 2011 at 5:22 AM, Johannes Rudolph
>> wrote:
>>> scala> val res0 = { allIf(theClazz) }
>>> res0: Seq[String] = ...
>>> scala> res0.size
>>> res16: Int = 9223
>>>
>>> scala> res0.distinct.size
>>> res17: Int = 49
>>>
>>> 9223 ?!? What you see is multiple interface inheritance at work.
>>
>> The interface minimization is checked in; the above now produces
>>
>> scala> res0.size
>> res2: Int = 2351
>>
>> scala> res0.distinct.size
>> res3: Int = 49
>
>
>
> --
> Johannes
>
> -----------------------------------------------
> Johannes Rudolph
> http://virtual-void.net

Is LinearAlloc per-classloader or per-application? Pardon my
ignorance ...

Nathan Moos

On Jan 9, 3:42 am, Johannes Rudolph
wrote:
> Hi Paul,
>
> I finally found time to do another proper test and it seems that at
> least the example code from SI-4620 now works with the latest snapshot
> (scala-compiler-2.10.0-20120105.025731-245.jar).
>
> I checked the resource consumption with my logging dalvikvm and the
> test app now seems to use only 4MB of LinearAlloc memory which is
> under the current limit of 5MB. In newer versions of Android (> 4)
> this limit has been increased to 8 MB so there should be some buffer
> space for the next time.
>
> I assume it was your fix which decreased resource consumption to this
> level. So thanks again!
>
> Johannes
>
> On Sat, Dec 17, 2011 at 5:51 PM, Johannes Rudolph
>
>
>
>
>
> wrote:
> > BTW, thanks for the effort here, it's much appreciated. I did one
> > superficial test with a SNAPSHOT version last week, and unfortunately
> > it still isn't working. I had no time yet to rerun with my customized
> > dalvik version to check how much things have improved exactly and
> > what's still missing. (Not every day is the right day to get your
> > mobile phone out of the continuous reboot loop caused by a broken
> > dalvikvm...)
>
> > On Mon, Dec 5, 2011 at 5:20 PM, Paul Phillips wrote:
> >> On Mon, Dec 5, 2011 at 5:22 AM, Johannes Rudolph
> >> wrote:
> >>> scala> val res0 = { allIf(theClazz) }
> >>> res0: Seq[String] = ...
> >>> scala> res0.size
> >>> res16: Int = 9223
>
> >>> scala> res0.distinct.size
> >>> res17: Int = 49
>
> >>> 9223 ?!? What you see is multiple interface inheritance at work.
>
> >> The interface minimization is checked in; the above now produces
>
> >> scala> res0.size
> >> res2: Int = 2351
>
> >> scala> res0.distinct.size
> >> res3: Int = 49
>
> > --
> > Johannes
>
> > -----------------------------------------------
> > Johannes Rudolph
> >http://virtual-void.net
>
> --
> Johannes
>
> -----------------------------------------------
> Johannes Rudolphhttp://virtual-void.net

It's per application, or more exactly per VM, see [1]. No need to
apologize, I only found out by reading the sources as well.

Here's the explaining comment from the sources:
/*
* Someday, retrieve the linear alloc struct associated with a particular
* class loader. For now, always use the boostrap loader's instance.
*/

[1] https://github.com/android/platform_dalvik/blob/master/vm/LinearAlloc.cp...

On Wed, Jan 11, 2012 at 2:27 PM, Nathan Moos wrote:
> Is LinearAlloc per-classloader or per-application? Pardon my
> ignorance ...
>
> Nathan Moos
>
> On Jan 9, 3:42 am, Johannes Rudolph
> wrote:
>> Hi Paul,
>>
>> I finally found time to do another proper test and it seems that at
>> least the example code from SI-4620 now works with the latest snapshot
>> (scala-compiler-2.10.0-20120105.025731-245.jar).
>>
>> I checked the resource consumption with my logging dalvikvm and the
>> test app now seems to use only 4MB of LinearAlloc memory which is
>> under the current limit of 5MB. In newer versions of Android (> 4)
>> this limit has been increased to 8 MB so there should be some buffer
>> space for the next time.
>>
>> I assume it was your fix which decreased resource consumption to this
>> level. So thanks again!
>>
>> Johannes
>>
>> On Sat, Dec 17, 2011 at 5:51 PM, Johannes Rudolph
>>
>>
>>
>>
>>
>> wrote:
>> > BTW, thanks for the effort here, it's much appreciated. I did one
>> > superficial test with a SNAPSHOT version last week, and unfortunately
>> > it still isn't working. I had no time yet to rerun with my customized
>> > dalvik version to check how much things have improved exactly and
>> > what's still missing. (Not every day is the right day to get your
>> > mobile phone out of the continuous reboot loop caused by a broken
>> > dalvikvm...)
>>
>> > On Mon, Dec 5, 2011 at 5:20 PM, Paul Phillips wrote:
>> >> On Mon, Dec 5, 2011 at 5:22 AM, Johannes Rudolph
>> >> wrote:
>> >>> scala> val res0 = { allIf(theClazz) }
>> >>> res0: Seq[String] = ...
>> >>> scala> res0.size
>> >>> res16: Int = 9223
>>
>> >>> scala> res0.distinct.size
>> >>> res17: Int = 49
>>
>> >>> 9223 ?!? What you see is multiple interface inheritance at work.
>>
>> >> The interface minimization is checked in; the above now produces
>>
>> >> scala> res0.size
>> >> res2: Int = 2351
>>
>> >> scala> res0.distinct.size
>> >> res3: Int = 49
>>
>> > --
>> > Johannes
>>
>> > -----------------------------------------------
>> > Johannes Rudolph
>> >http://virtual-void.net
>>
>> --
>> Johannes
>>
>> -----------------------------------------------
>> Johannes Rudolphhttp://virtual-void.net

It was mentioned earlier on this forum that a Google representative
said that multiple DEX files would solve the problem, is that invalid?

Nathan Moos

On Jan 11, 6:33 am, Johannes Rudolph
wrote:
> It's per application, or more exactly per VM, see [1]. No need to
> apologize, I only found out by reading the sources as well.
>
> Here's the explaining comment from the sources:
> /*
>  * Someday, retrieve the linear alloc struct associated with a particular
>  * class loader.  For now, always use the boostrap loader's instance.
>  */
>
> [1]https://github.com/android/platform_dalvik/blob/master/vm/LinearAlloc...
>
>
>
>
>
>
>
>
>
> On Wed, Jan 11, 2012 at 2:27 PM, Nathan Moos wrote:
> > Is LinearAlloc per-classloader or per-application? Pardon my
> > ignorance ...
>
> > Nathan Moos
>
> > On Jan 9, 3:42 am, Johannes Rudolph
> > wrote:
> >> Hi Paul,
>
> >> I finally found time to do another proper test and it seems that at
> >> least the example code from SI-4620 now works with the latest snapshot
> >> (scala-compiler-2.10.0-20120105.025731-245.jar).
>
> >> I checked the resource consumption with my logging dalvikvm and the
> >> test app now seems to use only 4MB of LinearAlloc memory which is
> >> under the current limit of 5MB. In newer versions of Android (> 4)
> >> this limit has been increased to 8 MB so there should be some buffer
> >> space for the next time.
>
> >> I assume it was your fix which decreased resource consumption to this
> >> level. So thanks again!
>
> >> Johannes
>
> >> On Sat, Dec 17, 2011 at 5:51 PM, Johannes Rudolph
>
> >> wrote:
> >> > BTW, thanks for the effort here, it's much appreciated. I did one
> >> > superficial test with a SNAPSHOT version last week, and unfortunately
> >> > it still isn't working. I had no time yet to rerun with my customized
> >> > dalvik version to check how much things have improved exactly and
> >> > what's still missing. (Not every day is the right day to get your
> >> > mobile phone out of the continuous reboot loop caused by a broken
> >> > dalvikvm...)
>
> >> > On Mon, Dec 5, 2011 at 5:20 PM, Paul Phillips wrote:
> >> >> On Mon, Dec 5, 2011 at 5:22 AM, Johannes Rudolph
> >> >> wrote:
> >> >>> scala> val res0 = { allIf(theClazz) }
> >> >>> res0: Seq[String] = ...
> >> >>> scala> res0.size
> >> >>> res16: Int = 9223
>
> >> >>> scala> res0.distinct.size
> >> >>> res17: Int = 49
>
> >> >>> 9223 ?!? What you see is multiple interface inheritance at work.
>
> >> >> The interface minimization is checked in; the above now produces
>
> >> >> scala> res0.size
> >> >> res2: Int = 2351
>
> >> >> scala> res0.distinct.size
> >> >> res3: Int = 49
>
> >> > --
> >> > Johannes
>
> >> > -----------------------------------------------
> >> > Johannes Rudolph
> >> >http://virtual-void.net
>
> >> --
> >> Johannes
>
> >> -----------------------------------------------
> >> Johannes Rudolphhttp://virtual-void.net
>
> --
> Johannes
>
> -----------------------------------------------
> Johannes Rudolphhttp://virtual-void.net

I think we have at least three different issues here:

1.) The dex-file binary format can only handle 65k methods. This is
what could be handled by multiple dex files. This is currently worked
around by using proguard or by pre-installing the Scala libraries to
the developer phone.

http://code.google.com/p/android/issues/detail?id=20814

2.) A deep interface hierarchy can lead to excessive LinearAlloc use
in dalvik vm. Scala's collection library has quite a "deep interface
hierarchy" and triggered this problem. The situation will be improved
on the Scala side by omitting redundant interface declarations (Scala
trunk version is already much better in this regard). Maybe this will
improved on the Dalvik side if the proposed fix will be accepted.

http://code.google.com/p/android/issues/detail?id=22586

3.) The problem Alexey reported earlier in this thread where in a
former Scala 2.10-SNAPSHOT version an `invokeinterface` bytecode is
generated for calls to methods defined in `java.lang.Object` for some
cases
which works in Hotspot but fails in Dalvik. This is now fixed in Scala.

Johannes

On Wed, Jan 11, 2012 at 8:11 PM, Nathan Moos wrote:
> It was mentioned earlier on this forum that a Google representative
> said that multiple DEX files would solve the problem, is that invalid?
>
> Nathan Moos
>
> On Jan 11, 6:33 am, Johannes Rudolph
> wrote:
>> It's per application, or more exactly per VM, see [1]. No need to
>> apologize, I only found out by reading the sources as well.
>>
>> Here's the explaining comment from the sources:
>> /*
>>  * Someday, retrieve the linear alloc struct associated with a particular
>>  * class loader.  For now, always use the boostrap loader's instance.
>>  */
>>
>> [1]https://github.com/android/platform_dalvik/blob/master/vm/LinearAlloc...
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Jan 11, 2012 at 2:27 PM, Nathan Moos wrote:
>> > Is LinearAlloc per-classloader or per-application? Pardon my
>> > ignorance ...
>>
>> > Nathan Moos
>>
>> > On Jan 9, 3:42 am, Johannes Rudolph
>> > wrote:
>> >> Hi Paul,
>>
>> >> I finally found time to do another proper test and it seems that at
>> >> least the example code from SI-4620 now works with the latest snapshot
>> >> (scala-compiler-2.10.0-20120105.025731-245.jar).
>>
>> >> I checked the resource consumption with my logging dalvikvm and the
>> >> test app now seems to use only 4MB of LinearAlloc memory which is
>> >> under the current limit of 5MB. In newer versions of Android (> 4)
>> >> this limit has been increased to 8 MB so there should be some buffer
>> >> space for the next time.
>>
>> >> I assume it was your fix which decreased resource consumption to this
>> >> level. So thanks again!
>>
>> >> Johannes
>>
>> >> On Sat, Dec 17, 2011 at 5:51 PM, Johannes Rudolph
>>
>> >> wrote:
>> >> > BTW, thanks for the effort here, it's much appreciated. I did one
>> >> > superficial test with a SNAPSHOT version last week, and unfortunately
>> >> > it still isn't working. I had no time yet to rerun with my customized
>> >> > dalvik version to check how much things have improved exactly and
>> >> > what's still missing. (Not every day is the right day to get your
>> >> > mobile phone out of the continuous reboot loop caused by a broken
>> >> > dalvikvm...)
>>
>> >> > On Mon, Dec 5, 2011 at 5:20 PM, Paul Phillips wrote:
>> >> >> On Mon, Dec 5, 2011 at 5:22 AM, Johannes Rudolph
>> >> >> wrote:
>> >> >>> scala> val res0 = { allIf(theClazz) }
>> >> >>> res0: Seq[String] = ...
>> >> >>> scala> res0.size
>> >> >>> res16: Int = 9223
>>
>> >> >>> scala> res0.distinct.size
>> >> >>> res17: Int = 49
>>
>> >> >>> 9223 ?!? What you see is multiple interface inheritance at work.
>>
>> >> >> The interface minimization is checked in; the above now produces
>>
>> >> >> scala> res0.size
>> >> >> res2: Int = 2351
>>
>> >> >> scala> res0.distinct.size
>> >> >> res3: Int = 49
>>
>> >> > --
>> >> > Johannes
>>
>> >> > -----------------------------------------------
>> >> > Johannes Rudolph
>> >> >http://virtual-void.net
>>
>> >> --
>> >> Johannes
>>
>> >> -----------------------------------------------
>> >> Johannes Rudolphhttp://virtual-void.net
>>
>> --
>> Johannes
>>
>> -----------------------------------------------
>> Johannes Rudolphhttp://virtual-void.net

IMHO I find one more problem place.

I/dalvikvm( 618): Could not find method
sun.misc.Unsafe.throwException, referenced from method
scala.concurrent.forkjoin.ForkJoinTask.rethrowException
W/dalvikvm( 618): VFY: unable to resolve virtual method 26641: Lsun/
misc/Unsafe;.throwException (Ljava/lang/Throwable;)V
D/dalvikvm( 618): VFY: replacing opcode 0x6e at 0x0004

Has dalvik proper support of sun.misc.Unsafe?

Best regards,
Alexey

On Sat, Jan 14, 2012 at 9:41 AM, Alexey Aksenov <alexey.ezh@gmail.com> wrote:

IMHO I find one more problem place.

I/dalvikvm(  618): Could not find method
sun.misc.Unsafe.throwException, referenced from method
scala.concurrent.forkjoin.ForkJoinTask.rethrowException
W/dalvikvm(  618): VFY: unable to resolve virtual method 26641: Lsun/
misc/Unsafe;.throwException (Ljava/lang/Throwable;)V
D/dalvikvm(  618): VFY: replacing opcode 0x6e at 0x0004

Try setting system property actors.enableForkJoin=false .  (Not suggesting as a long term answer.)

There's another possible solution:

https://github.com/jrudolph/scala-android-libs/issues/3

Can you try that and report back?

Johannes

On Sat, Jan 14, 2012 at 7:36 PM, Paul Phillips wrote:
>
>
> On Sat, Jan 14, 2012 at 9:41 AM, Alexey Aksenov
> wrote:
>>
>> IMHO I find one more problem place.
>>
>> I/dalvikvm(  618): Could not find method
>> sun.misc.Unsafe.throwException, referenced from method
>> scala.concurrent.forkjoin.ForkJoinTask.rethrowException
>> W/dalvikvm(  618): VFY: unable to resolve virtual method 26641: Lsun/
>> misc/Unsafe;.throwException (Ljava/lang/Throwable;)V
>> D/dalvikvm(  618): VFY: replacing opcode 0x6e at 0x0004
>
>
> Try setting system property actors.enableForkJoin=false .  (Not suggesting
> as a long term answer.)
>

On Sat, Jan 14, 2012 at 7:36 PM, Paul Phillips
wrote:> On Sat, Jan 14, 2012 at 9:41 AM, Alexey Aksenov
>> IMHO I find one more problem place.>>>>
I/dalvikvm(  618): Could not find method>>
sun.misc.Unsafe.throwException, referenced from method>>
scala.concurrent.forkjoin.ForkJoinTask.rethrowException>> W/dalvikvm(
618): VFY: unable to resolve virtual method 26641: Lsun/>>
misc/Unsafe;.throwException (Ljava/lang/Throwable;)V>> D/dalvikvm(
618): VFY: replacing opcode 0x6e at 0x0004>>> Try setting system
property actors.enableForkJoin=false .  (Not suggesting> as a long
term answer.)

Did you test this solution, Alexey?

I don't think it will work since it looks like this being a
verification problem. Apache Harmony seems to be missing
Unsafe.throwException altogether. If setting the property alone isn't
working can you try removing the complete "scala.concurrent.forkjoin"
package from your Scala jar?

Hello, Johannes

setting system property actors.enableForkJoin=false inappropriate. I
tried different combinations.

For example application activity singleton (that loading first) as
simple as:

package org.digimead.digi.inetd.sshd

import org.slf4j.LoggerFactory

class SSHDApplication extends android.app.Application {
protected val log =
LoggerFactory.getLogger(getClass.getName().replaceFirst("org.digimead.digi.inetd",
"o.d.d.i"))
System.setProperty("actors.enableForkJoin", "false")
log.debug("alive")
}

logcat:

I/ActivityManager( 78): Start proc org.digimead.digi.inetd.sshd for
activity org.digimead.digi.inetd.sshd/.SSHDActivity: pid=621 uid=10040
gids={3003}
I/dalvikvm( 621): Turning on JNI app bug workarounds for target SDK
version 7...
D/dalvikvm( 37): GC_EXPLICIT freed 37K, 4% free 9030K/9347K, paused
2ms+17ms
W/NetworkManagementSocketTagger( 78): setKernelCountSet(10040, 1)
failed with errno -2
D/dalvikvm( 37): GC_EXPLICIT freed <1K, 4% free 9031K/9347K, paused
2ms+78ms
I/AndroidLoggerFactory( 621): Logger name
'o.d.d.i.sshd.SSHDApplication' exceeds maximum length of 23
characters, using 'o.d.d.i.s*.SSHDApplica*' instead.