Native Scala base classes to help porting Scala to other platforms

On 6/29/11 2:01 PM, Simon Ochsenreither wrote:
> Conceptually, this is remotely related to scala-aux ("Source files
> under this directory cannot be compiled by normal means. They exist
> for bootstrapping and documentation purposes.").

Another connected problem we have right now is with eclipse. I'm not
the guy to explain it, but apparently we need to fully segregate the
low-level files (like scala-aux, and other files which have similar
qualities) from the rest of the library in order to be able to develop
the library in eclipse.

So in other words, regardless of the merits for other platforms, we need
a cleaner separation of levels on THIS platform.

Hi Paul,

> Another connected problem we have right now is with eclipse. I'm not
> the guy to explain it, but apparently we need to fully segregate the
> low-level files (like scala-aux, and other files which have similar
> qualities) from the rest of the library in order to be able to develop
> the library in eclipse.
>
Just wondering, how deeply are things like Scala's Double and
java.lang.Double intertwined?

> So in other words, regardless of the merits for other platforms, we need
> a cleaner separation of levels on THIS platform.

So is this a "Go ahead!" or "Meh"? :-)

Thanks and bye,

Simon

Hi, 
> If there was a common place where people working on ports> to different platforms could share some efforts> (I assume every port needs various changes here and there),> the time necessary to get Scala running on said> platforms could be reduced and would probably improve> implementation quality on non-JVM platforms. 
All right, the topic is deep. There are several degrees of "platform-support" (talking from the Scala.NET experience) (1) cross-compilation,(2) platform-specific standard library,(3) bootstrapped compiler Even for Level 1, it takes conscious effort avoiding sprinkling the compiler sources with "forMSIL" and "forJVM". The good news is that in this regard the compiler is doing quite well. Level 2: realistically, some subset (to be determined) of the library can be hoped to be maintained across platforms (unless of course huge resources are devoted to the endeavor). Level 3: the compiler sources use JDK or Standard library functionality, depending on what's more convenient. Progressively relying more and more on the library would simplify porting efforts.  As always, the problem boils down to manpower ...   Miguelhttp://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/
 
 

Hi Miguel,

> There are several degrees of "platform-support" (talking from the Scala.NET
> experience)

Imho Scala.NET has some special position here, because it more or less
runs with IKVM and can use IKVM's Java compatibility class library.

This makes it certainly easier and more efficient to implement and
reduces the need to have "native" Scala implementations of it. But the
advances the IKVM team makes implementing Java classes unfortunately
don't help other Scala ports.

In the end I think it would be easier to have the base classes in
Scala than reimplementing them for every platform (e. g. base classes
written in Java, C#, C, Oz, JavaScript ...).

> (1) cross-compilation,
> (2) platform-specific standard library,
> (3) bootstrapped compiler
>
> Even for Level 1, it takes conscious effort avoiding sprinkling the compiler
> sources with "forMSIL" and "forJVM". The good news is that in this regard
> the compiler is doing quite well.

That is something I'm not able to do and even if I was skilled enough
_and_ had patches already written, it wouldn't probably be merged
until christmas. :-)

> As always, the problem boils down to manpower ...

That's why I asked if there is interest in it in me doing it. :-)
I'm basically saying "I will work for free, but please tell me
beforehand if there is no interest in it or if someone in the Scala
team is working on something similar, so that efforts don't get
wasted/duplicated."

But I'm not able to extract any useful information from the Scala dev
team regarding that question.

It seems that Paul is often the only one who both seems to have some
connections to the "decision circle" _and_ the desire to communicate
these decisions with the wider "community". And Paul has certainly
better things to do than answering my questions, that's why I always
feel bad when he decides to spend time answering them.

If there is any trick you learned to better communicate with the Scala
devs, please tell me. Maybe I'm just doing it totally wrong.

Thanks and bye,

Simon

Simon, 
> Imho Scala.NET has some special position here, > because it more or less runs with IKVM and can use > IKVM's Java compatibility class library.
That's true. 

> In the end I think it would be easier to have > the base classes in Scala than reimplementing them > for every platform (e. g. base classes  > written in Java, C#, C, Oz, JavaScript ...).
The base classes. Can we recognize them when we see them? In some cases, yes: the compiler knows there's an Object.syncrhonized() method, and also knows how to map it to each platform. Whether something is in the "base classes" or not requires therefore some whitelisting and bookkeeping and (sometimes library only, sometimes compiler) implementation effort. 

> If there is any trick you learned to better communicate > with the Scala devs, please tell me. 
I'm part of the Scala team at EPFL! That's why! 

Coming back to the "base classes" point, which I guess is great because a lot can be accomplished there without touching the compiler. It's a great idea but I don't know how to whitelist what should be part of the "base classes" (for platforms other than JVM and .NET). And that's what makes the problem interesting (the "other" platforms, whatever they are). 
This is where the non-JVM non-MSIL platform experts should jump into the discussion ... 

Miguel lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

Out of curiosity, in what way is LLVM + VMKit any *less* privileged?


On 30 June 2011 15:31, Miguel Garcia <mgarcia512@yahoo.com> wrote:

Simon, 
> Imho Scala.NET has some special position here,  > because it more or less runs with IKVM and can use > IKVM's Java compatibility class library.
That's true. 

> In the end I think it would be easier to have > the base classes in Scala than reimplementing them > for every platform (e. g. base classes  > written in Java, C#, C, Oz, JavaScript ...).
The base classes. Can we recognize them when we see them? In some cases, yes: the compiler knows there's an Object.syncrhonized() method, and also knows how to map it to each platform. Whether something is in the "base classes" or not requires therefore some whitelisting and bookkeeping and (sometimes library only, sometimes compiler) implementation effort. 

> If there is any trick you learned to better communicate > with the Scala devs, please tell me. 
I'm part of the Scala team at EPFL! That's why! 

Coming back to the "base classes" point, which I guess is great because a lot can be accomplished there without touching the compiler. It's a great idea but I don't know how to whitelist what should be part of the "base classes" (for platforms other than JVM and .NET). And that's what makes the problem interesting (the "other" platforms, whatever they are). 
This is where the non-JVM non-MSIL platform experts should jump into the discussion ... 

Miguel  lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

Hi Miguel,

> The base classes. Can we recognize them when we see them? In some cases,
> yes: the compiler knows there's an Object.syncrhonized() method, and also
> knows how to map it to each platform. Whether something is in the "base
> classes" or not requires therefore some whitelisting and bookkeeping and
> (sometimes library only, sometimes compiler) implementation effort.

Yes, I have thought about that. My efforts are intended to be more
centered around those things which can work without patching the
compiler.

In the end, every platform needs to do some special tuning or
rewriting, so having common base classes is certainly not the 100%
solution.

But imho it is an important difference if the people porting Scala can
just grab a common, well-tested and maintained library and do their
adjustments as they see fit or if everyone is basically forced to
reimplement the whole thing _again_.

> I'm part of the Scala team at EPFL! That's why!

Ah ok. :-) I always thought you worked more or less for Microsoft (or
is MS just sponsoring it?) ... in my mind you were always that guy
doing that crazy MSIL stuff. Btw, the compiler corner is a great idea.
I love reading these kind of things.

> Coming back to the "base classes" point, which I guess is great because a
> lot can be accomplished there without touching the compiler. It's a great
> idea but I don't know how to whitelist what should be part of the "base
> classes" (for platforms other than JVM and .NET). And that's what makes the
> problem interesting (the "other" platforms, whatever they are).

I agree, although white-listing has only to happen on platforms which
have a choice. I'm not sure if LLVM and Ozma have really that.

> This is where the non-JVM non-MSIL platform experts should jump into the
> discussion ...

Yes, maybe Geoff and Sébastien can comment further...

Thanks and bye,

Simon

Kevin, 
LLVM + VMKit must be as amenable a platform to target as any other, which brings me back to what I was hinting at: I don't have development experience with that platform. 
Miguel http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

Hi everyone!

I'll answer the question "What are base classes?" from my non-JVM non-MSIL "expert" point of view ;-)

Base classes are those classes of the Java standard library (rt.jar), that are directly or indirectly used by the Scala standard library, or even the compiler.
Object, Class, String, System, are obviously part of these. But there is also java.util.Properties, Java collections (used by Properties e.g.), Big numbers, etc. Input/output classes are also important.

I do not consider special methods (like Object.synchronized) to define base classes. This is compiler-related. Base classes, as we understand them, are exclusively library-related, though they contain @native methods here and there.

Sébastien

On Thu, Jun 30, 2011 at 16:47, Simon Ochsenreither <simon@ochsenreither.de> wrote:

Hi Miguel,

The base classes. Can we recognize them when we see them? In some cases,
yes: the compiler knows there's an Object.syncrhonized() method, and also
knows how to map it to each platform. Whether something is in the "base
classes" or not requires therefore some whitelisting and bookkeeping and
(sometimes library only, sometimes compiler) implementation effort.

Yes, I have thought about that. My efforts are intended to be more centered around those things which can work without patching the compiler.

In the end, every platform needs to do some special tuning or rewriting, so having common base classes is certainly not the 100% solution.

But imho it is an important difference if the people porting Scala can just grab a common, well-tested and maintained library and do their adjustments as they see fit or if everyone is basically forced to reimplement the whole thing _again_.

Coming back to the "base classes" point, which I guess is great because a
lot can be accomplished there without touching the compiler. It's a great
idea but I don't know how to whitelist what should be part of the "base
classes" (for platforms other than JVM and .NET). And that's what makes the
problem interesting (the "other" platforms, whatever they are).

I agree, although white-listing has only to happen on platforms which have a choice. I'm not sure if LLVM and Ozma have really that.

This is where the non-JVM non-MSIL platform experts should jump into the
discussion ...

Yes, maybe Geoff and Sébastien can comment further...

Thanks and bye,


Simon

In my mind, the goal end state is to have no references to the java
package in core library subset and compiler sources.

I can see this happening in one of two ways right now:

1. Essentially remap the java APIs to some other package. For the JVM
the implementation of the remapped APIs could be a bunch of package
objects with type aliases and val forwarders.

For other platforms there would be an implementation of the APIs
written in Scala. The implementation would be split between things
that can be implemented portably and those that require
implementations specific to the target platform. For classes where
only a portion of the methods are not portably implementable there
would be a shared trait for the portable part extended by the
platform specific implementation classes.

Theoretically adding

import _root_.{scala.platform => java}

to the top of a source file would be sufficient to make it use these
classes instead of those in the JRE except for the implicit import of
java.lang._

2. Design a idomatic Scala platform API. Split the API into parts that
can be implemented portably and those that can't just as above.

In this case even the JVM target would require additional code to
implement the platform API on top of JRE classes.

This route would require signification modifications to the compiler
and library to use these classes instead of those in the JRE.

With this outlook the "base" classes are exactly those which are
used by the compiler and some core subset of library yet to be
defined (though Paul has done some thinking about this already).

Miguel, I'm not sure that I understand what you mean by whitelisting and
book-keeping. By this do you mean generating a list of java classes
which are allowable in the compiler and library and a process for
enforcing the constraint?

It might also work to do the first to get things moving and then do the
second. Indeed my preference right now is to do the first and this is
what I understand is more or less what Simon is proposing to do.

Sébastien, Geoff, 
(Not sure why Geoff's reply doesn't show up in Google groups yet, but my comments refer to it). 

A straightforward way to find out java.lang.* dependencies is the compiler option -Yno-importswhose effect is "Compile without importing scala.*, java.lang.*, or Predef."

A few more details on that "-Yno-imports", it affects the following: 

  /** List of symbols to import from in a root context.  Typically that   *  is java.lang, scala, and scala.Predef, in that order.  Exceptions:   *    *  -- if -Yno-imports is given, nothing is imported   *  -- if the unit is java defined, only java.lang is imported   *  -- if -Yno-predef is given, if the unit has an import of Predef   *     among its leading imports, or if the unit is scala.ScalaObject   *     or scala.Predef, Predef is not imported.   */  protected def rootImports(unit: CompilationUnit, tree: Tree): List[Symbol] = {    import definitions._    assert(isDefinitionsInitialized, "definitions uninitialized")
    if (settings.noimports.value) Nil    else if (unit.isJava) List(JavaLangPackage)    else if (settings.nopredef.value || treeInfo.noPredefImportForUnit(unit.body)) List(JavaLangPackage, ScalaPackage)    else List(JavaLangPackage, ScalaPackage, PredefModule)  }


> 2. Design a idomatic Scala platform API. Split the API > into parts that can be implemented portably and those > that can't just as above.> > In this case even the JVM target would require additional code to> implement the platform API on top of JRE classes.

I see. It's doable, it's for the common good, and so on. 

> I'm not sure that I understand what you mean by > whitelisting and book-keeping. By this do you mean generating > a list of java classes which are allowable in the compiler > and library and a process for enforcing the constraint?
What I had in mind is some automatic means to check whether platform dependencies had leaked into the codebase. 
A tool to help with this need not be fancy, -Yno-imports helps somewhat but a dedicated compiler plugin can do a better job. For example, to find out whether a callsite invokes JDK stuff one can test:   msym.owner.ownerChain contains JavaPackageClasswhere    for(msym <- a.tpe.deferredMembers  . . . Other examples in   http://lampsvn.epfl.ch/trac/scala/browser/scala-experimental/trunk/jdk2ikvm/src/scala/tools/jdk2ikvm/patchcmds/Generating.scala 

That plugin would report sometimes good news, e.g. that java.util.Properties is used but in fact it's already pretty much cordoned to scala.sys.SystemProperties (thanks, Paul!) 
/** A bidirectional map wrapping the java System properties. *  Changes to System properties will be immediately visible in the map, *  and modifications made to the map will be immediately applied to the *  System properties.  If a security manager is in place which prevents *  the properties from being read or written, the AccessControlException *  will be caught and discarded. * *  @author Paul Phillips *  @version 2.9 *  @since   2.9 */class SystemProperties extends mutable.Map[String, String] {


Miguel http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

Hi everyone,

one of the problems I'm currently looking at is numbers in all variations.

It seems like we basically have to extend ScalaNumber because it makes
the class eligible for certain compiler hacks for comparing numbers with
each other. But ScalaNumber extends java.lang.Number, which has all this
legacy stuff like intValue/longValue/... which duplicates the Scala methods.

My plan is to ignore that for a moment and come up with cruft-free
implementations, which might require some additional tuning to make them
run on the various platforms, but come without the whole legacy baggage.

So I would have a look at things like the value types and BigInt,
BigDecimal first. Implementing and getting these things right is hard,
so I have no idea if I'm able to do it and how long it takes, but at
least the have not too much connection to the underlying platform like
the things in util.cocurrent, reflect, io, etc.

But generally, if Geoff or Sébastien need some class, this will have
priority.

Is there a way to count the usage of various Java libraries in the Scala
library? I had a short look with IntelliJ, but no substantial numbers.

Thanks and bye,

Simon

I would like to stress the importance of not slowing down JVM code by adding levels of indirection.
- Tiark

On Jul 1, 2011, at 12:55 PM, Simon Ochsenreither wrote:

> Hi everyone,
>
> one of the problems I'm currently looking at is numbers in all variations.
>
> It seems like we basically have to extend ScalaNumber because it makes the class eligible for certain compiler hacks for comparing numbers with each other. But ScalaNumber extends java.lang.Number, which has all this legacy stuff like intValue/longValue/... which duplicates the Scala methods.
>
> My plan is to ignore that for a moment and come up with cruft-free implementations, which might require some additional tuning to make them run on the various platforms, but come without the whole legacy baggage.
>
> So I would have a look at things like the value types and BigInt, BigDecimal first. Implementing and getting these things right is hard, so I have no idea if I'm able to do it and how long it takes, but at least the have not too much connection to the underlying platform like the things in util.cocurrent, reflect, io, etc.
>
> But generally, if Geoff or Sébastien need some class, this will have priority.
>
> Is there a way to count the usage of various Java libraries in the Scala library? I had a short look with IntelliJ, but no substantial numbers.
>
> Thanks and bye,
>
>
> Simon

Hi Tiark,
> I would like to stress the importance of not slowing down JVM code by adding levels of indirection.
> - Tiark
afaiu no one has plans in that direction. I'm pretty sure Scala on the
JVM will always depend on Java classes.

The discussion is mostly about what to do when Scala runs on a different
platform.
A first step could be that the individual ports replace the the
references to java.lang with the native Scala ones after the compiler ran.

Thanks and bye,

Simon