scala in scala?

Do you mean that Scala should have its own VM? (← not really practical)Or that it should compile to native code? (← this would open up the Scala universe to a lot of people)Or that it should implement its own version of a JVM with Scala-specific extensions? (← better to just write a VM from scratch in that case) Should it target a different VM, such as Parrot VM, or a fully Scala-specialized front-end for VMKit? (← this would be incredible as well)
The nsc (native Scala compiler) is already written in Scala, so "Scala-in-Scala" is already a reality in that sense.

On Tue, Aug 24, 2010 at 10:35 PM, Raoul Duke <raould@gmail.com> wrote:

hi,

any thoughts on getting scala more independent of the jvm via
scala-in-scala-izing things? (cf. clojure moving that way it appears.)

sincerely.

On Tue, Aug 24, 2010 at 2:22 PM, David Flemström
wrote:
> Do you mean that...

my main thought was portability away from the jvm, either to other
vms, or to e.g. C.

On Tue, Aug 24, 2010 at 11:24 PM, Raoul Duke <raould@gmail.com> wrote:

or to e.g. C.

 That's what I meant by "native code". Yes, that would be interesting, especially on top of LLVM, or via VMKit AOT compilation which would make it possible to still use many Java libraries.

On Tue, Aug 24, 2010 at 2:24 PM, Raoul Duke wrote:
> my main thought was portability away from the jvm, either to other
> vms, or to e.g. C.

Just speaking for myself, I hope this never happens. I don't believe
that functional programming will ever become mainstream enough to
support a healthy software ecosystem on its own. The Scala/Clojure
model of piggybacking on a bigger, slower, more mainstream language
and living in symbiosis with it is the way forward, IMO.

I like Scala as a language very much but the main reason I prefer it
to Haskell or OCaml is that I find it extremely practical and
productive. With Scala I can plug into an existing Java software base
while writing code in a much nicer language. All the energy that would
otherwise have to go into building a VM, writing countless boring but
necessary libraries etc, instead in the Scala world goes to building
*new* things like Lift and the Twitter Scala messaging stuff.

TL;DR - We're getting a tremendous free ride on the JVM. Why on earth
would we think of throwing that away?

On Tue, Aug 24, 2010 at 3:00 PM, Miles Egan wrote:
> TL;DR - We're getting a tremendous free ride on the JVM. Why on earth
> would we think of throwing that away?

there are more possible use-cases on this earth than are dreamt of in
your particular philosophy. :-)

On Tue, Aug 24, 2010 at 10:35 PM, Raoul Duke wrote:
> hi,
>
> any thoughts on getting scala more independent of the jvm via
> scala-in-scala-izing things? (cf. clojure moving that way it appears.)
>
AFAIK clojure in clojure is an attempt to write the clojure compiler
and it s core libraries in clojure. Scala has that since version 2.0,
which came out in 2005.

There's a scala for .NET effort under way. The main problem is
re-using the core libraries. IKVM helps a lot in the .NET case.

Cheers

On Tue, Aug 24, 2010 at 3:03 PM, Raoul Duke wrote:
> there are more possible use-cases on this earth than are dreamt of in
> your particular philosophy. :-)

No doubt. Scala has a nice, simple story right now though: keep all
your Java infrastructure but migrate to a much nicer language.
Complicating that with various, incompatible backends and the
inevitable balkanization of communities and libraries that entails
seems like a serious misstep to me.

Just my $0.01 though. Obviously these decisions are in the hands of
people much sharper than I am.

On Tuesday August 24 2010, Miles Egan wrote:
> ... I don't believe that functional programming will ever become
> mainstream enough to support a healthy software ecosystem on its own.
> ...

Programming: It's either functional or dysfunctional.

RRS

On Tue, Aug 24, 2010 at 3:56 PM, Randall R Schulz wrote:
> On Tuesday August 24 2010, Miles Egan wrote:
>> ... I don't believe that functional programming will ever become
>> mainstream enough to support a healthy software ecosystem on its own.
>> ...
>
> Programming: It's either functional or dysfunctional.

Indeed. Of course, I would love to be wrong about this.

> From: Miles Egan [mailto:milesegan@gmail.com]
[...]
> Just speaking for myself, I hope this never happens. I don't believe
> that functional programming will ever become mainstream enough to
> support a healthy software ecosystem on its own.

1.) "ever" is a very, very long time frame ;-)
2.) Scala is not FP but OFP. It allows for imperative style, and for
mixing styles, which is an interesting feature beyond VMs.
3.) As a humble developer I learned very much from Scala and FP,
which changed my development approach unrevertable and helped
me a lot with recent problems I had to solve.
I expect this experience from most other programmers in the
future.

> The Scala/Clojure model of piggybacking on a bigger, slower, more
> mainstream language and living in symbiosis with it is the way forward, IMO.

Yes. Absolutely. But this is no argument against sideways.
Like Python and Ruby found its way from native to (J)VM-based,
there could be a way back.

> I like Scala as a language very much but the main reason I prefer it
> to Haskell or OCaml is that I find it extremely practical and
> productive.

> With Scala I can plug into an existing Java software base
> while writing code in a much nicer language. All the energy that would
> otherwise have to go into building a VM, writing countless boring but
> necessary libraries etc, instead in the Scala world goes to building
> *new* things like Lift and the Twitter Scala messaging stuff.

>
> TL;DR - We're getting a tremendous free ride on the JVM. Why on earth
> would we think of throwing that away?

One answer: I recently thought about a language to compile to a simple
single binary (.out / .exe) which could be deployed without the receiver
having to install a JRE or an interpreter first.
Which would simply run without warmup times.
Which would have a much smaller memory footprint.
Which would run on some webspace even if the provider would never install
a JRE (or other language environment) just for me. No Perl, no PHP.

The best here would be C. But ... well, C? :-/
It has functions, but also pointer arithmetic, alloc a.s.o.
The language should better be a bit more functional, but also simple
enough.
(Add some more thoughts in this direction here. You get the picture).

I ended up playing with Google Go now.
A really fascinating experience (e.g. structural typing as philosophy,
CSP based concurrency, fast compile and fast startup)

But: There are enough spots where I thought I would rather have a
native compilable Scala derivate at hand, to get the same runtime result
but the better language.
All the heavy JVM/Java-lib based stuff for my Job, plus the light
coding experience at home. That *would* be nice.

Just my 2ct
Det

Detering Dirk wrote:
> 3.) As a humble developer I learned very much from Scala and FP,
> which changed my development approach unrevertable and helped
> me a lot with recent problems I had to solve.
> I expect this experience from most other programmers in the
> future.
I would rate this as an extremely optimistic expectation.

> One answer: I recently thought about a language to compile to a simple
> single binary (.out / .exe) which could be deployed without the receiver
> having to install a JRE or an interpreter first.
> Which would simply run without warmup times.
> Which would have a much smaller memory footprint.
> Which would run on some webspace even if the provider would never install
> a JRE (or other language environment) just for me. No Perl, no PHP.
>
> The best here would be C. But ... well, C? :-/

This is not obvious to me. If you want to compile natively, why not
Haskell?

Regards
Andreas

Why piggyback on another language in the first place?  LLVM and GCC are capable (esp. LLVM) compiler back-ends for which new language front-ends can be written easily.

On Wed, Aug 25, 2010 at 5:15 PM, Andreas Flierl <andreas@flierl.eu> wrote:

Detering Dirk <Dirk.Detering@bitmarck.de> wrote:
> 3.) As a humble developer I learned very much from Scala and FP,
>     which changed my development approach unrevertable and helped
>     me a lot with recent problems I had to solve.
>     I expect this experience from most other programmers in the
>     future.
I would rate this as an extremely optimistic expectation.

> One answer:  I recently thought about a language to compile to a simple
> single binary (.out / .exe) which could be deployed without the receiver
> having to install a JRE or an interpreter first.
> Which would simply run without warmup times.
> Which would have a much smaller memory footprint.
> Which would run on some webspace even if the provider would never install
> a JRE (or other language environment) just for me.  No Perl, no PHP.
>
> The best here would be C. But ... well, C?  :-/

This is not obvious to me. If you want to compile natively, why not
Haskell?

Regards
Andreas

On Wed, Aug 25, 2010 at 8:03 AM, Detering Dirk
wrote:
> Like Python and Ruby found its way from native to (J)VM-based,
> there could be a way back.

There's a lot of pain wrt JRuby too. No C libraries work, for obvious
reasons, so that creates a schism. I also think that perhaps the
biggest lesson to be learned from the success of JRuby is that there
are many advantages to being on the JVM.

> But: There are enough spots where I thought I would rather have a
> native compilable Scala derivate at hand, to get the same runtime result
> but the better language.
> All the heavy JVM/Java-lib based stuff for my Job, plus the light
> coding experience at home. That *would* be nice.

Engineering is all about tradeoffs. Scala in its current incarnation
has chosen the tradeoffs most likely to get an FP lang off the ground
in a significant way. A bifurcated native/JVM approach causes far more
problems than it solves, IMO.

> This is not obvious to me. If you want to compile natively, why not
> Haskell?

Indeed it isn't, as I didn't metion a past experience with C,
nor that my imagination was stuck to familiar syntax to keep
the adaption smooth.

That aside, I think your suggestion is more than obvious
and perhaps I will give it a try when I have more time/energy
to do the jump.

But this does not change the picture reg. native Scala.

KR
Det

How would you propose to deal with Java libraries used from Scala in this scenario?
More importantly, what optimisations would you apply?  The JVM starts by interpreting code, then compiles it to native based on runtime statistics.  Not only will the optimisations vary between platforms, but the JVM will also intelligently NOT apply them.  Such a counter-optimisation would be inlining where the resultant code crosses a page boundary and so cannot be held entirely in cache.
Ahead-of-time compilers for Java bytecode have been and gone, many were expensive, but the main reason they failed wasn't cost, it was the simple fact that they don't really improve on JIT compilation (except for startup time and memory footprint in specialist scenarios)

On 25 August 2010 16:44, Miles Egan <milesegan@gmail.com> wrote:

On Wed, Aug 25, 2010 at 8:03 AM, Detering Dirk
<Dirk.Detering@bitmarck.de> wrote:
> Like Python and Ruby found its way from native to (J)VM-based,
> there could be a way back.

There's a lot of pain wrt JRuby too. No C libraries work, for obvious
reasons, so that creates a schism. I also think that perhaps the
biggest lesson to be learned from the success of JRuby is that there
are many advantages to being on the JVM.

> But: There are enough spots where I thought I would rather have a
> native compilable Scala derivate at hand, to get the same runtime result
> but the better language.
> All the heavy JVM/Java-lib based stuff for my Job, plus the light
> coding experience at home. That *would* be nice.

Engineering is all about tradeoffs. Scala in its current incarnation
has chosen the tradeoffs most likely to get an FP lang off the ground
in a significant way. A bifurcated native/JVM approach causes far more
problems than it solves, IMO.

--
miles

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

A job search on NYC Craigslist for "scala" returns 16 results. A
search for "haskell" returns 2. Certainly the Scala team deserve a lot
of credit for designing such a nice language, but this is also surely
a result of Scala's deep compatibility with the most popular software
development platform in the world. For those of us that hope one day
to make FP more than a hobby this matters.

On Wed, Aug 25, 2010 at 8:18 AM, David Flemström
wrote:
> Why piggyback on another language in the first place?  LLVM and GCC are
> capable (esp. LLVM) compiler back-ends for which new language front-ends can
> be written easily.

A bit off-topic, but what about D ? :-)

On Wed, Aug 25, 2010 at 5:03 PM, Detering Dirk <Dirk.Detering@bitmarck.de> wrote:

> From: Miles Egan [mailto:milesegan@gmail.com]
[...]
One answer:  I recently thought about a language to compile to a simple
single binary (.out / .exe) which could be deployed without the receiver
having to install a JRE or an interpreter first.
Which would simply run without warmup times.
Which would have a much smaller memory footprint.
Which would run on some webspace even if the provider would never install
a JRE (or other language environment) just for me.  No Perl, no PHP.

The best here would be C. But ... well, C?  :-/

On Wed, Aug 25, 2010 at 5:53 PM, Kevin Wright <kev.lee.wright@gmail.com> wrote:

How would you propose to deal with Java libraries used from Scala in this scenario?

As I said: one could write a front-end for VMKit specialized for Scala. VMKit is a VM that runs both CLI and JVM code using the same LLVM-based back-end, and JVM code (shipped in ordinary JARs etc) can for example invoke CLI code (in normal assemblies) seamlessly without marshaling.
Adding a third front-end, that is Scala-specific and uses a specialized instruction set would probably be worthwhile. It could feature full runtime type information, maybe built-in support for closures (already present in LLVM IR afaik) and mixins (Present in a sense via C++ multiple inheritance in the C++ front-end to LLVM), cheap threads, a more extensive AnyVal tree that could include richer (LLVM supports integer types of any bit-length and signedness, of course performing tons of JIT optimizations when possible), more type-safe (no auto-casts or boxing/implicit mess) and strongly typed value types, etc - essentially eliminating all of the issues the JVM introduces.
One might argue that introducing target-specific core library features will be fatal to language portability, but that's kind of irrelevant at this point anyways, because the CLI scalac target already uses a different core library compared to the JVM one, so the language is already unportable in that sense. One would have to write a Scala core library completely independent from the target platform (Including the implementation of String, all of the AnyVal stuff, and the IO library for example, which currently pose huge problems for those of us trying out Scala.NET) in order for Scala to be completely portable.
VMKit is a bit slower (pdf link) compared to traditional JVMs, but having a Scala-specific byte-code (or alternatively complementary .scalakit (or whatever) files besides JVM .class files in a JAR, containing more efficient VMKit-specific "overrides") would probably make up for a lot of that difference, since it would offer the back-end more optimization opportunities.

This sounds like introducing a tremendous amount of incidental
complexity into the common case for the benefit of a hypothetical and
likely quite marginal corner case.

On Wed, Aug 25, 2010 at 11:34 AM, David Flemström
wrote:
> VMKit is a bit slower (pdf link) compared to traditional JVMs, but having a
> Scala-specific byte-code (or alternatively complementary .scalakit (or
> whatever) files besides JVM .class files in a JAR, containing more efficient
> VMKit-specific "overrides") would probably make up for a lot of that
> difference, since it would offer the back-end more optimization
> opportunities.

Definitely, yes. It was just a thought.

On Wed, Aug 25, 2010 at 9:43 PM, Miles Egan <milesegan@gmail.com> wrote:

This sounds like introducing a tremendous amount of incidental
complexity into the common case for the benefit of a hypothetical and
likely quite marginal corner case.

On Wed, Aug 25, 2010 at 11:34 AM, David Flemström
<david.flemstrom@gmail.com> wrote:
> VMKit is a bit slower (pdf link) compared to traditional JVMs, but having a
> Scala-specific byte-code (or alternatively complementary .scalakit (or
> whatever) files besides JVM .class files in a JAR, containing more efficient
> VMKit-specific "overrides") would probably make up for a lot of that
> difference, since it would offer the back-end more optimization
> opportunities.

--
miles