Re: scala IO and modules: Working groups?

If the new IO includes ARM, it might be desirable to split that up into it's own working group to work very closely (I'm assuming substantial overlap membership anyway) with the IO working group. Or at least the IO working group should consider non-IO use cases if/when designing an ARM library.

--j

On Tue, Sep 8, 2009 at 7:20 AM, martin odersky <martin.odersky@epfl.ch> wrote:

Hi all,

I think the recent initiatives on modules and on I/O are very good
things. (And there's another effort springing up around transactional
memory that has also great potential. ) I wondered whether it would be
a good thing to assemble working groups around these efforts. The
purpose of a working group is to come up with a SID (Scala improvement
document), which should include a working implementation. The easiest
organizional scheme is probably to attach a working group to a SID
effort, so that there's at least a number that tells them apart. I
think also that the EPFL team should make an effort to have a
representative on both of these working groups.

If you wish to form a working group, please send mail with the title
of the proposal and the persons currently involved to Tony. We might
try to find other people inside or outside EPFL to join in the
discussion and will follow the group's work.

Cheers

I wouldn't mind helping/shepharding/leading the effort on the ARM stuff, as we're using that in production for non-I/O code right now.

- Josh

On Tue, Sep 8, 2009 at 12:10 PM, Jorge Ortiz <jorge.ortiz@gmail.com> wrote:

If the new IO includes ARM, it might be desirable to split that up into it's own working group to work very closely (I'm assuming substantial overlap membership anyway) with the IO working group. Or at least the IO working group should consider non-IO use cases if/when designing an ARM library.

--j

On Tue, Sep 8, 2009 at 7:20 AM, martin odersky <martin.odersky@epfl.ch> wrote:

Hi all,

I think the recent initiatives on modules and on I/O are very good
things. (And there's another effort springing up around transactional
memory that has also great potential. ) I wondered whether it would be
a good thing to assemble working groups around these efforts. The
purpose of a working group is to come up with a SID (Scala improvement
document), which should include a working implementation. The easiest
organizional scheme is probably to attach a working group to a SID
effort, so that there's at least a number that tells them apart. I
think also that the EPFL team should make an effort to have a
representative on both of these working groups.

If you wish to form a working group, please send mail with the title
of the proposal and the persons currently involved to Tony. We might
try to find other people inside or outside EPFL to join in the
discussion and will follow the group's work.

Cheers

On 8 Sep 2009, at 15:20, martin odersky wrote:

> I wondered whether it would be a good thing to assemble working
> groups around these efforts.

What do you mean by working groups, over and above the discussions
that have been taking place on this list already? One of the reasons
for having a discussion on a mailing list in the first place was to
(a) encourage discussion and (b) have those discussions searchable and
persisted in order to demonstrate openness. So if 'working group'
means more than this, can you elaborate on the specifics?

> The purpose of a working group is to come up with a SID (Scala
> improvement
> document), which should include a working implementation. The easiest
> organizional scheme is probably to attach a working group to a SID
> effort, so that there's at least a number that tells them apart.

The problem with the currently named SID (and its counterpart, SIP) is
that it assumes a oneness of everything. A document isn't the be-all
and end-all of a process; in fact, it would be a poor effort if a
working group only generated one revision of a document. Consider the
counterparts in other languages; Java Specification *Request*, Python
Enhancement *Proposal*. I think 'document' is entirely the wrong thing
to focus on for these additions. (I also disagree in the fact that
these must be submitted as a document in PDF when more open document
formats - like HTML - are just as capable for the most part; but
that's a separate concern.)

Furthermore, this is stretching it to breaking point. SIDs are
described as a document in which change can happen; but this therefore
assumes that the document comes first. So either you have off-the-
record debates, proposals, counter proposals and then finally write it
up and then submit it, or you have to start a SID without the D. Note
that pretty much all other enhancement avenues start off with a brief
proposal (and initial membership of that working group or sponsors)
which is then fleshed out, often with many drafts, before becoming an
accepted standard. In most cases, the output is a lot more than a
document (APIs, sample code, default implementation etc.). Again, the
D part of SID is pretty inaccurate.

> I think also that the EPFL team should make an effort to have a
> representative on both of these working groups.

Sounds like a reasonable plan.

> If you wish to form a working group, please send mail with the title
> of the proposal and the persons currently involved to Tony. We might
> try to find other people inside or outside EPFL to join in the
> discussion and will follow the group's work.

Who's Tony, and why should I know his/her e-mail address? And what is
different from what's been discussed so far on the list?

Alex

PS In respect of modules, there are a couple of unanswered questions
that only the EPFL team can answer, such as how wedded you are to the
existing Ant build and what constraints on other build tools are
present. For example, what about migrating the build process to a
Maven-based build? Is that already being considered elsewhere?

On Wed, Sep 9, 2009 at 10:03 AM, Alex Blewitt wrote:
> On 8 Sep 2009, at 15:20, martin odersky wrote:
>> The purpose of a working group is to come up with a SID (Scala improvement
>> document), which should include a working implementation. The easiest
>> organizional scheme is probably to attach a working group to a SID
>> effort, so that there's at least a number that tells them apart.
>
> The problem with the currently named SID (and its counterpart, SIP) is that
> it assumes a oneness of everything. A document isn't the be-all and end-all
> of a process; in fact, it would be a poor effort if a working group only
> generated one revision of a document.

I don't read Martin's proposal that way. A SID and an implementation
are intended as the output of the activity of a working group. In
getting there there would be lot's of (public, archived) discussion,
and presumably several iterations of proposals, counter-proposals,
drafts and prototypes.

Cheers,

Miles

Alex Blewitt wrote:
> On 8 Sep 2009, at 15:20, martin odersky wrote:
>
>> If you wish to form a working group, please send mail with the title
>> of the proposal and the persons currently involved to Tony. We might
>> try to find other people inside or outside EPFL to join in the
>> discussion and will follow the group's work.
>
> Who's Tony, and why should I know his/her e-mail address? And what is
> different from what's been discussed so far on the list?
>
> Alex
>

That would be me: Antonio Cunei, more frequently referred to as Toni. I
work at EPFL in Martin's group, taking care of many practical aspects
related to Scala development and documentation, including preparing
releases, maintaining the website, dealing with SVN, and handling mailing
lists and general interaction with the userbase. Nice to meet you! :)

Using this mailing list for discussion is perfectly all right; we thought
that it would be appropriate in any case for someone from EPFL to track the
discussion and provide, if needed, support about required technical
information and infrastructure. Therefore, if the discussion evolves to a
point where a separate mailing list or forum is needed, I will take care to
arrange the related practical details, for instance.

Out of the discussions/groups currently on scala-internals I will do my
best to follow the IO effort. I know little about the modularization
effort, but I can act as a reference and dispatch questions appropriately
within the core team until a more permanent arrangement is found; several
new people are joining us at EPFL at this time, and one of them could be
involved more closely in the modularization group.

In any event, for all questions and support related to mailing lists,
website, SVN branching and technical support, you can always contact me.

> PS In respect of modules, there are a couple of unanswered questions
> that only the EPFL team can answer, such as how wedded you are to the
> existing Ant build and what constraints on other build tools are
> present. For example, what about migrating the build process to a
> Maven-based build? Is that already being considered elsewhere?
>

The short answer is that we detest Ant. It is a tool that more or less
works for us and is independent on the specific platform, but we had
countless troubles with memory leaks and various Ant oddities, and tried
several times to replace it.

Concerning Maven, we find the possibility appealing but we don't have
specific in-house experience with it. The use of Maven is also related to
sbaz, our current package distribution infrastructure (no longer actively
maintained). You can find a general description of sbaz here:
http://www.scala-lang.org/node/93

For a while we considered replacing sbaz with Maven, and we asked our user
base whether anyone would be interested in that. The result was the
creation of a very active and productive group, which resulted in the
availability of Scala via Maven from the scala-tools.org website. That
website is not directly managed by us, it is a user-supported resource;
however, our build scripts were patched so that each new Scala release is
uploaded there as well.

You can find all the discussion related to Scala and Maven in the section
"The Maven Project" of http://www.scala-lang.org/node/292 and
on http://code.google.com/p/esmi/wiki/ScalaMavenSupport
plus on their Google Group page: http://groups.google.com/group/maven-and-scala

You should probably contact Joshua Suereth if you are interested in
coordinating with/reviving that project. For the time being, in any case,
we are still stuck with sbaz and the Ant script for the main distribution.

We also discussed on occasion to use sbt rather than Ant: sbt is
specifically a Scala tool, and might be a good solution for us, but so far
we have not looked into that in sufficient detail. You can find further
information on sbt on this page: http://code.google.com/p/simple-build-tool

I hope this information is of help.
Toni

Hey all,

On Wed, 2009-09-09 at 11:55 +0200, Antonio Cunei wrote:
> We also discussed on occasion to use sbt rather than Ant: sbt is
> specifically a Scala tool, and might be a good solution for us, but so far
> we have not looked into that in sufficient detail. You can find further
> information on sbt on this page: http://code.google.com/p/simple-build-tool

Based on discussions between Paul Phillips and Mark Harrah on IRC, I am
aware that some work has been done in this area.

I believe it would be useful to know what is the current status. Mark,
can you fill us in? Thanks.

Best,
Ismael

On Wed, Sep 9, 2009 at 10:55 AM, Antonio Cunei wrote:
> Alex Blewitt wrote:
>>
>> On 8 Sep 2009, at 15:20, martin odersky wrote:
>>
>>> If you wish to form a working group, please send mail with the title
>>> of the proposal and the persons currently involved to Tony. We might
>>> try to find other people inside or outside EPFL to join in the
>>> discussion and will follow the group's work.
>>
>> Who's Tony, and why should I know his/her e-mail address? And what is
>> different from what's been discussed so far on the list?
>>
>> Alex
>>
>
> That would be me: Antonio Cunei, more frequently referred to as Toni. I work
> at EPFL in Martin's group, taking care of many practical aspects related to
> Scala development and documentation, including preparing releases,
> maintaining the website, dealing with SVN, and handling mailing lists and
> general interaction with the userbase. Nice to meet you! :)
>
> Using this mailing list for discussion is perfectly all right; we thought
> that it would be appropriate in any case for someone from EPFL to track the
> discussion and provide, if needed, support about required technical
> information and infrastructure. Therefore, if the discussion evolves to a
> point where a separate mailing list or forum is needed, I will take care to
> arrange the related practical details, for instance.
>
> Out of the discussions/groups currently on scala-internals I will do my best
> to follow the IO effort. I know little about the modularization effort, but
> I can act as a reference and dispatch questions appropriately within the
> core team until a more permanent arrangement is found; several new people
> are joining us at EPFL at this time, and one of them could be involved more
> closely in the modularization group.
>
> In any event, for all questions and support related to mailing lists,
> website, SVN branching and technical support, you can always contact me.
>
>> PS In respect of modules, there are a couple of unanswered questions that
>> only the EPFL team can answer, such as how wedded you are to the existing
>> Ant build and what constraints on other build tools are present. For
>> example, what about migrating the build process to a Maven-based build? Is
>> that already being considered elsewhere?
>>
>
> The short answer is that we detest Ant. It is a tool that more or less works
> for us and is independent on the specific platform, but we had countless
> troubles with memory leaks and various Ant oddities, and tried several times
> to replace it.
>
> Concerning Maven, we find the possibility appealing but we don't have
> specific in-house experience with it. The use of Maven is also related to
> sbaz, our current package distribution infrastructure (no longer actively
> maintained). You can find a general description of sbaz here:
> http://www.scala-lang.org/node/93
>
> For a while we considered replacing sbaz with Maven, and we asked our user
> base whether anyone would be interested in that. The result was the creation
> of a very active and productive group, which resulted in the availability of
> Scala via Maven from the scala-tools.org website. That website is not
> directly managed by us, it is a user-supported resource; however, our build
> scripts were patched so that each new Scala release is uploaded there as
> well.
>
> You can find all the discussion related to Scala and Maven in the section
> "The Maven Project" of http://www.scala-lang.org/node/292 and
> on http://code.google.com/p/esmi/wiki/ScalaMavenSupport
> plus on their Google Group page:
> http://groups.google.com/group/maven-and-scala
>
> You should probably contact Joshua Suereth if you are interested in
> coordinating with/reviving that project. For the time being, in any case, we
> are still stuck with sbaz and the Ant script for the main distribution.

Have you considered ivy? This being a halfway house between maven and
ant? The problem in migrating from an ant build to a maven one is that
it is a big bang and takes a lot of investment. Also an ivy build can
produce artifacts that will work in a maven environment so would
possibly make the scala tools project easier to maintain?

2 cents etc.

Regards,

Dave

>
> We also discussed on occasion to use sbt rather than Ant: sbt is
> specifically a Scala tool, and might be a good solution for us, but so far
> we have not looked into that in sufficient detail. You can find further
> information on sbt on this page: http://code.google.com/p/simple-build-tool
>
> I hope this information is of help.
> Toni
>

David Savage wrote:
> On Wed, Sep 9, 2009 at 10:55 AM, Antonio Cunei wrote:
>> You should probably contact Joshua Suereth if you are interested in
>> coordinating with/reviving that project. For the time being, in any case, we
>> are still stuck with sbaz and the Ant script for the main distribution.
>
> Have you considered ivy? This being a halfway house between maven and
> ant? The problem in migrating from an ant build to a maven one is that
> it is a big bang and takes a lot of investment. Also an ivy build can
> produce artifacts that will work in a maven environment so would
> possibly make the scala tools project easier to maintain?
>

We know about Ivy, the suggestion came up various times on the scala-tools
mailing list. As in the case of Maven, however, the issue is that of
concretely reimplementing the build/distribution system and maintaining it
on the medium term.

Right now the Ant/sbaz solution works reasonably well, so in order to
replace our building and distribution infrastructure we would have to have
something that is proven to be stable, useful, and not significantly more
complicated to maintain than what we do have now.

Developing a solid Maven/Ivy/sbt building and/or distribution mechanism,
and further modularizing the current build process, is something that
requires both time and specific skills with those tools, something that we
do not have in-house. That is why we are extremely interested in help from
the community in order to develop alternatives that may better serve the
Scala user base.

As I mentioned, my own specific knowledge on the subject is rather limited
however, so I'll let more knowledgeable people elaborate on the path to
follow in order to achieve that goal. But please let me know if I can
assist you in this effort.

On a related subject, concerning Maven/OSGi etc, Martin also observed that
project Jigsaw also entails modularization. Any attempt to modularize Scala
should also keep an eye on their progress and the conventions they use.

Toni

Using this mailing list for discussion is perfectly all right; we thought that it would be appropriate in any case for someone from EPFL to track the discussion and provide, if needed, support about required technical information and infrastructure. Therefore, if the discussion evolves to a point where a separate mailing list or forum is needed, I will take care to arrange the related practical details, for instance.

Yes, "creating a working group" means defining who wants to work on that project,
define how these people communicate and store information (e.g. google groups + wiki),
who is the main contact from inside EPFL.
Of course, the general mailing list is the place to start the discussion, find people, get
opinions if general questions come up during the work etc.

 

PS In respect of modules, there are a couple of unanswered questions that only the EPFL team can answer, such as how wedded you are to the existing Ant build and what constraints on other build tools are present. For example, what about migrating the build process to a Maven-based build? Is that already being considered elsewhere?

The short answer is that we detest Ant. It is a tool that more or less works for us and is independent on the specific platform, but we had countless troubles with memory leaks and various Ant oddities, and tried several times to replace it.

Concerning Maven, we find the possibility appealing but we don't have specific in-house experience with it. The use of Maven is also related to sbaz, our current package distribution infrastructure (no longer actively maintained).

This is not entirely true, James Matlik has plans to extend sbaz and allow it to
install maven packages into a standard scalal distribution.

 

You can find a general description of sbaz here:
http://www.scala-lang.org/node/93

For a while we considered replacing sbaz with Maven, and we asked our user base whether anyone would be interested in that. The result was the creation of a very active and productive group, which resulted in the availability of Scala via Maven from the scala-tools.org website. That website is not directly managed by us, it is a user-supported resource; however, our build scripts were patched so that each new Scala release is uploaded there as well.

The conclusion of the investigation was that maven cannot possibly provide the
functionality we need from sbaz. We want to have an independent (only JVM) distribution
that people can download, unpack and run "scala" and "scalac". We need a
tool which can install additional libraries, documentation or scripts into this
downloaded distribution, something like ruby gem or haskell cabal / hackage.

What sbaz is missing:
 - versioned dependencies
 - infrastructure
   - browse packages online like http://hackage.haskell.org/packages/hackage.html
   - good tools and documentation for creating packages

An tight integration of sbaz and maven packages (ideally in both directions) is
probably important so that none of the two repositories suffers rot.

Lukas

On Wed, Sep 9, 2009 at 11:03 AM, Alex Blewitt wrote:
> On 8 Sep 2009, at 15:20, martin odersky wrote:
>
>> I wondered whether it would be a good thing to assemble working groups
>> around these efforts.
>
> What do you mean by working groups, over and above the discussions that have
> been taking place on this list already? One of the reasons for having a
> discussion on a mailing list in the first place was to (a) encourage
> discussion and (b) have those discussions searchable and persisted in order
> to demonstrate openness. So if 'working group' means more than this, can you
> elaborate on the specifics?

Discussions are fine, but in the end somebody has to do it: Choose the
technology, develop the build scripts, develop the packagings, try
them out, write the documentation, etc. This could be a group of
people, or it could be a well-motivated individual. I was just
thinking that because of the size of the task a group is better suited
to do it.

> PS In respect of modules, there are a couple of unanswered questions that
> only the EPFL team can answer, such as how wedded you are to the existing
> Ant build and what constraints on other build tools are present. For
> example, what about migrating the build process to a Maven-based build? Is
> that already being considered elsewhere?
>
We have a lot of things on our hands. This means we do not have the
resources to change infrastructure and build scripts ourselves. In a
resource constrained environment like ours we have to keep to `if it
ain't broken don't fix it". So if a new build method is needed, then
part of the task of this working group would be to develop a new build
script, which might well be based on different technology if necessary
(e.g. SBT vs Ant).

Cheers

On Wed, Sep 9, 2009 at 2:01 PM, Antonio Cunei wrote:
> David Savage wrote:
>>
>> On Wed, Sep 9, 2009 at 10:55 AM, Antonio Cunei
>> wrote:
>>>
>>> You should probably contact Joshua Suereth if you are interested in
>>> coordinating with/reviving that project. For the time being, in any case,
>>> we
>>> are still stuck with sbaz and the Ant script for the main distribution.
>>
>> Have you considered ivy? This being a halfway house between maven and
>> ant? The problem in migrating from an ant build to a maven one is that
>> it is a big bang and takes a lot of investment. Also an ivy build can
>> produce artifacts that will work in a maven environment so would
>> possibly make the scala tools project easier to maintain?
>>
>
> We know about Ivy, the suggestion came up various times on the scala-tools
> mailing list. As in the case of Maven, however, the issue is that of
> concretely reimplementing the build/distribution system and maintaining it
> on the medium term.

Makes sense, just wanted to raise it on this thread to make sure it
wasn't forgotten as there's often a tendency to see ant or maven as
only options. Just like runtime languages I like the idea that there
is a spectrum of build languages each with it's own particular
benefits. Of course you also run into the maintenance issues with a
build system just like a runtime so it's a good idea to go with one
that's got a good level of support behind it.

>
> Right now the Ant/sbaz solution works reasonably well, so in order to
> replace our building and distribution infrastructure we would have to have
> something that is proven to be stable, useful, and not significantly more
> complicated to maintain than what we do have now.
>
> Developing a solid Maven/Ivy/sbt building and/or distribution mechanism, and
> further modularizing the current build process, is something that requires
> both time and specific skills with those tools, something that we do not
> have in-house. That is why we are extremely interested in help from the
> community in order to develop alternatives that may better serve the Scala
> user base.
>
> As I mentioned, my own specific knowledge on the subject is rather limited
> however, so I'll let more knowledgeable people elaborate on the path to
> follow in order to achieve that goal. But please let me know if I can assist
> you in this effort.

It's definitely an area that interests me, but I'm only able to devote
a small amount of time at the moment as there are lots of other
threads going on at the moment...if you have any pointers on the build
system as it currently stands I'll make a note to have a look
through...

>
> On a related subject, concerning Maven/OSGi etc, Martin also observed that
> project Jigsaw also entails modularization. Any attempt to modularize Scala
> should also keep an eye on their progress and the conventions they use.

Makes sense, though from my understanding the jigsaw work seems to
moving at glacial speed but at the same time seems to change it's mind
pretty frequently on what it is - brownian software architectures? At
the risk of soapboxing...for myself I like OSGi 'cause I can use it
now, Jigsaw may bring other benefits but I don't really want to have
to wait for java 7. In case you missed it there was also this proposal
[1] to bridge the gap between jigsaw and osgi - not sure if it's got
anywhere though?

Regards,

Dave

[1] http://altair.cs.oswego.edu/pipermail/jsr294-modularity-eg/2009-August/0...

>
> Toni
>

On 9 Sep 2009, at 15:11, David Savage
wrote:

> On Wed, Sep 9, 2009 at 2:01 PM, Antonio Cunei
> wrote:
>> On a related subject, concerning Maven/OSGi etc, Martin also
>> observed that
>> project Jigsaw also entails modularization. Any attempt to
>> modularize Scala
>> should also keep an eye on their progress and the conventions they
>> use.
>
> Makes sense, though from my understanding the jigsaw work seems to
> moving at glacial speed

The requirements I posted before suggested that we should try and
treat as abstract modules rather than necessarily be tied to one
system. It should be noted though that in order to continue IDE
development, OSGi support is needed.

As and when Jigsaw or equivalent is available, we could consider using
that as well - but it's a work in progress at the moment.

Alec

On Wednesday 09 September 2009 06:03, Ismael Juma wrote:
> Hey all,
>
> On Wed, 2009-09-09 at 11:55 +0200, Antonio Cunei wrote:
> > We also discussed on occasion to use sbt rather than Ant: sbt is
> > specifically a Scala tool, and might be a good solution for us, but so far
> > we have not looked into that in sufficient detail. You can find further
> > information on sbt on this page:
http://code.google.com/p/simple-build-tool
>
> Based on discussions between Paul Phillips and Mark Harrah on IRC, I am
> aware that some work has been done in this area.
>
> I believe it would be useful to know what is the current status. Mark,
> can you fill us in? Thanks.

First, I'll try to set up the context. Sorry if this repeats what people
already know.

One view of sbt is that you can use it to write your own tasks for your build.
You might consider this Ant-like, except that the build is written in Scala.
sbt provides useful APIs for doing this (path, I/O, process, ...) and a core
set of tasks. There is also Walter Chang's interface to Ant [1] that might
be helpful.

From a different view, you can work with sbt's conventions for how a project
can be built. This can reduce initial work setting up a build and, again,
provides a core set of useful functionality. You can use automatic
dependency management for your libraries (using Ivy) or manage them manually.
You might consider this a Maven-like view.

Either of these views could be used for the Scala build and I believe sbt is
capable of providing (at a minimum) the current functionality of the current
build from either perspective.

What I think are the targets of the status request are additional features
provided by some of the core tasks. In order to use certain features, such
as partial recompilation, built-in testing support, running Jetty, and
unifying logging for all of these, sbt has to be compiled against the version
of Scala being used to build a project. Because the Scala build needs to
build a compiler and then use it, this would require building sbt against
this new compiler to get the features mentioned.

You can, of course, fork the compiler or call it reflectively in its own
classloader, like any other build tool can. You just don't get partial
recompilation.

The status then, I believe, is that some potentially desirable features are
not accessible, but sbt can at least provide what the ant build provides.

I have a basic build file that can be used to compile Scala using the latest
nightly. I believe this is roughly equivalent to the 'quick' target, except
that the latest nightly is used as the compiler instead of locker. Perhaps
the gaps could be filled in by forking or using the existing ant tasks. I
have not looked into this.

As for a longer-term solution, I have redesigned the architecture of sbt as
part of an experimental project xsbt[2]. This decouples the version of Scala
used to build the project from the version of Scala used for the project
definition (and hence the version used by the running instance of sbt). What
is required is that the code for sbt's analysis phase be source compatible
with the version of Scala doing the compiling. This experiment worked, but
because it required doing things in quite a different way, it will be some
time to swap this into sbt ('swap' being a gross understatement).

Thanks for reading!
-Mark

[1] http://wiki.github.com/weihsiu/antelese
[2] http://github.com/harrah/xsbt/

On 9 Sep 2009, at 14:55, martin odersky wrote:

On Wed, Sep 9, 2009 at 11:03 AM, Alex Blewitt<alex.blewitt@gmail.com> wrote:

On 8 Sep 2009, at 15:20, martin odersky wrote:

I wondered whether it would be a good thing to assemble working groups

around these efforts.

What do you mean by working groups, over and above the discussions that have

been taking place on this list already?

Discussions are fine, but in the end somebody has to do it: Choose the
technology, develop the build scripts, develop the packagings, try
them out, write the documentation, etc. This could be a group of
people, or it could be a well-motivated individual. I was just
thinking that because of the size of the task a group is better suited
to do it.

Agreed, but we're probably not even at the beginning of this list yet. In any case, I'm sure that various approaches will need to be tried, if only to meet the requirements. There have been a few people chiming in on this thread (and for EPFL's presence, Miles Sabin is interested in making this happen).

PS In respect of modules, there are a couple of unanswered questions that

only the EPFL team can answer, such as how wedded you are to the existing

Ant build and what constraints on other build tools are present.

We have a lot of things on our hands. This means we do not have the
resources to change infrastructure and build scripts ourselves.

Right, I wasn't proposing a TODO list that would have to be executed by EPFL directly. It was rather a question of what constraints there are (e.g. we can't use tools with an X in the name, or that it must run on Y OS) that we otherwise are faced with. The ability to run on a Java 1.5/1.6 VM is pretty much a given (note that Java 1.5 is EOL in a month and a half's time, though having it also be able to run on 1.5 is unlikely to be an issue). And presumably, the entire kit must be open source or at least easily downloadable.

In a resource constrained environment like ours we have to keep to `if it
ain't broken don't fix it". So if a new build method is needed, then
part of the task of this working group would be to develop a new build
script, which might well be based on different technology if necessary
(e.g. SBT vs Ant).

I suspect at the very least we'll need to do a source re-org, not the least of which was to meet one of the requirements (keep one location per module/project). So stuff like scala/actors might need to be punted out to their own location.
And as noted elsewhere, the choice of build tool should not prevent running Scala outside of any particular environment, so regardless of how the modules are built, they should be runnable (a) on the command line with a vanilla JVM and (b) in an OSGi VM in order to be used by the IDE.
From a practical perspective, being able to host (a) wiki pages, and (b) requirements/status docs in a single location (as opposed to buried in mail threads) would be a useful tool to have.
Alex

On Wed, Sep 9, 2009 at 8:04 PM, Alex Blewitt wrote:
> On 8 Sep 2009, at 15:20, martin odersky wrote:
> > Discussions are fine, but in the end somebody has to do it: Choose the
> > technology, develop the build scripts, develop the packagings, try
> > them out, write the documentation, etc. This could be a group of
> > people, or it could be a well-motivated individual. I was just
> > thinking that because of the size of the task a group is better suited
> > to do it.
>
> Agreed, but we're probably not even at the beginning of this list yet. In
> any case, I'm sure that various approaches will need to be tried, if only to
> meet the requirements. There have been a few people chiming in on this
> thread (and for EPFL's presence, Miles Sabin is interested in making this
> happen).

With the proviso that I work with EPFL rather than being from EPFL ...

I'm willing to do the necessary work to reorganize the library source
trees in SVN, modify the various build scripts accordingly and ensure
that appropriate OSGi metadata is available.

This is a relatively modest undertaking, but even so needs the buy-in
of the people (primarily at EPFL) who work on the Scala library source
trees ... moving things around in SVN can be disruptive.

Wholesale migration to Maven or Sbt or whatever is a completely
different kettle of fish. It may very well be that a simple
reorganization would be a useful first step in that direction, but
other contributors would be needed to go the whole way.

Cheers,

Miles

Having spend a lot of time weeding through scala's build system, I feel inclined to add some input here.


1) Adding Ivy to scala's current build system does not seem like a win.   The dependency section of the scala build process is actually a smaller contributor to complexity/effort (unlike other projects I've been on).  I ended up using the maven-ant-task just to deploy artifacts into a maven repository.  Ignoring the hype over Ivy vs. Maven dependency management for a bit, the maven repository is the current "standard" in the sense that every tool can interface with it.  If we seriously want to consider using Ivy, then we should seriously consider using SBT where the API for declaring dependencies and deployment is sane.  Also there are a number of big wins to SBT.

2) No matter what build system you migrate Sabbus to, you're going to have to fight with the lifecycle of the build.  Maven could make this difficult as you would almost need a custom lifecycle for the scala bootstrap process (no mean task there).  I'm pretty sure we could bend maven far enough for this, but I'm not sure I'm buying the win in terms of effort.  I love maven, and use it all the time, but the scala build system is very far outside of its lifecycle.

3) SBT seems like a win in terms of effort vs. gain.  It has amazing performance, is written in Scala (and therefore much easier for Scala developers to be able to maintain than a Java-1.4-API'd tool), and has the flexibility to support Sabbus's lifecycle.  It also supports mutli-module projects, so I don't see any major reasons to use maven over SBT at this point.  SBT would need to add some form of BND integration, but BND is a very nice, easy API to talk to at this point.  I had debated trying to port scala's build environment into SBT, however time is at a premium and I've minimized my commitments to things that I need for work, or high-priority bugs in maven-related areas (as the "coordinator", I feel obligated).

I love the idea of modularizing Scala.  I think working on libraries individually will be great.  Please make sure you don't ignore the existing build process or make assumptions that are not true.

I took a shot at modularizing Scala a while back and ran into some issues (that could be solved with 2.8 features).  In all of our efforts, I'd love to make sure we don't over-engineer a solution.  The solution should be beneficial to users *and* transparent or beneficial to developers.

I'm williing to participate in discussion and offer what I know.  In a few months I could even help work with porting to a new build system.  However currently I just don't have the time.


- Josh

All~

If part of the difficulty is having a truly customized build lifecycle, I would highly recommend SCons.  It runs on all the major platforms (and probably a few minor ones) and handles custom builders very cleanly.

Matt

On Wed, Sep 9, 2009 at 8:58 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

Having spend a lot of time weeding through scala's build system, I feel inclined to add some input here.


1) Adding Ivy to scala's current build system does not seem like a win.   The dependency section of the scala build process is actually a smaller contributor to complexity/effort (unlike other projects I've been on).  I ended up using the maven-ant-task just to deploy artifacts into a maven repository.  Ignoring the hype over Ivy vs. Maven dependency management for a bit, the maven repository is the current "standard" in the sense that every tool can interface with it.  If we seriously want to consider using Ivy, then we should seriously consider using SBT where the API for declaring dependencies and deployment is sane.  Also there are a number of big wins to SBT.

2) No matter what build system you migrate Sabbus to, you're going to have to fight with the lifecycle of the build.  Maven could make this difficult as you would almost need a custom lifecycle for the scala bootstrap process (no mean task there).  I'm pretty sure we could bend maven far enough for this, but I'm not sure I'm buying the win in terms of effort.  I love maven, and use it all the time, but the scala build system is very far outside of its lifecycle.

3) SBT seems like a win in terms of effort vs. gain.  It has amazing performance, is written in Scala (and therefore much easier for Scala developers to be able to maintain than a Java-1.4-API'd tool), and has the flexibility to support Sabbus's lifecycle.  It also supports mutli-module projects, so I don't see any major reasons to use maven over SBT at this point.  SBT would need to add some form of BND integration, but BND is a very nice, easy API to talk to at this point.  I had debated trying to port scala's build environment into SBT, however time is at a premium and I've minimized my commitments to things that I need for work, or high-priority bugs in maven-related areas (as the "coordinator", I feel obligated).

I love the idea of modularizing Scala.  I think working on libraries individually will be great.  Please make sure you don't ignore the existing build process or make assumptions that are not true.

I took a shot at modularizing Scala a while back and ran into some issues (that could be solved with 2.8 features).  In all of our efforts, I'd love to make sure we don't over-engineer a solution.  The solution should be beneficial to users *and* transparent or beneficial to developers.

I'm williing to participate in discussion and offer what I know.  In a few months I could even help work with porting to a new build system.  However currently I just don't have the time.


- Josh

My 2 cents:

• Modularizing Scala is a very good idea, especially if we can also deal with the fragile version issue
• I'd be interested in seeing Scala bootstrap itself into an OSGi container so the version dependency issue can be more clearly managed.  This means that typing "scala" from the command line to get the Scala prompt would actually invoke Felix (or some other lightweight OSGi container) to run code
• Modularizing should also keep in mind a custom class loader so we can do PaulP's ShakeIns (mixing a trait into a runtime instance)
• I'd love to see everything that's no in the scala and scala.runtime packages (and the referenced collections) be in separate modules
• Build-wise I default to Maven, but I think Josh's analysis is dead-on.  I also would not like to see depencies on anything that's no JVM-based
• I think as part of this process, the Scala code base should move to Git.  Git fundimentally changes the development process and encourages branches and exploration as well as some significantly better processes for approving changes on the main branch.
• I'd like to see some significant improvements in the Scala release process.  Time-based milestone releases would be great as well as a real beta cycle for near-release of dot versions.  I think as part of modularizing Scala, enhancing the process would be a huge win and ultimately reduce the friction for what EPFL has to deal with so they can work more on research and design and not worry.

Actual steps:

• I'd love it if someone (Josh, Nathan, someone) would fork PaulP's GitHub version of Scala and see if they could SBT it
• Run a dependency analysis on the various packages in Scala already.  Which ones are hard-coded into the compiler?  Which ones reference others?  If we could make a list of "zero dependency" packages, we could see what can be put into modules right away

I'm also currious as to any actual requirements for shipping 2.8 by certain dates.  If there are requirements, let's know what those dates are.  If there are no hard dates, I suggest that we make 2.8 the "big breaking release" and fix more than collections.

On Wed, Sep 9, 2009 at 5:58 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

Having spend a lot of time weeding through scala's build system, I feel inclined to add some input here.


1) Adding Ivy to scala's current build system does not seem like a win.   The dependency section of the scala build process is actually a smaller contributor to complexity/effort (unlike other projects I've been on).  I ended up using the maven-ant-task just to deploy artifacts into a maven repository.  Ignoring the hype over Ivy vs. Maven dependency management for a bit, the maven repository is the current "standard" in the sense that every tool can interface with it.  If we seriously want to consider using Ivy, then we should seriously consider using SBT where the API for declaring dependencies and deployment is sane.  Also there are a number of big wins to SBT.

2) No matter what build system you migrate Sabbus to, you're going to have to fight with the lifecycle of the build.  Maven could make this difficult as you would almost need a custom lifecycle for the scala bootstrap process (no mean task there).  I'm pretty sure we could bend maven far enough for this, but I'm not sure I'm buying the win in terms of effort.  I love maven, and use it all the time, but the scala build system is very far outside of its lifecycle.

3) SBT seems like a win in terms of effort vs. gain.  It has amazing performance, is written in Scala (and therefore much easier for Scala developers to be able to maintain than a Java-1.4-API'd tool), and has the flexibility to support Sabbus's lifecycle.  It also supports mutli-module projects, so I don't see any major reasons to use maven over SBT at this point.  SBT would need to add some form of BND integration, but BND is a very nice, easy API to talk to at this point.  I had debated trying to port scala's build environment into SBT, however time is at a premium and I've minimized my commitments to things that I need for work, or high-priority bugs in maven-related areas (as the "coordinator", I feel obligated).

I love the idea of modularizing Scala.  I think working on libraries individually will be great.  Please make sure you don't ignore the existing build process or make assumptions that are not true.

I took a shot at modularizing Scala a while back and ran into some issues (that could be solved with 2.8 features).  In all of our efforts, I'd love to make sure we don't over-engineer a solution.  The solution should be beneficial to users *and* transparent or beneficial to developers.

I'm williing to participate in discussion and offer what I know.  In a few months I could even help work with porting to a new build system.  However currently I just don't have the time.


- Josh

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

If we want to identify and tease apart dependencies, then it must be worthwhile having someone try and sweet-talk us a community licence for one of the more powerful DSM tools such as lattix LDM or structure101
Preferably someone with better diplomatic skills than myself :)
For the curious, there's a definition of DSM and list of software on wikipedia:http://en.wikipedia.org/wiki/Design_Structure_Matrix


On Fri, Sep 11, 2009 at 12:46 AM, David Pollak <feeder.of.the.bears@gmail.com> wrote:

My 2 cents:

• Modularizing Scala is a very good idea, especially if we can also deal with the fragile version issue
• I'd be interested in seeing Scala bootstrap itself into an OSGi container so the version dependency issue can be more clearly managed.  This means that typing "scala" from the command line to get the Scala prompt would actually invoke Felix (or some other lightweight OSGi container) to run code
• Modularizing should also keep in mind a custom class loader so we can do PaulP's ShakeIns (mixing a trait into a runtime instance)
• I'd love to see everything that's no in the scala and scala.runtime packages (and the referenced collections) be in separate modules
• Build-wise I default to Maven, but I think Josh's analysis is dead-on.  I also would not like to see depencies on anything that's no JVM-based
• I think as part of this process, the Scala code base should move to Git.  Git fundimentally changes the development process and encourages branches and exploration as well as some significantly better processes for approving changes on the main branch.
• I'd like to see some significant improvements in the Scala release process.  Time-based milestone releases would be great as well as a real beta cycle for near-release of dot versions.  I think as part of modularizing Scala, enhancing the process would be a huge win and ultimately reduce the friction for what EPFL has to deal with so they can work more on research and design and not worry.

Actual steps:

• I'd love it if someone (Josh, Nathan, someone) would fork PaulP's GitHub version of Scala and see if they could SBT it
• Run a dependency analysis on the various packages in Scala already.  Which ones are hard-coded into the compiler?  Which ones reference others?  If we could make a list of "zero dependency" packages, we could see what can be put into modules right away

I'm also currious as to any actual requirements for shipping 2.8 by certain dates.  If there are requirements, let's know what those dates are.  If there are no hard dates, I suggest that we make 2.8 the "big breaking release" and fix more than collections.

On Wed, Sep 9, 2009 at 5:58 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

Having spend a lot of time weeding through scala's build system, I feel inclined to add some input here.


1) Adding Ivy to scala's current build system does not seem like a win.   The dependency section of the scala build process is actually a smaller contributor to complexity/effort (unlike other projects I've been on).  I ended up using the maven-ant-task just to deploy artifacts into a maven repository.  Ignoring the hype over Ivy vs. Maven dependency management for a bit, the maven repository is the current "standard" in the sense that every tool can interface with it.  If we seriously want to consider using Ivy, then we should seriously consider using SBT where the API for declaring dependencies and deployment is sane.  Also there are a number of big wins to SBT.

2) No matter what build system you migrate Sabbus to, you're going to have to fight with the lifecycle of the build.  Maven could make this difficult as you would almost need a custom lifecycle for the scala bootstrap process (no mean task there).  I'm pretty sure we could bend maven far enough for this, but I'm not sure I'm buying the win in terms of effort.  I love maven, and use it all the time, but the scala build system is very far outside of its lifecycle.

3) SBT seems like a win in terms of effort vs. gain.  It has amazing performance, is written in Scala (and therefore much easier for Scala developers to be able to maintain than a Java-1.4-API'd tool), and has the flexibility to support Sabbus's lifecycle.  It also supports mutli-module projects, so I don't see any major reasons to use maven over SBT at this point.  SBT would need to add some form of BND integration, but BND is a very nice, easy API to talk to at this point.  I had debated trying to port scala's build environment into SBT, however time is at a premium and I've minimized my commitments to things that I need for work, or high-priority bugs in maven-related areas (as the "coordinator", I feel obligated).

I love the idea of modularizing Scala.  I think working on libraries individually will be great.  Please make sure you don't ignore the existing build process or make assumptions that are not true.

I took a shot at modularizing Scala a while back and ran into some issues (that could be solved with 2.8 features).  In all of our efforts, I'd love to make sure we don't over-engineer a solution.  The solution should be beneficial to users *and* transparent or beneficial to developers.

I'm williing to participate in discussion and offer what I know.  In a few months I could even help work with porting to a new build system.  However currently I just don't have the time.


- Josh

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

On Thursday 10 September 2009, David Pollak wrote:
> - Run a dependency analysis on the various packages in Scala already.
> Which ones are hard-coded into the compiler? Which ones reference
> others? If we could make a list of "zero dependency" packages, we could see
> what can be put into modules right away

As a start, here is a graphviz dot file containing the package-level
dependencies for the Scala library based on compile-time source dependency
analysis.

For a basic image:
dot -Tpng -o pkgDeps.png pkgDeps.dot
or
neato -Goverlap=false -Tpng -o pkgDeps.png pkgDeps.dot

(Better visualization suggestions welcome!)

Thanks,
Mark

How was it generated?

On Fri, Sep 11, 2009 at 12:29 AM, Mark Harrah <harrah@bu.edu> wrote:

On Thursday 10 September 2009, David Pollak wrote:
>    - Run a dependency analysis on the various packages in Scala already.
>    Which ones are hard-coded into the compiler?  Which ones reference
> others? If we could make a list of "zero dependency" packages, we could see
> what can be put into modules right away

As a start, here is a graphviz dot file containing the package-level
dependencies for the Scala library based on compile-time source dependency
analysis.

For a basic image:
 dot -Tpng -o pkgDeps.png pkgDeps.dot
or
 neato -Goverlap=false -Tpng -o pkgDeps.png pkgDeps.dot

(Better visualization suggestions welcome!)

Thanks,
Mark

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

For those without graphviz, can you summarise?

Note that the original proposal identified some candidate modules,
which will probably correspond with some of these. The issue is more
of the symmetric relations (and whether there is a any refactoring
that can be done to break them) - but that is likely to evolve later
in the process.

There may also be different use cases for some packages; the current
scala.io might be broken up into the mechanics of IO (streams,
exceptions) and dealing with particular file types (Source et al).

Alex

Sent from my (new) iPhone

On 11 Sep 2009, at 04:29, Mark Harrah wrote:

> On Thursday 10 September 2009, David Pollak wrote:
>> - Run a dependency analysis on the various packages in Scala
>> already.
>> Which ones are hard-coded into the compiler? Which ones reference
>> others? If we could make a list of "zero dependency" packages, we
>> could see
>> what can be put into modules right away
>
> As a start, here is a graphviz dot file containing the package-level
> dependencies for the Scala library based on compile-time source
> dependency
> analysis.
>
> For a basic image:
> dot -Tpng -o pkgDeps.png pkgDeps.dot
> or
> neato -Goverlap=false -Tpng -o pkgDeps.png pkgDeps.dot
>
> (Better visualization suggestions welcome!)
>
> Thanks,
> Mark
>

On 11 Sep 2009, at 00:46, David Pollak
wrote:

> My 2 cents:
> I'd be interested in seeing Scala bootstrap itself into an OSGi
> container so the version dependency issue can be more clearly
> managed. This means that typing "scala" from the command line to
> get the Scala prompt would actually invoke Felix (or some other
> lightweight OSGi container) to run code
One of the requirements identified has been to allow Scala to run
outside OSGi and for the command line scala to continue as before by
concatenating JARs.

So whilst it should still be possible to do this, we probably can't
make it the default. That doesn't prevent an additional tool/script
booting an OSGi environment and getting the system up but you still
run into questions like what bundle(s) process the command line args
etc.
> Modularizing should also keep in mind a custom class loader so we
> can do PaulP's ShakeIns (mixing a trait into a runtime instance)
The modularisation effort is more about compile-time separation than
runtime at the moment. And whilst we need OSGi for the IDE support, it
doesn't necessarily preclude other types of runtime.
> Build-wise I default to Maven
What does lift do with versioned dependencies between modules? What
happens when you need to update the version across modules? Or do you
use version ranges?
> I think as part of this process, the Scala code base should move to
> Git.
Whilst I agree in principle (and I hope it's taken on board), it's
really separate from modularisation and one shouldn't depend on (or
preclude) the other.
> I'd like to see some significant improvements in the Scala release
> process.
That would also be welcome, but same as above.
> I'd love it if someone (Josh, Nathan, someone) would fork PaulP's
> GitHub version of Scala and see if they could SBT it

Yes, would be interesting to know whether that works. Prehaps we can
start small (with just actors) to test the approach?

> I'm also currious as to any actual requirements for shipping 2.8 by
> certain dates. If there are requirements, let's know what those
> dates are. If there are no hard dates, I suggest that we make 2.8
> the "big breaking release" and fix more than collections.

One of the other points I made is that we need to get better at
indicating breakage by the version number. OSGi version semantics
suggest bumping up the major version when there's breakage, minor for
compatible enhancements and micro for bug fixes et al.

So instead of a 2.8 release, it should ideally be a 3.0 release if
it's going to be a big bang.

The other point to note is that modules should ideally have their own
version numbers (rather than just inheriting whatever the "scala
release number" is) and evolve independently. However, until we do the
split ups, this may be difficult to do in practice.

Alex

The elephant in the room here is MSIL/.net

Jigsaw and osgi are Java technologies, so we'll *have* to use some
sort of abstraction in order that scala modules can also be used on
the Microsoft platform.

On Wednesday, September 9, 2009, Alex Blewitt wrote:
> On 9 Sep 2009, at 15:11, David Savage wrote:
>
>
> On Wed, Sep 9, 2009 at 2:01 PM, Antonio Cunei wrote:
>
> On a related subject, concerning Maven/OSGi etc, Martin also observed that
> project Jigsaw also entails modularization. Any attempt to modularize Scala
> should also keep an eye on their progress and the conventions they use.
>
>
> Makes sense, though from my understanding the jigsaw work seems to
> moving at glacial speed
>
>
> The requirements I posted before suggested that we should try and treat as abstract modules rather than necessarily be tied to one system. It should be noted though that in order to continue IDE development, OSGi support is needed.
>
> As and when Jigsaw or equivalent is available, we could consider using that as well - but it's a work in progress at the moment.
>
> Alec
>

On 11 Sep 2009, at 09:17, Kevin Wright wrote:

> The elephant in the room here is MSIL/.net

Isn't it always? Still, at least it's SVN so it's always got a trunk.

> Jigsaw and osgi are Java technologies, so we'll *have* to use some
> sort of abstraction in order that scala modules can also be used on
> the Microsoft platform.

Right - in the requirements, I've avoided using any Java-specific
terms to describe Scala's modules. The only constraint was that they
also have to be OSGi bundles (at least, in the Java land) in order to
play nice with the IDE work.

There may come a time where we need to add dynamicity to the modules;
but for now, the dependency is directed, so a Scala module could
correspond with a .Net assembly on the .Net platform. The key is to
split up (big thing) into (little thing) (little thing) (little thing)
logically, after which it should be possible to transform said (little
thing)s into a unit of the appropriate flavour afterwards.

However, I don't know enough about how scala works on .Net, or .Net
assemblies in general, to know how to accomplish this. Does the
current build system build for .Net concurrently with the scala-
library.jar, or is there a separate build tool for that?

Alex

On 10 Sep 2009, at 01:58, Josh Suereth wrote:

> 1) Adding Ivy to scala's current build system does not seem like a
> win ... the maven repository is the current "standard" in the sense
> that every tool can interface with it.

Yes, that's a good point. Though before writing it off, does Ivy offer
any specific benefits over/above a Maven-style build (David S would
probably know the answer to this).

> 2) No matter what build system you migrate Sabbus to, you're going
> to have to fight with the lifecycle of the build.

Agreed, though it should be possible to have a bootstrap process that
refers to the latest build in order to generate the current build.
After all, that's what happens with the current Ant build system for
the compiler. What we need to ensure is that the compiler and the
compiler's dependencies are bootstrapped in the most appropriate way.

> I love maven, and use it all the time, but the scala build system is
> very far outside of its lifecycle.

I don't think it's that far out; we're just looking at a bootstrap
compiler, rather than anything more complicated. (Or I'm
misunderstanding the difficulties in the scala build system)

> 3) SBT seems like a win in terms of effort vs. gain. It has amazing
> performance, is written in Scala (and therefore much easier for
> Scala developers to be able to maintain than a Java-1.4-API'd tool),
> and has the flexibility to support Sabbus's lifecycle. It also
> supports mutli-module projects, so I don't see any major reasons to
> use maven over SBT at this point.

Does SBT provide versioned dependencies (e.g. scala-actors depends on
scala-core 2.8..2.9) ? It seems like we'd need that in the build
system, if the module dependencies are to be versioned. Looking at the
documentation of SBT, it looks like it makes certain assumptions about
the names of modules (as well as lack of version numbers in those
modules).

> SBT would need to add some form of BND integration, but BND is a
> very nice, easy API to talk to at this point.

Yes, that's true. However, don't forget that we also need to ensure
the IDE is catered for ...

> I love the idea of modularizing Scala. I think working on libraries
> individually will be great. Please make sure you don't ignore the
> existing build process or make assumptions that are not true.

Would like to know whether SBT has versioned dependencies, and that
BND can be wired in somehow to to the build process. In addition, it
would be good to see how a build system could be set up to depend on
(e.g.) scala-core, scala-actors and so forth. If not, then I'm not
sure that the modularisation effort should depend on a tool that isn't
ready for that purpose yet (though I could see it being used at a
later stage). The longer we have Scala-library as one gigantic module,
the more difficult it is going to be to pick it apart at a later stage.

Ultimately, if we get a standard layout for Scala source files (e.g.
src/main/scala) and actually do the splitting, then it should be
possible to migrate to/from different build tools at a later stage.
The key is unpicking (and moving) the Scala source files ...

Alex

On Fri, Sep 11, 2009 at 10:06 AM, Alex Blewitt wrote:
>> I love maven, and use it all the time, but the scala build system is very
>> far outside of its lifecycle.
>
> I don't think it's that far out; we're just looking at a bootstrap compiler,
> rather than anything more complicated. (Or I'm misunderstanding the
> difficulties in the scala build system)

A lot of the complexity of the current build system is there to
support reasonably quick incremental build times for people working on
the compiler and the standard libraries (which in part, don't forget,
are both compiled by and used by the compiler, so are also part of the
bootstrapping process).

Could a Maven-based setup support both incremental and fully
bootstrapped final builds?

Cheers,

Miles

"Could a Maven-based setup support both incremental and fully bootstrapped final builds?"

I reckon it *could*, even if we have to go with multiple poms.  but I also wouldn't like to be the one maintaining that little nightmare...

Hi, Mark,

On Thursday September 10 2009, Mark Harrah wrote:
> On Thursday 10 September 2009, David Pollak wrote:
> > - Run a dependency analysis on the various packages in Scala
> > already. ...
>
> As a start, here is a graphviz dot file containing the package-level
> dependencies for the Scala library based on compile-time source
> dependency analysis.
>
> For a basic image:
> dot -Tpng -o pkgDeps.png pkgDeps.dot
> or
> neato -Goverlap=false -Tpng -o pkgDeps.png pkgDeps.dot

As we talked about on #scala, I spent some time trying to get a better
rendering of this graph. I didn't have particularly good luck, but I
made some minor improvements in very non-inspired ways:

- Shorten the node labels
scala -> s
collection -> c
mutable -> m
immutable -> i
xml -> x
combinator -> cmbtr
reflect -> refl
runtime -> rt
util -> u
control -> ctrl
syntatical -> syn

- Remove inbound dependencies to select packages:
scala (or s)
scala.runtime (s.rt)

- Add the "splines=true" to the graph properties. This keeps the arcs
from crossing the nodes, but also makes them nearly impossible to
follow, since in many places they overlap and merge.

- Use a smaller, sans-serif font (*)

This is the invocation I used:

% neato -Goverlap=false -Gsplines=true \
-Nfontname=LucidaSans -Nfontsize=9 \
-Tpng -o pkgDeps.png pkgDeps.dot

I attached the munged pkgDeps.dot file (the "removed" inbound
dependencies are just commented, not removed from the file).

> (Better visualization suggestions welcome!)
>
> Thanks,
> Mark

Randall Schulz

Thanks Randall. Removing scala and scala.runtime helps.

To answer Daniel's question, I ran sbt 0.5.3's graph-pkg action on the Scala
library.

-Mark

On Friday 11 September 2009, Randall R Schulz wrote:
> Hi, Mark,
>
> On Thursday September 10 2009, Mark Harrah wrote:
> > On Thursday 10 September 2009, David Pollak wrote:
> > > - Run a dependency analysis on the various packages in Scala
> > > already. ...
> >
> > As a start, here is a graphviz dot file containing the package-level
> > dependencies for the Scala library based on compile-time source
> > dependency analysis.
> >
> > For a basic image:
> > dot -Tpng -o pkgDeps.png pkgDeps.dot
> > or
> > neato -Goverlap=false -Tpng -o pkgDeps.png pkgDeps.dot
>
> As we talked about on #scala, I spent some time trying to get a better
> rendering of this graph. I didn't have particularly good luck, but I
> made some minor improvements in very non-inspired ways:
>
> - Shorten the node labels
> scala -> s
> collection -> c
> mutable -> m
> immutable -> i
> xml -> x
> combinator -> cmbtr
> reflect -> refl
> runtime -> rt
> util -> u
> control -> ctrl
> syntatical -> syn
>
> - Remove inbound dependencies to select packages:
> scala (or s)
> scala.runtime (s.rt)
>
> - Add the "splines=true" to the graph properties. This keeps the arcs
> from crossing the nodes, but also makes them nearly impossible to
> follow, since in many places they overlap and merge.
>
> - Use a smaller, sans-serif font (*)
>
> This is the invocation I used:
>
> % neato -Goverlap=false -Gsplines=true \
> -Nfontname=LucidaSans -Nfontsize=9 \
> -Tpng -o pkgDeps.png pkgDeps.dot
>
>
> I attached the munged pkgDeps.dot file (the "removed" inbound
> dependencies are just commented, not removed from the file).
>
> > (Better visualization suggestions welcome!)
> >
> > Thanks,
> > Mark
>
> Randall Schulz

On 11 Sep 2009, at 10:30, Miles Sabin wrote:

> On Fri, Sep 11, 2009 at 10:06 AM, Alex Blewitt
> wrote:
>>> I love maven, and use it all the time, but the scala build system
>>> is very
>>> far outside of its lifecycle.
>>
>> I don't think it's that far out; we're just looking at a bootstrap
>> compiler,
>> rather than anything more complicated. (Or I'm misunderstanding the
>> difficulties in the scala build system)
>
> A lot of the complexity of the current build system is there to
> support reasonably quick incremental build times for people working on
> the compiler and the standard libraries (which in part, don't forget,
> are both compiled by and used by the compiler, so are also part of the
> bootstrapping process).

Agreed. Though it would be interesting to see what modules the
compiler depends on; it possibly is a subset of the current scala-
library. It seems that there are really two issues:

1) Can it be bootstrapped?
2) Is it quick/incremental?

Regarding the bootstrap process; Maven can generate an installable
module and install it into the local repo, so that the next time it
runs, it can use that value. If the maven-scala-plugin depended on
(say) scala-compiler-LATEST, then a subsequent run would be against
this new module. So one could achieve this bootstrapping process by
running Maven twice (or even having an 'ant' script to do that for
you). That of course depends on ...

Is it quick - part of the problem with the current setup is that re-
running a build involves recompiling the world. If we've not had any
changes in (say) the actors module, we shouldn't need to recompile the
actors module to re-run the build, even if the compiler ends up
depending on the actors module. Being able to explicitly break out
these dependencies will give savings in this manner, like Make's
"Nothing to be done for ..."

Regarding incremental compilation; others who have used the Scala
plugin more extensively than I will know the answer to this one. I
think it's as likely that the maven-scala-plugin will be able to do
this as sbt will; after all, the issue is usually not whether
foo.scala is newer than foo.class, because bar.scala might be older
than bar.class (i.e. up-to-date) but inherit a trait changed in
foo.scala.

What is true is that maven doesn't have to do a clean between builds,
so that if you've previously run a compile pass over a set of
projects, those outputs will be there next time you run maven. If the
maven-scala-plugin is capable of using that to perform a lesser
operation than compile-the-world, then that's great - if not, then it
will not be incremental. However, enabling maven-scala-plugin to be
incremental would then allow other projects which use it to become
noticably faster as well.

So, I'd be curious to know how sbt manages to calculate its
dependencies for each file.

Alex

The maven-scala-plugin's recompile strategy is not as robust as SBT's.  I looked into using SBT's for the maven-scala-pluigin as well, but unfortunately, haven't had time to even attempt to pull it in.

SBT makes use of Ivy for its dependency management.  Also, it's easier to customize tasks inside scala.  I'm still working on supporting custom task definition in maven.   As I stated earlier, I truly believe SBT is the right solution for this task.  I understand maven well enough to know the work involved to update scala's build process.  I truly believe this isn't the best of ideas, as there's too much inflexibility (or at least too much work to make it flexible) for what Scala wants. 

If you want to discuss how a solution in maven would look, we can.  I feel we have more control over SBT's build at this point (less users => more pull) to make sure that we can get the optimizations we would need for multi-module builds.  At work we use a highly modular build, and it's really tough to do "just" recompilation when you're start mixing scala in the picture.

- Josh



On Thu, Sep 17, 2009 at 4:47 AM, Alex Blewitt <alex.blewitt@gmail.com> wrote:

On 11 Sep 2009, at 10:30, Miles Sabin wrote:

On Fri, Sep 11, 2009 at 10:06 AM, Alex Blewitt <alex.blewitt@gmail.com> wrote:

I love maven, and use it all the time, but the scala build system is very
far outside of its lifecycle.

I don't think it's that far out; we're just looking at a bootstrap compiler,
rather than anything more complicated. (Or I'm misunderstanding the
difficulties in the scala build system)

A lot of the complexity of the current build system is there to
support reasonably quick incremental build times for people working on
the compiler and the standard libraries (which in part, don't forget,
are both compiled by and used by the compiler, so are also part of the
bootstrapping process).

Agreed. Though it would be interesting to see what modules the compiler depends on; it possibly is a subset of the current scala-library. It seems that there are really two issues:

1) Can it be bootstrapped?
2) Is it quick/incremental?

Regarding the bootstrap process; Maven can generate an installable module and install it into the local repo, so that the next time it runs, it can use that value. If the maven-scala-plugin depended on (say) scala-compiler-LATEST, then a subsequent run would be against this new module. So one could achieve this bootstrapping process by running Maven twice (or even having an 'ant' script to do that for you). That of course depends on ...

Is it quick - part of the problem with the current setup is that re-running a build involves recompiling the world. If we've not had any changes in (say) the actors module, we shouldn't need to recompile the actors module to re-run the build, even if the compiler ends up depending on the actors module. Being able to explicitly break out these dependencies will give savings in this manner, like Make's "Nothing to be done for ..."

Regarding incremental compilation; others who have used the Scala plugin more extensively than I will know the answer to this one. I think it's as likely that the maven-scala-plugin will be able to do this as sbt will; after all, the issue is usually not whether foo.scala is newer than foo.class, because bar.scala might be older than bar.class (i.e. up-to-date) but inherit a trait changed in foo.scala.

What is true is that maven doesn't have to do a clean between builds, so that if you've previously run a compile pass over a set of projects, those outputs will be there next time you run maven. If the maven-scala-plugin is capable of using that to perform a lesser operation than compile-the-world, then that's great - if not, then it will not be incremental. However, enabling maven-scala-plugin to be incremental would then allow other projects which use it to become noticably faster as well.

So, I'd be curious to know how sbt manages to calculate its dependencies for each file.

Alex

On 17 Sep 2009, at 21:57, Josh Suereth wrote:

> The maven-scala-plugin's recompile strategy is not as robust as
> SBT's. I looked into using SBT's for the maven-scala-pluigin as
> well, but unfortunately, haven't had time to even attempt to pull it
> in.

Yeah, the problem isn't just in the simple stuff but generating the
right transitive closure for the dependencies of each file - it's a
tricky task. One of the reasons Java tends to compile quickly is
because of the one public class to file name relationship, something
which Scala doesn't have. The other issue is that of implicits, which
can widen the dependency scope inadvertently ...

> SBT makes use of Ivy for its dependency management. Also, it's
> easier to customize tasks inside scala. I'm still working on
> supporting custom task definition in maven. As I stated earlier, I
> truly believe SBT is the right solution for this task. I understand
> maven well enough to know the work involved to update scala's build
> process. I truly believe this isn't the best of ideas, as there's
> too much inflexibility (or at least too much work to make it
> flexible) for what Scala wants.

I think as long as we can pull out the locations of the sources into
independent SCM locations, and define a logical dependency graph
between them, and be flexible on the build. We may be able to do this
with an extension of the current Ant build; it might be the case that
Maven or SBT (or even something else) could then be used to build it
afterwards.

Hopefully the goal of modularisation is orthogonal to the choice of
build tool, so as long as we can get started with building, we don't
need to commit to any one specific idea. We should probably explore
both. (NB there are other aspects, like generating the BND information
for the IDE, that might need to be factored in.)

> If you want to discuss how a solution in maven would look, we can.
> I feel we have more control over SBT's build at this point (less
> users => more pull) to make sure that we can get the optimizations
> we would need for multi-module builds. At work we use a highly
> modular build, and it's really tough to do "just" recompilation when
> you're start mixing scala in the picture.

There's another aspect here; inter-module dependencies. For example:

B extends A
B is in module b
A is in module a

If A changes, we need to recompile B. But if they're in different
modules, how do we know that this dependency has been invalidated? At
the very least, an automated build should do a clean build each time
to catch these things (http://runcoderun.com/dpp/liftweb suggests the
average full build is around 8 minutes).

Alex