Milestone Release Process Proposal

+1 if it can be co-ordinated properly
It could also create a lot of work in reconfiguring Trac to support the workflow

On Wed, Aug 12, 2009 at 3:27 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

Hey Scala Committers/EPFL,

I was curious if anyone else thought it might be useful to make "Milestone" releases in the future of scala.  The idea here is to make "mini releases" of trunk after significant items are feature-complete.  These releases can and will have bugs in them.   I'd really love to see these happen throughout the development of a release for the following reasons:

• It promotes better communication between the community and the committers on what features will be in the latest release
• These would be made much earlier than Releace Candidate builds and would therefore have more opportunity to be tested
  
• It promotes the community of libraries built around scala
• By making milestone releases, active scala libraries can support branches against the latest language features
• This encourages lots of testing/use of the new libraries
• This allows the entire scala community (lift, scalaz, sbt, etc.) to evolve with the scala library, somewhat like how the eclipse project evolves.
• Earlier feedback from users
  

The scala community has already been requesting for scala-tools.org to provision some kind of "stable reference" of scala 2.8.0, and we've been happy to oblige them.   However in the future, I was thinking these kinds of decisions should be made with the insight/guidance of the core scala committers.   In 2.8.0 alone, I see the "named-and-default" parameters feature meriting a feature-preview-release as well as the new collections API. 


I'm curious on your insight into the matter, and hope to hear from all of you!

- Josh

Josh,

I am not sure the proposal is very practical, or even advisable. The
development process of the various features takes place in a very
articulated way, sometimes but not always in separate branches, and always
in parallel. That means that:
- once an individual new feature that is being developed in trunk is
feature-complete, trunk might be totally unusable because other features
being developed concurrently are undergoing extensive development or
rewriting. So a release may be impossible for some time
- if a feature developed in a branch is complete, testing still offers no
information about how it will interact with all the rest that is going on
at the same time elsewhere. Once merged to trunk, we are again in the same
case as above
My opinion is that the development process in the compiler is too organic
to be reduced to a sequence of milestones, mainly because the various
features are not at all developed sequentially. Further, they frequently
interact, and need to be developed together.
At present we do plan to release at some stage a beta version, but that
will happen only when all the new stuff that we expect for 2.8 reaches, in
parallel, a testable state. In the meantime, we always have the nightly
builds, and the feedback we gather from the community is absolutely
fantastic in that respect.
So, forcing development through milestone releases might in fact hinder,
rather than improve, the development process in my view.
Toni

Josh Suereth wrote:
> Hey Scala Committers/EPFL,
>
> I was curious if anyone else thought it might be useful to make "Milestone"
> releases in the future of scala. The idea here is to make "mini releases"
> of trunk after significant items are feature-complete. These releases can
> and will have bugs in them. I'd really love to see these happen throughout
> the development of a release for the following reasons:
>
>
> - It promotes better communication between the community and the
> committers on what features will be in the latest release
> - These would be made much earlier than Releace Candidate builds and
> would therefore have more opportunity to be tested
> - It promotes the community of libraries built around scala
> - By making milestone releases, active scala libraries can support
> branches against the latest language features
> - This encourages lots of testing/use of the new libraries
> - This allows the entire scala community (lift, scalaz, sbt, etc.) to
> evolve with the scala library, somewhat like how the eclipse project
> evolves.
> - Earlier feedback from users
>
> The scala community has already been requesting for scala-tools.org to
> provision some kind of "stable reference" of scala 2.8.0, and we've been
> happy to oblige them. However in the future, I was thinking these kinds of
> decisions should be made with the insight/guidance of the core scala
> committers. In 2.8.0 alone, I see the "named-and-default" parameters
> feature meriting a feature-preview-release as well as the new collections
> API.
>
>
> I'm curious on your insight into the matter, and hope to hear from all of
> you!
>
> - Josh
>

Josh,

I am not sure the proposal is very practical, or even advisable. The
development process of the various features takes place in a very
articulated way, sometimes but not always in separate branches, and always
in parallel. That means that:
- once an individual new feature that is being developed in trunk is
feature-complete, trunk might be totally unusable because other features
being developed concurrently are undergoing extensive development or
rewriting. So a release may be impossible for some time
- if a feature developed in a branch is complete, testing still offers no
information about how it will interact with all the rest that is going on
at the same time elsewhere. Once merged to trunk, we are again in the same
case as above
My opinion is that the development process in the compiler is too organic
to be reduced to a sequence of milestones, mainly because the various
features are not at all developed sequentially. Further, they frequently
interact, and need to be developed together.
At present we do plan to release at some stage a beta version, but that
will happen only when all the new stuff that we expect for 2.8 reaches, in
parallel, a testable state. In the meantime, we always have the nightly
builds, and the feedback we gather from the community is absolutely
fantastic in that respect.
So, forcing development through milestone releases might in fact hinder,
rather than improve, the development process in my view.
Toni

Josh Suereth wrote:
> Hey Scala Committers/EPFL,
>
> I was curious if anyone else thought it might be useful to make "Milestone"
> releases in the future of scala. The idea here is to make "mini releases"
> of trunk after significant items are feature-complete. These releases can
> and will have bugs in them. I'd really love to see these happen throughout
> the development of a release for the following reasons:
>
>
> - It promotes better communication between the community and the
> committers on what features will be in the latest release
> - These would be made much earlier than Releace Candidate builds and
> would therefore have more opportunity to be tested
> - It promotes the community of libraries built around scala
> - By making milestone releases, active scala libraries can support
> branches against the latest language features
> - This encourages lots of testing/use of the new libraries
> - This allows the entire scala community (lift, scalaz, sbt, etc.) to
> evolve with the scala library, somewhat like how the eclipse project
> evolves.
> - Earlier feedback from users
>
> The scala community has already been requesting for scala-tools.org to
> provision some kind of "stable reference" of scala 2.8.0, and we've been
> happy to oblige them. However in the future, I was thinking these kinds of
> decisions should be made with the insight/guidance of the core scala
> committers. In 2.8.0 alone, I see the "named-and-default" parameters
> feature meriting a feature-preview-release as well as the new collections
> API.
>
>
> I'm curious on your insight into the matter, and hope to hear from all of
> you!
>
> - Josh
>

Toni,

I definitely understand your position, and in no way do I want to hinder development of scala!   My initial thoughts for such milestone releases would be to take the first nightly that has all existing tests passing.  My thinking is that the community can help identify feature mis-match in the milestones along with the core developers.  Ideally, this feedback could help identify areas of concern that might have gone unnoticed until later in the development cycle.

If this is too much of a burden on scala development, then I ask that nothing be done.   I know Scala 2.8.0 has been gaining lots of interest and I know some of the libraries have been asking for a "stable identifier" to attempt to integrate to 2.8.0 and provide their users with "nightly" releases.   It's important to use the same version of Scala accross multiple libraries for this reason.

Right now, the community is just "mini-releasing"   stable nightlies for this purpose.   If that's the way we do things moving forward, then I'm ok with that.  I was just thinking perhaps the core development team could provide more insight into when these "mini releases" should be made.

- Josh

On Wed, Aug 12, 2009 at 6:48 AM, Antonio Cunei <antonio.cunei@epfl.ch> wrote:

Josh,

I am not sure the proposal is very practical, or even advisable. The development process of the various features takes place in a very articulated way, sometimes but not always in separate branches, and always in parallel. That means that:
- once an individual new feature that is being developed in trunk is feature-complete, trunk might be totally unusable because other features being developed concurrently are undergoing extensive development or rewriting. So a release may be impossible for some time
- if a feature developed in a branch is complete, testing still offers no information about how it will interact with all the rest that is going on at the same time elsewhere. Once merged to trunk, we are again in the same case as above
My opinion is that the development process in the compiler is too organic to be reduced to a sequence of milestones, mainly because the various features are not at all developed sequentially. Further, they frequently interact, and need to be developed together.
At present we do plan to release at some stage a beta version, but that will happen only when all the new stuff that we expect for 2.8 reaches, in parallel, a testable state. In the meantime, we always have the nightly builds, and the feedback we gather from the community is absolutely fantastic in that respect.
So, forcing development through milestone releases might in fact hinder, rather than improve, the development process in my view.
Toni

Josh Suereth wrote:

Hey Scala Committers/EPFL,

I was curious if anyone else thought it might be useful to make "Milestone"
releases in the future of scala.  The idea here is to make "mini releases"
of trunk after significant items are feature-complete.  These releases can
and will have bugs in them.   I'd really love to see these happen throughout
the development of a release for the following reasons:


  - It promotes better communication between the community and the
  committers on what features will be in the latest release
  - These would be made much earlier than Releace Candidate builds and
  would therefore have more opportunity to be tested
  - It promotes the community of libraries built around scala
     - By making milestone releases, active scala libraries can support
     branches against the latest language features
     - This encourages lots of testing/use of the new libraries
     - This allows the entire scala community (lift, scalaz, sbt, etc.) to
     evolve with the scala library, somewhat like how the eclipse project
     evolves.
  - Earlier feedback from users

The scala community has already been requesting for scala-tools.org to
provision some kind of "stable reference" of scala 2.8.0, and we've been
happy to oblige them.   However in the future, I was thinking these kinds of
decisions should be made with the insight/guidance of the core scala
committers.   In 2.8.0 alone, I see the "named-and-default" parameters
feature meriting a feature-preview-release as well as the new collections
API.


I'm curious on your insight into the matter, and hope to hear from all of
you!

- Josh

> Right now, the community is just "mini-releasing"   stable nightlies for
> this purpose.   If that's the way we do things moving forward, then I'm ok
> with that.  I was just thinking perhaps the core development team could
> provide more insight into when these "mini releases" should be made.
our build server can easily be used to see whether tests are passing
etc: http://scala-webapps.epfl.ch/hudson/

adriaan

On Wed, August 12, 2009 2:40 pm, Adriaan Moors wrote:
>> Right now, the community is just "mini-releasing"   stable nightlies for
>> this purpose.   If that's the way we do things moving forward, then I'm
>> ok
>> with that.  I was just thinking perhaps the core development team could
>> provide more insight into when these "mini releases" should be made.
> our build server can easily be used to see whether tests are passing
> etc: http://scala-webapps.epfl.ch/hudson/
>
> adriaan
>

Also, I understand that ever since we have Hudson in place, only those
nightly builds that pass all tests are pushed to the website. So, in
principle, every nightly on the website is good for testing (assuming the
desired features are present in some form :)
Toni

Just to throw fuel on the fire...
I know it's a bit late in the day for 2.8, but for future efforts has anyone seriously considered migrating to distributed version control? (i.e. Git)
Basically we could use anything with strong enough branch/merge support such that features can be worked on in different branches, allowing for feature-specific milestones in advance of global Scala milestones.

The other benefit is that a co-ordinator could collect assorted patches for (e.g.. scala.swing), merge them into a local repo, and then push the collected changes as a single changeset into the master repo.  This makes it far easier for someone with expertise in a given component to manage submissions for that component.



On Wed, Aug 12, 2009 at 4:36 PM, Antonio Cunei <antonio.cunei@epfl.ch> wrote:

On Wed, August 12, 2009 2:40 pm, Adriaan Moors wrote:
>> Right now, the community is just "mini-releasing"   stable nightlies for
>> this purpose.   If that's the way we do things moving forward, then I'm
>> ok
>> with that.  I was just thinking perhaps the core development team could
>> provide more insight into when these "mini releases" should be made.
> our build server can easily be used to see whether tests are passing
> etc: http://scala-webapps.epfl.ch/hudson/
>
> adriaan
>

Also, I understand that ever since we have Hudson in place, only those
nightly builds that pass all tests are pushed to the website. So, in
principle, every nightly on the website is good for testing (assuming the
desired features are present in some form :)
Toni

I think the happy medium then is that the community can take a nightly (which passes all tests) and attach some kind of static identifier too it so that cross-scala-projects can begin playing with the "bleeding edge" and ensure that their needs are meet and that they can release in lock-step with Scala core. 

Anyway, thanks for the input.

- Josh

On Wed, Aug 12, 2009 at 11:36 AM, Antonio Cunei <antonio.cunei@epfl.ch> wrote:

On Wed, August 12, 2009 2:40 pm, Adriaan Moors wrote:
>> Right now, the community is just "mini-releasing"   stable nightlies for
>> this purpose.   If that's the way we do things moving forward, then I'm
>> ok
>> with that.  I was just thinking perhaps the core development team could
>> provide more insight into when these "mini releases" should be made.
> our build server can easily be used to see whether tests are passing
> etc: http://scala-webapps.epfl.ch/hudson/
>
> adriaan
>

Also, I understand that ever since we have Hudson in place, only those
nightly builds that pass all tests are pushed to the website. So, in
principle, every nightly on the website is good for testing (assuming the
desired features are present in some form :)
Toni

I think I would take that a little bit further.  I'm neither a Subversion nor a Hudson expert, but it seems like it wouldn't be too hard to create a branch called, say "stable_trunk," into which Hudson automatically merges changesets after all tests have passed, and then have the psuedo-nightly built from there.  That way projects that track closely to trunk would have a choice of using the nightly or pulling from svn.  Having the branch could help better link problems that affect downstream projects to the changesets that created them.
This may also be beneficial to core Scala developers who are working in their own branches, as they can pull changesets from the stable branch instead of trunk and thus be less affected by instability in trunk.

Stable would then be defined as "all tests pass."  The quality of the process would improve as the test suite improves.
I think in order for this to work well tests that don't pass would need to be left in trunk while they are being worked.  On many occasions I've noticed tests being swept to the side until they pass again.  This practice undermines any attempt to use the test suite as a quick gauge for stability.
On Wed, Aug 12, 2009 at 12:02 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

I think the happy medium then is that the community can take a nightly (which passes all tests) and attach some kind of static identifier too it so that cross-scala-projects can begin playing with the "bleeding edge" and ensure that their needs are meet and that they can release in lock-step with Scala core. 

Anyway, thanks for the input.

- Josh

On Wed, Aug 12, 2009 at 11:36 AM, Antonio Cunei <antonio.cunei@epfl.ch> wrote:

On Wed, August 12, 2009 2:40 pm, Adriaan Moors wrote:
>> Right now, the community is just "mini-releasing"   stable nightlies for
>> this purpose.   If that's the way we do things moving forward, then I'm
>> ok
>> with that.  I was just thinking perhaps the core development team could
>> provide more insight into when these "mini releases" should be made.
> our build server can easily be used to see whether tests are passing
> etc: http://scala-webapps.epfl.ch/hudson/
>
> adriaan
>

Also, I understand that ever since we have Hudson in place, only those
nightly builds that pass all tests are pushed to the website. So, in
principle, every nightly on the website is good for testing (assuming the
desired features are present in some form :)
Toni

--
http://erikengbrecht.blogspot.com/

First let me note that I fully agree with the general sentiment behind
this thread. Now to disagree:

On Thu, Aug 13, 2009 at 08:32:16AM -0400, Erik Engbrecht wrote:
> it seems like it wouldn't be too hard to create a branch called, say
> "stable_trunk," into which Hudson automatically merges changesets
> after all tests have passed, and then have the psuedo-nightly built
> from there.

The implied premise here is that trunk is broken a lot. But it isn't.
I use trunk all day every day, and if we ignore the MSIL build (which is
indeed broken a lot) it is very rare for trunk to be in a state where
the whole test suite isn't passing. Some of us get a whole bunch of
angry robot emails whenever the test suite blows up, so it's hard to
miss an extended sequence of failures.

> I think in order for this to work well tests that don't pass would
> need to be left in trunk while they are being worked. On many
> occasions I've noticed tests being swept to the side until they pass
> again.

On many occasions, really? I can think of a couple times martin has done
this, such as when he was checking in gigantic patches like the new
collections or changes in implicit resolution logic, but I can't think
of "many" times and I don't think it's ever done casually. When you
have a giant patch against trunk with some tests failing for obscure
reasons you are in a race against the clock because either trunk
development must be halted, your patch is drifting out of sync, or you
disable some tests and fix them later. I would not be inclined to take
that third choice off the table entirely out of loyalty to the primacy
of the test suite.

That's not to say the test suite doesn't need a lot more attention, as
it definitely does.

Paul,
I didn't mean to imply that trunk is broken a lot if you define "not broken" as "all tests pass," which I think is the only workable definition.  As for deactivating tests, I've noticed it on several occasions, not just the huge set that were deactivated when Martin committed the new collections library.  Perhaps that hasn't been done in a while, I haven't had time to actively follow trunk for a couple months now, but I still think it's an undesirable practice.

paulp said:

When you
have a giant patch against trunk with some tests failing for obscure
reasons you are in a race against the clock because either trunk
development must be halted, your patch is drifting out of sync, or you
disable some tests and fix them later.  I would not be inclined to take
that third choice off the table entirely out of loyalty to the primacy
of the test suite.

Why must trunk development be halted if there are some tests that don't pass?  Effectively by temporarily disabling a test you are saying "I know this doesn't pass, and I'm working on it, go about your business and keep on developing."  Communicating the same thing could be accomplished through a commit messaging noting that the test doesn't pass, creating a ticket in Trac, or a simple email to scala-dev or scala-internals.

I don't think trunk being broken is an inherently bad thing.  It's under active development, of course there are going to be things that are broken.  My feeling is somewhat the opposite.  The fact that the test suite almost always indicates that it is not broken and that there are a plethora of bugs in Trac seem to indicate something is out of balance.

-Erik


On Thu, Aug 13, 2009 at 9:26 AM, Paul Phillips <paulp@improving.org> wrote:

First let me note that I fully agree with the general sentiment behind
this thread.  Now to disagree:

On Thu, Aug 13, 2009 at 08:32:16AM -0400, Erik Engbrecht wrote:
> it seems like it wouldn't be too hard to create a branch called, say
> "stable_trunk," into which Hudson automatically merges changesets
> after all tests have passed, and then have the psuedo-nightly built
> from there.

The implied premise here is that trunk is broken a lot.  But it isn't.
I use trunk all day every day, and if we ignore the MSIL build (which is
indeed broken a lot) it is very rare for trunk to be in a state where
the whole test suite isn't passing.  Some of us get a whole bunch of
angry robot emails whenever the test suite blows up, so it's hard to
miss an extended sequence of failures.

> I think in order for this to work well tests that don't pass would
> need to be left in trunk while they are being worked.  On many
> occasions I've noticed tests being swept to the side until they pass
> again.

On many occasions, really? I can think of a couple times martin has done
this, such as when he was checking in gigantic patches like the new
collections or changes in implicit resolution logic, but I can't think
of "many" times and I don't think it's ever done casually.  When you
have a giant patch against trunk with some tests failing for obscure
reasons you are in a race against the clock because either trunk
development must be halted, your patch is drifting out of sync, or you
disable some tests and fix them later.  I would not be inclined to take
that third choice off the table entirely out of loyalty to the primacy
of the test suite.

That's not to say the test suite doesn't need a lot more attention, as
it definitely does.

--
Paul Phillips      | Adultery is the application of democracy to love.
Moral Alien        |     -- H. L. Mencken
Empiricist         |
pull his pi pal!   |----------* http://www.improving.org/paulp/ *----------

--
http://erikengbrecht.blogspot.com/

On Thu, Aug 13, 2009 at 10:21:26AM -0400, Erik Engbrecht wrote:
> Why must trunk development be halted if there are some tests that
> don't pass?

Oh, I didn't mean to place options 1 and 3 in opposition, but 1 and 2:
if you have a huge patch against trunk which you are not committing,
either trunk development halts or it goes out of sync.

> Communicating the same thing could be accomplished through a commit
> messaging noting that the test doesn't pass, creating a ticket in
> Trac, or a simple email to scala-dev or scala-internals.

I don't disagree in theory, but I think we would need a significantly
less noisy reporting infrastructure. Right now as soon as any test is
failing, it is much harder to spot additional failures, whereas the
difference between "no failures" and "some failures" is clear cut.

> The fact that the test suite almost always indicates that it is not
> broken and that there are a plethora of bugs in Trac seem to indicate
> something is out of balance.

It indicates at least in part that it tests almost exclusively for
specific regressions. This is its greatest deficiency. A lot of the
tickets in trac do have tests in "pending", which it would certainly be
nice to integrate into the overall test strategy. Right now "pending"
might as well be "graveyard", and almost invariably when a bug is fixed
the test in pending doesn't actually switch over to passed, because the
output has to be exactly right or its syntax has bitrotted or etc. etc.

On a couple occasions I have manually dredged pending for tickets which
have been fixed, but this is complicated by the unfortunate ones which
only fail on certain JVMs or fail nondeterministically.

One simple helpful thing we could do is have another nightly report
which notices anytime the output log of a pending test changes at all.

On Thu, Aug 13, 2009 at 4:52 PM, Paul Phillips wrote:
> It indicates at least in part that it tests almost exclusively for
> specific regressions.  This is its greatest deficiency.

If tests needs to be written maybe this is a great way to involve the
community more? I imagine that actual compiler development is rather
complex, but writing tests should be rather easy. A test is also a
very small piece of code, and thus a small effort. From what I
understand of crowd sourcing this is ideal conditions.

So given the right infrastructure I guess a massive amount of tests
can be produced by the community.

BR,
John

On Thu, Aug 13, 2009 at 07:49:33PM +0200, John Nilsson wrote:
> If tests needs to be written maybe this is a great way to involve the
> community more? I imagine that actual compiler development is rather
> complex, but writing tests should be rather easy. A test is also a
> very small piece of code, and thus a small effort. From what I
> understand of crowd sourcing this is ideal conditions.

I'm pretty sure I've made exactly that suggestion at least a couple
times, but the response has been a bit underwhelming (I think we've
received a grand total of zero tests thus far.) What you say is
absolutely true in theory, but it's only "crowdsourcing" if it saves the
sourcer some time - so far it feels more like "recruiting", and
unsuccessful recruiting at that.

On Aug 13, 2009, at 11:16 AM, Paul Phillips wrote:

> On Thu, Aug 13, 2009 at 07:49:33PM +0200, John Nilsson wrote:
>> If tests needs to be written maybe this is a great way to involve the
>> community more? I imagine that actual compiler development is rather
>> complex, but writing tests should be rather easy. A test is also a
>> very small piece of code, and thus a small effort. From what I
>> understand of crowd sourcing this is ideal conditions.
>
> I'm pretty sure I've made exactly that suggestion at least a couple
> times, but the response has been a bit underwhelming (I think we've
> received a grand total of zero tests thus far.) What you say is
> absolutely true in theory, but it's only "crowdsourcing" if it saves
> the
> sourcer some time - so far it feels more like "recruiting", and
> unsuccessful recruiting at that.

I submitted a test which was committed.

And if the overhead in submitting patches were made much easier, like
a Github model where people can fork and have patches in their local
tree which are merged into the main tree, then that would make
contributing that much easier.

Regards,
Blair

On Thu, Aug 13, 2009 at 12:47:07PM -0700, Blair Zajac wrote:
> And if the overhead in submitting patches were made much easier, like
> a Github model where people can fork and have patches in their local
> tree which are merged into the main tree, then that would make
> contributing that much easier.

Not that I'm dying to manage the gateway, but there's nothing stopping
anyone from doing exactly that and sending me pull requests.

http://github.com/paulp/scala/tree/master