Scala Meeting Report for 2011-07-26

On Wed, Jul 27, 2011 at 04:10, Vlad Ureche wrote:
>
> OSCon - Community software
> --------------------------
> Revisit the idea of a community software
>  * blessed, official
>  * not yet in trunk
>  * more than just tools
>
> Make sure
>  * projects don’t overlap (best of breed from each field)

A problem with that would be the mutual exclusion of Specs and
Scalatest, which are two of the brightest stars in the Scala
ecosystem, but sure do overlap. I don't see any one of them "going
away" either.

> A problem with that would be the mutual exclusion of Specs and
> Scalatest, which are two of the brightest stars in the Scala
> ecosystem, but sure do overlap. I don't see any one of them "going
> away" either.
>

And nor should they. Isn't it more useful to be promoting all those
projects that are viable for real world use, rather than just the
"best" in each case?

As Daniel says, both Specs and ScalaTest do similar things, and, as a
user, I think _I_ should get to choose which I prefer to use.

Ta,
Andrew

On 7/27/11 11:32 PM, Andrew wrote:
> As Daniel says, both Specs and ScalaTest do similar things, and, as a
> user, I think _I_ should get to choose which I prefer to use.

Which was the scenario where you lack that choice?

> > As Daniel says, both Specs and ScalaTest do similar things, and, as a
> > user, I think _I_ should get to choose which I prefer to use.
>
> Which was the scenario where you lack that choice?

I mean, if you are "blessing" only the "best of breed from each field"
in terms of community software, then new users may not be aware that
there are other completely viable options available.

On 7/28/11 12:50 AM, Andrew wrote:
> I mean, if you are "blessing" only the "best of breed from each field"
> in terms of community software, then new users may not be aware that
> there are other completely viable options available.

I'm ready to gamble that people who aspire to program in scala can
master google almost as easily.

Applying some editorial would be the point of the exercise. Dropping
every "viable option" on new users borders on maliciousness. Do you
enter into new domains looking forward to being confronted by every
viable option?

Also, there is work associated with such an endeavor which is
proportional to the number of projects. Are you volunteering?

> I'm ready to gamble that people who aspire to program in scala can
> master google almost as easily.
>

Really? I have seen some stackoverflow questions that make me
wonder... :-)

> Applying some editorial would be the point of the exercise.  Dropping
> every "viable option" on new users borders on maliciousness.  Do you
> enter into new domains looking forward to being confronted by every
> viable option?
>

I guess the question here is regarding the definition of viable. I am
not suggesting a list of every _possible_ option, and I am for the
editorial process, but the stated idea was making sure projects "don’t
overlap (best of breed from each field)", which seems very
restrictive.

> Also, there is work associated with such an endeavor which is
> proportional to the number of projects.  Are you volunteering?

Well, I am not sure that "single-user/small user group in an academic
setting" is really they target audience; I haven't used enough of the
tools in a serious enough manner to have opinions on most of them.

On Thu, Jul 28, 2011 at 9:21 AM, Andrew wrote:
> I guess the question here is regarding the definition of viable. I am
> not suggesting a list of every _possible_ option, and I am for the
> editorial process, but the stated idea was making sure projects "don’t
> overlap (best of breed from each field)", which seems very
> restrictive.

I'm not involved with the project, but I would guess that the _aim_ is
to avoid overlapping projects. That's a good thing. I'd imagine that
there may be exceptions for libraries that are widely used like
ScalaTest and Specs. But I could be wrong.

Best,
Ismael

On 28/07/2011 10:34, Ismael Juma wrote:
> On Thu, Jul 28, 2011 at 9:21 AM, Andrew wrote:
>> I guess the question here is regarding the definition of viable. I am
>> not suggesting a list of every _possible_ option, and I am for the
>> editorial process, but the stated idea was making sure projects "don’t
>> overlap (best of breed from each field)", which seems very
>> restrictive.
> I'm not involved with the project, but I would guess that the _aim_ is
> to avoid overlapping projects. That's a good thing. I'd imagine that
> there may be exceptions for libraries that are widely used like
> ScalaTest and Specs. But I could be wrong.
>

And I would add that highlighting a project has never made impossible
for other open source projects to exists and even replace the first one
as "the best". And moreover, I guess that "blessed, official" community
software are not set forever, and if better alternative emerge they will
be endorsed to.

(precision: I'm in no way a Scala team member, nor a TypeSafe one)

On Thu, Jul 28, 2011 at 10:49 AM, Francois <fanf42@gmail.com> wrote:

On 28/07/2011 10:34, Ismael Juma wrote:

On Thu, Jul 28, 2011 at 9:21 AM, Andrew<andrew.webb@gmail.com>  wrote:

I guess the question here is regarding the definition of viable. I am
not suggesting a list of every _possible_ option, and I am for the
editorial process, but the stated idea was making sure projects "don’t
overlap (best of breed from each field)", which seems very
restrictive.

I'm not involved with the project, but I would guess that the _aim_ is
to avoid overlapping projects. That's a good thing. I'd imagine that
there may be exceptions for libraries that are widely used like
ScalaTest and Specs. But I could be wrong.

And I would add that highlighting a project has never made impossible for other open source projects to exists and even replace the first one as "the best". And moreover, I guess that "blessed, official" community software are not set forever, and if better alternative emerge they will be endorsed to.

(precision: I'm in no way a Scala team member, nor a TypeSafe one)

Oh, it's "Typesafe" <-- small s :-)
 

The idea here is to avoid overlapping features from multiple libraries.  In real life, there are always exceptions but expect these to be exceptions, not the rule.  The final idea of this is to have a downloadable complete Scala community 'platform' that fills in the gaps of the standard library. 
Something akin to the scala-incubator idea.
- Josh

On 28 July 2011 12:28, Josh Suereth <joshua.suereth@gmail.com> wrote:

The idea here is to avoid overlapping features from multiple libraries.  In real life, there are always exceptions but expect these to be exceptions, not the rule.  The final idea of this is to have a downloadable complete Scala community 'platform' that fills in the gaps of the standard library. 

Premature blessing of these libraries is probably something we want to be at pains to avoid. If we take json (de)serialization support as an example, there is code in the core lib that tackles this which, as far as I can see, nobody uses and is broken. Then there is json support provided by several large projects (e.g. lift, akka) that is widely used and works. I'm not picking on json, but I would expect that for much of the shopping list of features that a developer may look for, that it is provided as a side-product of several 3rd party projects, and with varying levels of lock-in to those projects.
There are also cases where there legitimately are different ways to do things which will suit different developers better. Testing, web frameworks, remoting, ... one size does not fit all. However, perhaps there would be no excuse for having multiple DSLs for integrating Scala with spring and OSGi. Like json support, these sorts of things really should be standardised if and when consensus emerges about what is needed.
Not sure I've added any clarity to this discussion. Premature blessing and adoption is harmful. Making potential users aware of 'best of breed' solutions and libraries is very good.
Matthew 

Something akin to the scala-incubator idea.
- Josh

--
Dr Matthew PocockVisitor, School of Computing Science, Newcastle Universitymailto: turingatemyhamster@gmail.com gchat: turingatemyhamster@gmail.commsn: matthew_pocock@yahoo.co.uk irc.freenode.net: drdozertel: (0191) 2566550mob: +447535664143

While I agree with what you're saying, i think it may be too early to make assumptions on what libraries will be in the extension.  There are many many reasons for this extension, and there are some potential pitfalls we will try to avoid.   As you suggest, adopting a library too early could cause issues.  However, the beauty of the community extension libraries is that it will be, by definition, more volatile than the standard library.   We have the option of dropping/deprecating a library in favor of  a better library when the time comes.
The amount of volatility in the community 'platform' is still up for debate, but it is known that it will be easier to make changes in the community libraries than in the standard library.
I agree with you that the JSON support in alternative frameworks is probably more efficient and better than the standard.  I also agree with you about lock-in on various software stacks.  
As for the choosing of libraries, Specs and Scalatest offer a tough decision.  I have no idea what the result will be there.  However, we could probably all agree that scalacheck is *unique* and best of breed for property testing.  The fact that Spec, Scalatest *and* Scala itself use it is a sign that it's primed and ready to be in the community library.
Since we haven't decided on the decision process itself for how libraries are included, now is the best time to weigh in with ideas.  
Finally, in my rambling reply, having a library be 'blessed' into the community extensions *may* just push enough community momentum behind it that it becomes significantly better than before it was chosen.   This is a potential counter-example to your argument.
So basically, while I think you make good points, I don't think they outweigh the benefit of having a community platform.   We just have to be very cautious about what we include and retain the ability to recover from poor choices.
On Thu, Jul 28, 2011 at 8:07 AM, Matthew Pocock <turingatemyhamster@gmail.com> wrote:

On 28 July 2011 12:28, Josh Suereth <joshua.suereth@gmail.com> wrote:

The idea here is to avoid overlapping features from multiple libraries.  In real life, there are always exceptions but expect these to be exceptions, not the rule.  The final idea of this is to have a downloadable complete Scala community 'platform' that fills in the gaps of the standard library. 

Premature blessing of these libraries is probably something we want to be at pains to avoid. If we take json (de)serialization support as an example, there is code in the core lib that tackles this which, as far as I can see, nobody uses and is broken. Then there is json support provided by several large projects (e.g. lift, akka) that is widely used and works. I'm not picking on json, but I would expect that for much of the shopping list of features that a developer may look for, that it is provided as a side-product of several 3rd party projects, and with varying levels of lock-in to those projects.
There are also cases where there legitimately are different ways to do things which will suit different developers better. Testing, web frameworks, remoting, ... one size does not fit all. However, perhaps there would be no excuse for having multiple DSLs for integrating Scala with spring and OSGi. Like json support, these sorts of things really should be standardised if and when consensus emerges about what is needed.
Not sure I've added any clarity to this discussion. Premature blessing and adoption is harmful. Making potential users aware of 'best of breed' solutions and libraries is very good.
Matthew 

Something akin to the scala-incubator idea.
- Josh

--
Dr Matthew PocockVisitor, School of Computing Science, Newcastle Universitymailto: turingatemyhamster@gmail.com gchat: turingatemyhamster@gmail.commsn: matthew_pocock@yahoo.co.uk irc.freenode.net: drdozertel: (0191) 2566550mob: +447535664143

On 28 July 2011 14:04, Josh Suereth <joshua.suereth@gmail.com> wrote:

So basically, while I think you make good points, I don't think they outweigh the benefit of having a community platform.   We just have to be very cautious about what we include and retain the ability to recover from poor choices.

My points where intended as a note of caution, not an indication that I'd like to see the enterprise blocked. I happen to think that some form of community platform for scala will be a very good thing, especially once the core scala libs are modularised and as long as things can change status as-and-when it makes sense. Community platform both fosters and requires community involvement for it to be successful. Java effectively has this with the commons libs and the horribly messy java.net/JSR process. We can probably do better :D
Matthew --
Dr Matthew PocockVisitor, School of Computing Science, Newcastle Universitymailto: turingatemyhamster@gmail.com gchat: turingatemyhamster@gmail.commsn: matthew_pocock@yahoo.co.uk irc.freenode.net: drdozertel: (0191) 2566550mob: +447535664143

That brings up a great point.   JSR was... interesting.   What can we do better here?
I'd love to have ideas for how to ensure the process of selecting and retaining libraries in the Scala community platform doesn't fall into the same state as the JSRs.   Don't want to throw out the baby with the bathwater though.
- Josh

On Thu, Jul 28, 2011 at 9:53 AM, Matthew Pocock <turingatemyhamster@gmail.com> wrote:

On 28 July 2011 14:04, Josh Suereth <joshua.suereth@gmail.com> wrote:

So basically, while I think you make good points, I don't think they outweigh the benefit of having a community platform.   We just have to be very cautious about what we include and retain the ability to recover from poor choices.

My points where intended as a note of caution, not an indication that I'd like to see the enterprise blocked. I happen to think that some form of community platform for scala will be a very good thing, especially once the core scala libs are modularised and as long as things can change status as-and-when it makes sense. Community platform both fosters and requires community involvement for it to be successful. Java effectively has this with the commons libs and the horribly messy java.net/JSR process. We can probably do better :D
Matthew --
Dr Matthew PocockVisitor, School of Computing Science, Newcastle University mailto: turingatemyhamster@gmail.com gchat: turingatemyhamster@gmail.commsn: matthew_pocock@yahoo.co.uk irc.freenode.net: drdozertel: (0191) 2566550mob: +447535664143

I hope what I am interpreting this conversation as has not been taken out of context or diluted of its original intent.
What I am hearing is that there exists a handful of individuals here trying to "enrich" the community by providing a limited view of what's available. 
I think this is counter intuitive in a number of ways.
First, "best of breed" for a given domain problem sounds very subjective. Especially when selected by a few. To me, this sounds much less like a community and more like a corporation's way of solving a problem. "Hi, I'm ..., and I endorsed this library".
This narrow view for someone coming into the language can have negative affect if they determine the blessed libraries to be unsuitable for their needs.  It should be a goal to get more people to come into the language and be satisfied with solutions. For them, having one blessed option that is unsuitable for them may beg the question "Is this the best scala has to offer"? 
Best of breed also implies there is only one best way for solving a problem.  Libraries solve lots of problems in lots of different ways based on use cases, even when they do overlap in features. In some cases, this is intentional because people come up with new ways of solving problems when existing ones fall short. I take the example of people who work on libraries for writing web apps. I am sure most people who do this in the scala community have heard of Lift before and I am happy to see that just because there was an established and maintained web library already out there that had features they needed, it didn't stop them from trying to solve the problem in different ways.
Potential developers who see there is already a "blessed" library for the problem they want to solve may actually discourage them from trying to do something new, because the potential that the visibility of their library will have a diminished visibility for new users of the language. It's bad for a community to not have this influx of fresh ideas.
While I don't consider WalMart to be in anyway best of breed (and probably a bad example here :)), it does come to mind when I think about the impact of what is visible for someone from out of town coming into town and that being the first thing they see. Mom and pop shops often go out of business because of this and in the same manner, small well-written but less visible libraries die due to poor visibility in a community. 
I again apologize if I misinterpreted this thread in any way but I feel _very strongly_ about the need for a strong Scala community which fosters growth and new ideas. Dedicating an effort to create a narrow view of what's out there really feels like its going against the tide of what we should be doing.
-Doug Tangren
http://lessis.me

On Thu, Jul 28, 2011 at 10:12:40AM -0400, Josh Suereth said
> That brings up a great point. JSR was... interesting. What can we do
> better here?
>
> I'd love to have ideas for how to ensure the process of selecting and
> retaining libraries in the Scala community platform doesn't fall into the
> same state as the JSRs. Don't want to throw out the baby with the
> bathwater though.

Maybe we could take some inspiration from the Haskell Platform process:
http://trac.haskell.org/haskell-platform/wiki/AddingPackages

It's nice that they've provided rationale for much of the process and
policy. For example the rationale for the policy on packages duplicating
functionality is:

To avoid user confusion and reduce the maintenance burden, we wish
to avoid having several packages provide the same or significantly
overlapping functionality. It is fine to provide packages that solve
the same problem but provide significantly different or
complementary approaches.

I guess I should point out that I have no idea how well the participants
feel the process works but the end result (the platform packages) seem
pretty good in my eyes.

This is more the style I think we're all looking for.   Essentially, a set of libraries that you can download with the Scala distribution to ease entry into scala for new comers.   The balance to get right is:

• Ability of the libraries themselves to change
• Ability to swap in/out libraries
• Reduction of complexity for newcomers to Scala (i.e. *every* option is too much when starting).  I think this is the big thing here.   I think we could do much worse than Haskell here).  I also think we echo their ideas on packages here.
• Ability to keep on top of Scala releases and enter into the development cycle (i.e. the compiler "can't" break these libraries for a release and in turn these libraries have to play nicely with the build process required to ease testing against Scala nightlies).

If this library causes any kind of community rift, decrease of innovation or other Bad Thing(tm), then we've failed and the whole thing should be scrapped.
Also note that this project would not negate having an indexed project listing website akin to CPAN or Cabal.   I would love to see scala-tools.org become better at this, but it's on the bottom of a long list of TODOs.

- Josh
On Thu, Jul 28, 2011 at 11:09 AM, Geoff Reedy <geoff@programmer-monk.net> wrote:

On Thu, Jul 28, 2011 at 10:12:40AM -0400, Josh Suereth said
> That brings up a great point.   JSR was... interesting.   What can we do
> better here?
>
> I'd love to have ideas for how to ensure the process of selecting and
> retaining libraries in the Scala community platform doesn't fall into the
> same state as the JSRs.   Don't want to throw out the baby with the
> bathwater though.

Maybe we could take some inspiration from the Haskell Platform process:
http://trac.haskell.org/haskell-platform/wiki/AddingPackages

It's nice that they've provided rationale for much of the process and
policy. For example the rationale for the policy on packages duplicating
functionality is:

   To avoid user confusion and reduce the maintenance burden, we wish
   to avoid having several packages provide the same or significantly
   overlapping functionality. It is fine to provide packages that solve
   the same problem but provide significantly different or
   complementary approaches.

I guess I should point out that I have no idea how well the participants
feel the process works but the end result (the platform packages) seem
pretty good in my eyes.

I wonder if it would make sense to tackle that goal from a slightly different perspective.

What about helping the community form "interest groups", and actively encouraging people to contribute in their special area of interest?

Maybe a first start would be looking at the "scala team maintainer groups" http://www.scala-lang.org/node/ and provide a separate table for "community interest groups", which points to people interested in working on it, existing solutions, places to discuss, etc.?

This approach would look like a bit more "inclusive" imho.

Just some thoughts ...

AH, I must apologize if this has the impression of not being inclusive.  It is not meant to be so.
Again, input on the best possible process is appreciated.  This collection of libraries should represent the community and serve the community as best as possible.   In fact, there's no way they could exist without community involvement.  We're listening to what you're saying, so please rather than assuming there's some grand master evil plan here, PLEASE PLEASE make suggestions on the kinds of process you would like to see behind such a library.
Now, onto the idea to have interest groups.  I really like this idea.   So for a core area, we could start a maintainer group around that and they can be responsible for selecting/improving libraries, akin to the Maintainers idea, except in the community library instead of others.
One thing I'll say, and feel free to disagree, is that I think having a single point of failure for decisions is ideal.  So while everything should be democratic to an extent, but we need an executive with Veto privileges.   In that vein, I think appointing a "czar" for each area that can rule in tied votes and veto things that don't have enough majority could help as well.
Now onto the hard questions with such a process:

• How do you appoint members to the interest groups?
• How do you select a czar?
• How do we prevent this from becoming as slow and frustrating as the JSR process?
• How do we prevent popularity contests for technical merit?

Again, no plans are final, everything is being thought through.   So now is the time to give us input to prevent failure.
- Josh
On Thu, Jul 28, 2011 at 12:23 PM, Simon Ochsenreither <simon.ochsenreither@googlemail.com> wrote:

I wonder if it would make sense to tackle that goal from a slightly different perspective.

What about helping the community form "interest groups", and actively encouraging people to contribute in their special area of interest?

Maybe a first start would be looking at the "scala team maintainer groups" http://www.scala-lang.org/node/ and provide a separate table for "community interest groups", which points to people interested in working on it, existing solutions, places to discuss, etc.?

This approach would look like a bit more "inclusive" imho.

Just some thoughts ...

> One thing I'll say, and feel free to disagree, is that I think having a
> single point of failure for decisions is ideal. So while everything should
> be democratic to an extent, but we need an executive with Veto privileges.
> In that vein, I think appointing a "czar" for each area that can rule in
> tied votes and veto things that don't have enough majority could help as
> well.

I think this only matters if the "interest group" wants to get some
package into the standard distribution.

Currently the opposite thing seems to be a desired goal: people would
mostly prefer having a more lightweight standard library. I too would
love to have separate jar files for the parser combinators, for XML,
etc. (of course without breaking compatibility, but having smaller
JARs would make developing with andoird much easier, e. g. no need to
run pro-guard during development, only for release!)

If that will ever work, "integrating some community project" isn't
much more than changing the package name to "scala.xyz" and adding the
jar file to the distribution.

As a concrete example, I could pretty much think of an XML interest
group (around the EPFL-led scala.xml AND the community-led anti-xml),
an interest group around tetsing, with Specs, ScalaTest, etc., an
interest group around the website and the documentation, and various
interest groups "waiting for incubation" like "scala-logging",
"scala-datetime", "scala-units&measurements", ...

Imho it is just a bit too early to think about these "heavy"
governance structures, let's just see if and how much people are even
interested in these ideas ...

Just some thoughts.

Thanks and bye,

Simon

BTW: I was thinking if having a "Scala works with XYZ" page would help
beginners.

Often things work seamless, but people like to now if someone has
already tested it before them.

What about having some sort of list of "Java" things like Struts, JSF,
AspectJ, JodaTime, Apache Commons, Guice, Spring, etc. which give
people some hints how well certain technology interoperates with Scala?

On Thu, Jul 28, 2011 at 12:52 PM, Simon Ochsenreither <simon@ochsenreither.de> wrote:

One thing I'll say, and feel free to disagree, is that I think having a
single point of failure for decisions is ideal.  So while everything should
be democratic to an extent, but we need an executive with Veto privileges.
In that vein, I think appointing a "czar" for each area that can rule in
tied votes and veto things that don't have enough majority could help as
well.

I think this only matters if the "interest group" wants to get some package into the standard distribution.

Currently the opposite thing seems to be a desired goal: people would mostly prefer having a more lightweight standard library. I too would love to have separate jar files for the parser combinators, for XML, etc. (of course without breaking compatibility, but having smaller JARs would make developing with andoird much easier, e. g. no need to run pro-guard during development, only for release!)

Funny you mention that, that's been one of my long term personal goals.  We'll see what the cards have in store. 

If that will ever work, "integrating some community project" isn't much more than changing the package name to "scala.xyz" and adding the jar file to the distribution.

True.  In fact, that's what this Community Extension project is all about.   This is more a matter of projects agree to abide by some build tool restrictions so   (1) All libraries can be built in aggregate against Scala version XYZ   (2) All libraries can have common releases that are guaranteed to work with each oither. 

As a concrete example, I could pretty much think of an XML interest group (around the EPFL-led scala.xml AND the community-led anti-xml), an interest group around tetsing, with Specs, ScalaTest, etc., an interest group around the website and the documentation, and various interest groups "waiting for incubation" like "scala-logging", "scala-datetime", "scala-units&measurements", ...

All good ideas.   I'd also add an IO group and perhaps a "Scripting library" group. 

Imho it is just a bit too early to think about these "heavy" governance structures, let's just see if and how much people are even interested in these ideas ...

 Agreed that heavy governance is bad.  However chaos is also unproductive.  I personally want to see something happen, in the very least, to help ensure testing of scala is easier and to avoid breaking 'core' scala projects.  I'm not sure what the middle ground is here, but that's what I personally want us to acheive.

It would perhaps be beneficial to the whole process if there would be
some goal statement up front. The idea of "something like this" has
been around for a while and there were probably a lot of face-to-face
discussions around this topic at Scala Days and other occasions so
many of you might already have quite an understanding where this
initiative is targetting at. Others, like me, couldn't be there so I
only have a vague intuition where this is going to.

I see at least these topics discussed here:
* motivating and organizing community involvement around
specification and maintance of common Scala libraries
* building a listing of available Scala libraries to ease the
discovery of Scala components. (As a user of this I would wish to
discover libraries by topic, but also by popularity to get up to speed
what is current in the Scala community; then some indicator of
freshness of a library and where to go for more information, and
perhaps some documentation/Scaladoc hub)
* putting together a bundle of blessed libraries for exactly what
purpose? How exactly is this going to help new users? (I could imagine
a big download where all the libraries are ready-to-go integrated in
some template Eclipse project or similar, together with some
integrated downloadable documentation document/tutorial showing how to
use them together)
* putting important libraries into the continuous integration pipeline

Those topics are interconnected but I think it makes sense to keep the
goals for all those difference aspects clear in mind before starting
some initiative.

I'm really looking forward to see those things happen!

Johannes

On Thu, Jul 28, 2011 at 6:35 PM, Josh Suereth wrote:
> AH, I must apologize if this has the impression of not being inclusive.  It
> is not meant to be so.
> Again, input on the best possible process is appreciated.  This collection
> of libraries should represent the community and serve the community as best
> as possible.   In fact, there's no way they could exist without community
> involvement.  We're listening to what you're saying, so please rather than
> assuming there's some grand master evil plan here, PLEASE PLEASE make
> suggestions on the kinds of process you would like to see behind such a
> library.
> Now, onto the idea to have interest groups.  I really like this idea.   So
> for a core area, we could start a maintainer group around that and they can
> be responsible for selecting/improving libraries, akin to the Maintainers
> idea, except in the community library instead of others.
> One thing I'll say, and feel free to disagree, is that I think having a
> single point of failure for decisions is ideal.  So while everything should
> be democratic to an extent, but we need an executive with Veto privileges.
> In that vein, I think appointing a "czar" for each area that can rule in
> tied votes and veto things that don't have enough majority could help as
> well.
> Now onto the hard questions with such a process:
>
> How do you appoint members to the interest groups?
> How do you select a czar?
> How do we prevent this from becoming as slow and frustrating as the JSR
> process?
> How do we prevent popularity contests for technical merit?
>
> Again, no plans are final, everything is being thought through.   So now is
> the time to give us input to prevent failure.
> - Josh
> On Thu, Jul 28, 2011 at 12:23 PM, Simon Ochsenreither
> wrote:
>>
>> I wonder if it would make sense to tackle that goal from a slightly
>> different perspective.
>>
>> What about helping the community form "interest groups", and actively
>> encouraging people to contribute in their special area of interest?
>>
>> Maybe a first start would be looking at the "scala team maintainer groups"
>> http://www.scala-lang.org/node/ and provide a separate table for "community
>> interest groups", which points to people interested in working on it,
>> existing solutions, places to discuss, etc.?
>>
>> This approach would look like a bit more "inclusive" imho.
>>
>> Just some thoughts ...
>
>

I think you summed up the goals in a good, nice and tidy summary.   I believe the only openly stated goal was something akin to Haskell's community libraries but not as hard to find good libraries as cabal.   From there, it's been this discussion :)
On Thu, Jul 28, 2011 at 1:14 PM, Johannes Rudolph <johannes.rudolph@googlemail.com> wrote:

It would perhaps be beneficial to the whole process if there would be
some goal statement up front. The idea of "something like this" has
been around for a while and there were probably a lot of face-to-face
discussions around this topic at Scala Days and other occasions so
many of you might already have quite an understanding where this
initiative is targetting at. Others, like me, couldn't be there so I
only have a vague intuition where this is going to.

I see at least these topics discussed here:
 * motivating and organizing community involvement around
specification and maintance of common Scala libraries
 * building a listing of available Scala libraries to ease the
discovery of Scala components. (As a user of this I would wish to
discover libraries by topic, but also by popularity to get up to speed
what is current in the Scala community; then some indicator of
freshness of a library and where to go for more information, and
perhaps some documentation/Scaladoc hub)
 * putting together a bundle of blessed libraries for exactly what
purpose? How exactly is this going to help new users? (I could imagine
a big download where all the libraries are ready-to-go integrated in
some template Eclipse project or similar, together with some
integrated downloadable documentation document/tutorial showing how to
use them together)
 * putting important libraries into the continuous integration pipeline

Just a note that the two goals above this are tied.  Bundling all the libraries into a common "build" so it can be incorporated into the continuous integration pipeline is the biggest reason a library should want to be in the bundle.   From a Scala QA perspective, this is like automatically testing the compiler against a ton of real-life code.  
 

Those topics are interconnected but I think it makes sense to keep the
goals for all those difference aspects clear in mind before starting
some initiative.

I'm really looking forward to see those things happen!

Johannes

On Thu, Jul 28, 2011 at 6:35 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> AH, I must apologize if this has the impression of not being inclusive.  It
> is not meant to be so.
> Again, input on the best possible process is appreciated.  This collection
> of libraries should represent the community and serve the community as best
> as possible.   In fact, there's no way they could exist without community
> involvement.  We're listening to what you're saying, so please rather than
> assuming there's some grand master evil plan here, PLEASE PLEASE make
> suggestions on the kinds of process you would like to see behind such a
> library.
> Now, onto the idea to have interest groups.  I really like this idea.   So
> for a core area, we could start a maintainer group around that and they can
> be responsible for selecting/improving libraries, akin to the Maintainers
> idea, except in the community library instead of others.
> One thing I'll say, and feel free to disagree, is that I think having a
> single point of failure for decisions is ideal.  So while everything should
> be democratic to an extent, but we need an executive with Veto privileges.
> In that vein, I think appointing a "czar" for each area that can rule in
> tied votes and veto things that don't have enough majority could help as
> well.
> Now onto the hard questions with such a process:
>
> How do you appoint members to the interest groups?
> How do you select a czar?
> How do we prevent this from becoming as slow and frustrating as the JSR
> process?
> How do we prevent popularity contests for technical merit?
>
> Again, no plans are final, everything is being thought through.   So now is
> the time to give us input to prevent failure.
> - Josh
> On Thu, Jul 28, 2011 at 12:23 PM, Simon Ochsenreither
> <simon.ochsenreither@googlemail.com> wrote:
>>
>> I wonder if it would make sense to tackle that goal from a slightly
>> different perspective.
>>
>> What about helping the community form "interest groups", and actively
>> encouraging people to contribute in their special area of interest?
>>
>> Maybe a first start would be looking at the "scala team maintainer groups"
>> http://www.scala-lang.org/node/ and provide a separate table for "community
>> interest groups", which points to people interested in working on it,
>> existing solutions, places to discuss, etc.?
>>
>> This approach would look like a bit more "inclusive" imho.
>>
>> Just some thoughts ...
>
>



--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

Josh,
Allow me to think out loud about this for a while.
The idea of the common build infrastructure is appealing, so the language and the libraries can mutually QA each other.  But actually making this work presents would require jumping many logistical hurdles and coordinating a lot of disparate efforts.  It has to be worth the effort.
It seems to me the existing situation on this isn't terrible.  The Scala team publishes nightlies, and library authors and library users hopefully notice if a new nightly breaks their library.
The problem we've had repeatedly in the past is that library authors and users don't necessarily test every nightly, and then all of a sudden the core Scala team drops a release candidate on us, often with little or no warning, and then the library authors suddenly have to scramble to fix their libraries.
In the 2.9 series we didn't even have any betas before the release candidates started; I think we all know how well that worked out, with the parallel collection hierarchy needing to be redone after we were already in the release candidate phase.
I think these issues are mostly (though not perfectly) solvable with better communication.  If the Scala team would announce, at least a few days (or a week?)  ahead of time, "Hey, we're preparing a beta or a release candidate, these changes were backported fixing these tickets, please test your libraries with the latest nightly," most of these sorts of problems could be averted.  A library might still get broken by a last minute change, but hey, that's why release candidates are candidates and not releases.
Another stated motivation for the blessed libraries is "reduction of complexity for newcomers" part.  It isn't clear to me how this will help.  Newcomers are going to be using IntelliJ, or Eclipse, or NetBeans, or the text editor of their choice.  Plus they'll probably be using sbt or maven.  Even if Typesafe provided some "platform" mega-download that dumped a bunch of libraries onto this newcomer's hard drive, that wouldn't help them actually get the libraries going in their particular development environment.  Helping them do that is a documentation problem.
Derek Chen-Becker's jibe elsewhere about sbaz 2.0 is actually apropos. Scala already tried "hey users, here's a thing that downloads blessed packages for you".  It failed.  Why did it fail?  Because the standard way JVM developers get libraries today is using sbt or maven, and for very good reason: so the particular versions used are recorded as part of a repeatable build process.
I do think it would help newcomers to have a place they could go to find out about best-of-breed libraries.  So, what's the simplest thing that could possibly work for that?  A web page might be enough.  It could list the blessed libraries by category, and for each library, it could give you the right XML nugget to use in your Maven config and the right Scala nugget to use in your sbt config, suitable for copying and pasting.  And a direct download link too.
If we want this web page to be a little fancier there could be menus where you select your Scala version, sbt version, etc and it gives you the appropriate dependency or link.  Even something like this would be a lot to easier to build than a platform bundle or sbaz 2.0 or what have you.
The whole concept of bundling seems problematic to me.  Library authors are constantly fixing bugs and releasing minor updates. Suppose I need bugfix X to library Y, how do I get it if I'm getting my libraries from a bundle?  Rather than steering people away from them, I think we want to encourage people to use sbt or maven so that if they need to upgrade a library, it's as simple as bumping a version number in that dependency.
Finally, about politics.  For Typesafe to officially bless libraries could get very political very fast, and the decision making on this will use up some of the core team's bandwidth.  It has to be worth it.
I think a newcomer to the language might certainly be interested in the answer to the question, "What libraries are officially esteemed and blessed by the creators of the language?"  But that isn't the actual question in a newcomer's mind.  The actual question they have is, "What library does everybody use to do X?"  This question could be answered by polling the users a few times a year and posting the results.
As a bonus, the poll could include questions like "What version of Scala are you using?" and "What do you edit your code in?" and "What build tool do you use?"  Newcomers want to know what people are using in these categories, too, not only what libraries are popular.
One final point: I agree with the people who said they're more interested in seeing the Scala library dismantled into modules, than in seeing libraries blessed and bundled.
Anyway, everything I've said is certainly debatable...!  I think this has been a productive discussion so far and I'm looking forward to more of it.
-- Seth Tisue | Northwestern University | http://tisue.netlead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

On 29 July 2011 15:31, Seth Tisue <sethtisue@gmail.com> wrote:

Josh,
Allow me to think out loud about this for a while.
The idea of the common build infrastructure is appealing, so the language and the libraries can mutually QA each other.  But actually making this work presents would require jumping many logistical hurdles and coordinating a lot of disparate efforts.  It has to be worth the effort.
It seems to me the existing situation on this isn't terrible.  The Scala team publishes nightlies, and library authors and library users hopefully notice if a new nightly breaks their library.

I have some compiler bugs that I'd like to fix and they are really tricky. I'd like to have a luxury of throwing as much of Scala code on my patched compiler as possible and check *myself* the results. Library authors might not notice that particular problem for weeks because you need to have very specific setup of e.g. Eclipse to trigger Eclipse's crash.
--
Grzegorz Kossakowski

On Friday, July 29, 2011 9:42:01 AM UTC-4, Grzegorz Kossakowski wrote:

I have some compiler bugs that I'd like to fix and they are really tricky. I'd like to have a luxury of throwing as much of Scala code on my patched compiler as possible and check *myself* the results. Library authors might not notice that particular problem for weeks because you need to have very specific setup of e.g. Eclipse to trigger Eclipse's crash.

Do you know about Yuvi Masory's "Scala corpus"? https://github.com/alacscala/scala-corpus

-- Seth Tisue | Northwestern University | http://tisue.netlead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

On 7/29/11 6:31 AM, Seth Tisue wrote:
> It seems to me the existing situation on this isn't terrible. The Scala
> team publishes nightlies, and library authors and library users
> hopefully notice if a new nightly breaks their library.

Not to disagree too much, but I think it's terrible. (Before anyone
goes tweeting half-cocked, if I say something is terrible it means
"relative to what is possible with a realistic amount of effort", not in
absolute terms.) We incur an unacceptable number of regressions which
could be caught with continuous integration of other software. Even if
we did catch all these during the development cycle (which as you know,
we don't) it is far more difficult to pinpoint and repair regressions
weeks or months down the line than it would be if the canary died the
same day.

Humans should be removed from the loop whenever possible. Any situation
which can be characterized as "library authors and library users
hopefully notice" something is highly suboptimal unless the noticing
we're hoping they're doing inherently requires the application of
human-level intelligence. Which in this case, as in most others, it
does not.

The surface area of the distribution is too large to rely completely on
tests even if our test suite was stellar. The computing resources and
ongoing effort necessary to perform continuous integration with a
selection of projects are miniscule compared to the benefits provided,
even if the only benefit was extempore's saved time.

So this reason alone is plenty of justification for regular builds of
some subset of projects. From there it's only sensible to pick useful
projects and publish the artifacts which are being built anyway.

Don't smother too many babies in the crib. It's hard to predict what
will work out and what won't. If in doubt, usually the best way to find
out is to try it.

On 29 July 2011 16:39, Seth Tisue <sethtisue@gmail.com> wrote:

On Friday, July 29, 2011 9:42:01 AM UTC-4, Grzegorz Kossakowski wrote:

I have some compiler bugs that I'd like to fix and they are really tricky. I'd like to have a luxury of throwing as much of Scala code on my patched compiler as possible and check *myself* the results. Library authors might not notice that particular problem for weeks because you need to have very specific setup of e.g. Eclipse to trigger Eclipse's crash.

Do you know about Yuvi Masory's "Scala corpus"? https://github.com/alacscala/scala-corpus

How to build all of them with 5 minutes of my time, and plenty of CPU time? That's the whole point I and Paul are trying to convey.
--
Grzegorz Kossakowski

On Fri, Jul 29, 2011 at 4:41 PM, Paul Phillips wrote:
> Don't smother too many babies in the crib.  It's hard to predict what will
> work out and what won't.  If in doubt, usually the best way to find out is
> to try it.

That's why it would be best to separate technical and possibly
political things.

Integrating some key projects into the build chain for trunk builds
can start by just asking some project owners to work together to get
that done. That shouldn't be too hard. While you are at it you can set
up a process of how other projects can apply for being integrated into
the core build chain and what the preconditions are. This can all be
done in the background without anyone starting to ask questions.

> From there it's only sensible to pick useful projects
> and publish the artifacts which are being built anyway.

You mean automatically for new (pre-)releases? That would certainly be useful.

That said, I don't see how that brings the "bundling" into play which
seemed to drive the discussion this far.

Think of this more as the following things:
(1) Scala Fresh Take 2:  Where I actually get something done.   This is exactly what Paul is stated.   Regardless of any other point, this project will be built for no other purpose than assisting the QA of Scala.   I'm really hoping prominent projects in the community will assist in this effort, but the point is that Scala needs more tests :)
(2) Think of this new community library set as "what the standard library should be".  A bunch of loosely affiliated (modular) libraries covering many aspects of development.  By keeping them as separate libraries, they can make incremental bug fixes *much* quicker than the entire library and by releasing them with the standard library you ensure coverage of many areas with a single scala release.   I think this could make all our lives a bit easier.   Note that this does *not* entail a new package manager.   The libraries would be deployed with maven both individually and with an aggregate artifact that could be depended on.  So, yes this could simplify things for users.  Don't know what you want? Just pull all the JARS for now and toy around.
That said, until the specifics of how libraries are chosen and how the entire project is managed, we'll be actively pursuing #1 and trying to create a #2 everyone is happy with.

On Fri, Jul 29, 2011 at 10:41 AM, Paul Phillips <paulp@improving.org> wrote:

On 7/29/11 6:31 AM, Seth Tisue wrote:

It seems to me the existing situation on this isn't terrible. The Scala
team publishes nightlies, and library authors and library users
hopefully notice if a new nightly breaks their library.

Not to disagree too much, but I think it's terrible.  (Before anyone goes tweeting half-cocked, if I say something is terrible it means "relative to what is possible with a realistic amount of effort", not in absolute terms.) We incur an unacceptable number of regressions which could be caught with continuous integration of other software.  Even if we did catch all these during the development cycle (which as you know, we don't) it is far more difficult to pinpoint and repair regressions weeks or months down the line than it would be if the canary died the same day.

Humans should be removed from the loop whenever possible.  Any situation which can be characterized as "library authors and library users hopefully notice" something is highly suboptimal unless the noticing we're hoping they're doing inherently requires the application of human-level intelligence.  Which in this case, as in most others, it does not.

The surface area of the distribution is too large to rely completely on tests even if our test suite was stellar.  The computing resources and ongoing effort necessary to perform continuous integration with a selection of projects are miniscule compared to the benefits provided, even if the only benefit was extempore's saved time.

So this reason alone is plenty of justification for regular builds of some subset of projects.  From there it's only sensible to pick useful projects and publish the artifacts which are being built anyway.

Don't smother too many babies in the crib.  It's hard to predict what will work out and what won't.  If in doubt, usually the best way to find out is to try it.

On Fri, Jul 29, 2011 at 10:41 AM, Paul Phillips wrote:
> Don't smother too many babies in the crib.

Seth smothers babies, you say? Elsewhere in your email you state that
"Humans should be removed from the loop whenever possible", so I think
you and I are actually on the same team there.

> The computing resources and
> ongoing effort necessary to perform continuous integration with a selection
> of projects are miniscule compared to the benefits provided, even if the
> only benefit was extempore's saved time.

Strong words! You seem sure, so, I'm inclined to believe it, then.

On Fri, Jul 29, 2011 at 11:00 AM, Johannes Rudolph
wrote:
> That's why it would be best to separate technical and possibly
> political things
> [...]
> This can all be
> done in the background without anyone starting to ask questions.

+1

The infrastructure discussion seems pretty much separate from the
discussion about officially blessing and bundling stuff for the
public. No need to hold up the former because of uncertainty over the
latter.

On Fri, Jul 29, 2011 at 5:00 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

Think of this more as the following things:
(1) Scala Fresh Take 2:  Where I actually get something done.   This is exactly what Paul is stated.   Regardless of any other point, this project will be built for no other purpose than assisting the QA of Scala.   I'm really hoping prominent projects in the community will assist in this effort, but the point is that Scala needs more tests :)

I agree.
 

(2) Think of this new community library set as "what the standard library should be".  A bunch of loosely affiliated (modular) libraries covering many aspects of development.  By keeping them as separate libraries, they can make incremental bug fixes *much* quicker than the entire library and by releasing them with the standard library you ensure coverage of many areas with a single scala release.   I think this could make all our lives a bit easier.   Note that this does *not* entail a new package manager.   The libraries would be deployed with maven both individually and with an aggregate artifact that could be depended on.  So, yes this could simplify things for users.  Don't know what you want? Just pull all the JARS for now and toy around.

For me (2) is also a proving ground. We do get quite a few contributions. Sometimes we have accepted them, and regretted it afterwards. At other times, we have been too conservative out of fear or exhaustion to accept worthwhile packages. The problem is that, if something is in trunk, it has to work well, and the Scala team has to support it. But the core team has neither the capacity nor the expertise to test and support many outside contributions.

That's why a community library is attractive. It's a half way status where
packages are prominent enough that hopefully many people will use them, and at the same time not critical enough to be cast in stone. It gives us tand the authors time to observe (1) whether it's the right solution, and (2) whether there will be long-term maintenance. If both of these are true, and the package is widely used, it may well make it into trunk subsequently.

Cheers

 -- Martin

On 7/29/11 8:38 AM, martin odersky wrote:
> At other times, we have been too conservative out of fear or
> exhaustion to accept worthwhile packages.

Out of fear or exhaustion! I wish I'd come up with that.

I agree that the infrastructure part (Scala Fresh) should be a separate discussion from the "blessed libraries" part.
The former would _benefit_ from having overlapping libraries, whereas the latter might not.
--j
On Fri, Jul 29, 2011 at 11:17 AM, Seth Tisue <seth@tisue.net> wrote:

On Fri, Jul 29, 2011 at 11:00 AM, Johannes Rudolph
<johannes.rudolph@googlemail.com> wrote:
> That's why it would be best to separate technical and possibly
> political things
> [...]
> This can all be
> done in the background without anyone starting to ask questions.

+1

The infrastructure discussion seems pretty much separate from the
discussion about officially blessing and bundling stuff for the
public. No need to hold up the former because of uncertainty over the
latter.

--
Seth Tisue | Northwestern University | http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

Hi,

On 29 July 2011 15:41, Paul Phillips <paulp@improving.org> wrote:

Humans should be removed from the loop whenever possible.  Any situation which can be characterized as "library authors and library users hopefully notice" something is highly suboptimal unless the noticing we're hoping they're doing inherently requires the application of human-level intelligence.  Which in this case, as in most others, it does not.

The surface area of the distribution is too large to rely completely on tests even if our test suite was stellar.  The computing resources and ongoing effort necessary to perform continuous integration with a selection of projects are miniscule compared to the benefits provided, even if the only benefit was extempore's saved time.

So this reason alone is plenty of justification for regular builds of some subset of projects.

Up to here, I completely agree. Where the projects to be hosted on github or somewhere else where you can clone/checkout the source, there's no need for tight integration between their repository and your CI though.   

 From there it's only sensible to pick useful projects and publish the artifacts which are being built anyway.

This is where I don't quite agree. It comes down to 'useful projects'. The projects that are useful to you in finding regressions and issues with the compiler will probably be different to those projects that consider it useful to be part of this continuous integration project, and this will differ from what is useful to an end-user hunting for some code that addresses today's pressing need.  

Don't smother too many babies in the crib.  It's hard to predict what will work out and what won't.  If in doubt, usually the best way to find out is to try it.

Back to agreeing with you again :D


--
Dr Matthew PocockVisitor, School of Computing Science, Newcastle Universitymailto: turingatemyhamster@gmail.com gchat: turingatemyhamster@gmail.commsn: matthew_pocock@yahoo.co.uk irc.freenode.net: drdozertel: (0191) 2566550mob: +447535664143