Collections Testsuite needed?

If you'd like to start looking into it, please do!  I'm working on an SBT build for Scala right now that should ease the usage of standard scala testing frameworks for such an endeavor.  In the meantime you could do one of two things:
(1) Fork scala on github and add scalacheck collections tests using the partest-scalacheck framework (little documentation, just follow examples and ask for help)(2) Create a separate project that contains all the nitty gritty collections tests.  We can fire this one off in hudson after the nightly against the nightly.
I'll be more than willing to help out as I can.  Once the git migration and sbt builds are done, improving testing on Scalac is one of my personal aims.
- Josh

On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller <ruediger.keller@googlemail.com> wrote:

Hello,

in an interview earlier this year Paul Phillips said that a test suite
of correctness tests for the Scala collections framework was
desperately needed.

Is this still the case and is still nobody working on it?

If so, I will start looking into this. If I understand correctly, it
should run with partest which brings it's own fork of scalacheck.

Regards,
Rüdiger

I think Paul was also looking for automatic, exhaustive test generation for the collections types. To me, it seems that without a huge amount of manual effort the best route to doing that would be to construct some formal model of how the collections types should behave, and be able to generate tests from that which exhaustively cover the possible usage patterns or certainly just generate thousands (millions?) of tests that cover a significant portion of the likely usage. Yuvi might have some thoughts on that, as he is building a fuzzer for the compiler and might know of good strategies for doing something like this.
Iain

On Mon, Aug 1, 2011 at 9:13 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

If you'd like to start looking into it, please do!  I'm working on an SBT build for Scala right now that should ease the usage of standard scala testing frameworks for such an endeavor.  In the meantime you could do one of two things:
(1) Fork scala on github and add scalacheck collections tests using the partest-scalacheck framework (little documentation, just follow examples and ask for help)(2) Create a separate project that contains all the nitty gritty collections tests.  We can fire this one off in hudson after the nightly against the nightly.
I'll be more than willing to help out as I can.  Once the git migration and sbt builds are done, improving testing on Scalac is one of my personal aims.
- Josh

On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller <ruediger.keller@googlemail.com> wrote:

Hello,

in an interview earlier this year Paul Phillips said that a test suite
of correctness tests for the Scala collections framework was
desperately needed.

Is this still the case and is still nobody working on it?

If so, I will start looking into this. If I understand correctly, it
should run with partest which brings it's own fork of scalacheck.

Regards,
Rüdiger

On 8/1/11 6:02 AM, Rüdiger Keller wrote:
> Is this still the case and is still nobody working on it?

There's a promising looking start here:

https://github.com/balshor/scala-collection-test

> If so, I will start looking into this. If I understand correctly, it
> should run with partest which brings it's own fork of scalacheck.

The only reason scalacheck is "forked" is because if we tried to develop
against a scalacheck jar, any change in trunk which was binary
incompatible with scalacheck would break the test suite. There are no
changes of interest to the source.

It doesn't need to (and likely shouldn't) use partest. The important
thing is that it be runnable from sbt 0.10.

Oh, so much feedback. Nice. :-)

Ok, I will start looking into this, but I can make no promises. I will
create a separate project based on scalacheck, buildable with sbt
0.10.

Not sure if I should contact "balshor", though. I do this to learn
something about testing with scalacheck as well, so I'm not sure if I
just want to jump on the wagon. On the other hand, doing everything
twice is kinda dumb, too. :-D

We'll see.

Cheers,
Rüdiger

2011/8/1 Paul Phillips :
> On 8/1/11 6:02 AM, Rüdiger Keller wrote:
>>
>> Is this still the case and is still nobody working on it?
>
> There's a promising looking start here:
>
> https://github.com/balshor/scala-collection-test
>
>> If so, I will start looking into this. If I understand correctly, it
>> should run with partest which brings it's own fork of scalacheck.
>
> The only reason scalacheck is "forked" is because if we tried to develop
> against a scalacheck jar, any change in trunk which was binary incompatible
> with scalacheck would break the test suite.  There are no changes of
> interest to the source.
>
> It doesn't need to (and likely shouldn't) use partest.  The important thing
> is that it be runnable from sbt 0.10.
>

Hi,

I'm looking for some advice.

Would you recommend using Scalacheck only, or would you additionally
use Specs2 or ScalaTest?

I have no experience with either of them (which is part of the reason
I'm doing this), but I would like to avoid getting into too many
things at once.

Regards,
Rüdiger

2011/8/1 Josh Suereth :
> If you'd like to start looking into it, please do!  I'm working on an SBT
> build for Scala right now that should ease the usage of standard scala
> testing frameworks for such an endeavor.  In the meantime you could do one
> of two things:
> (1) Fork scala on github and add scalacheck collections tests using the
> partest-scalacheck framework (little documentation, just follow examples and
> ask for help)
> (2) Create a separate project that contains all the nitty gritty collections
> tests.  We can fire this one off in hudson after the nightly against the
> nightly.
> I'll be more than willing to help out as I can.  Once the git migration and
> sbt builds are done, improving testing on Scalac is one of my personal aims.
> - Josh
>
> On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller
> wrote:
>>
>> Hello,
>>
>> in an interview earlier this year Paul Phillips said that a test suite
>> of correctness tests for the Scala collections framework was
>> desperately needed.
>>
>> Is this still the case and is still nobody working on it?
>>
>> If so, I will start looking into this. If I understand correctly, it
>> should run with partest which brings it's own fork of scalacheck.
>>
>> Regards,
>> Rüdiger
>
>

Whichever one supports sbt 0.10 is the clincher.  For the new Scala build, please ensure *everything* works with sbt 0.10 and then it will be trivial for us to pull in the project for nightlies.  Otherwise, you may be working on something that would be too difficult to integrate.
I know scalacheck is ok to use.  Not sure about any other testing lib yet.

On Tue, Aug 2, 2011 at 8:00 AM, Ruediger Keller <ruediger.keller@rk42.de> wrote:

Hi,

I'm looking for some advice.

Would you recommend using Scalacheck only, or would you additionally
use Specs2 or ScalaTest?

I have no experience with either of them (which is part of the reason
I'm doing this), but I would like to avoid getting into too many
things at once.

Regards,
Rüdiger


2011/8/1 Josh Suereth <joshua.suereth@gmail.com>:
> If you'd like to start looking into it, please do!  I'm working on an SBT
> build for Scala right now that should ease the usage of standard scala
> testing frameworks for such an endeavor.  In the meantime you could do one
> of two things:
> (1) Fork scala on github and add scalacheck collections tests using the
> partest-scalacheck framework (little documentation, just follow examples and
> ask for help)
> (2) Create a separate project that contains all the nitty gritty collections
> tests.  We can fire this one off in hudson after the nightly against the
> nightly.
> I'll be more than willing to help out as I can.  Once the git migration and
> sbt builds are done, improving testing on Scalac is one of my personal aims.
> - Josh
>
> On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller
> <ruediger.keller@googlemail.com> wrote:
>>
>> Hello,
>>
>> in an interview earlier this year Paul Phillips said that a test suite
>> of correctness tests for the Scala collections framework was
>> desperately needed.
>>
>> Is this still the case and is still nobody working on it?
>>
>> If so, I will start looking into this. If I understand correctly, it
>> should run with partest which brings it's own fork of scalacheck.
>>
>> Regards,
>> Rüdiger
>
>

Yes, I'm basing this on sbt 0.10, so no worries there.

sbt 0.10 supports all three testing libs out of the box, it seems, so
that should be no problem either.

Perhaps I will go with scalacheck only at first and later I tend to
examine specs2, too.

Regards,
Rüdiger

2011/8/2 Josh Suereth :
> Whichever one supports sbt 0.10 is the clincher.  For the new Scala build,
> please ensure *everything* works with sbt 0.10 and then it will be trivial
> for us to pull in the project for nightlies.  Otherwise, you may be working
> on something that would be too difficult to integrate.
> I know scalacheck is ok to use.  Not sure about any other testing lib yet.
>
> On Tue, Aug 2, 2011 at 8:00 AM, Ruediger Keller
> wrote:
>>
>> Hi,
>>
>> I'm looking for some advice.
>>
>> Would you recommend using Scalacheck only, or would you additionally
>> use Specs2 or ScalaTest?
>>
>> I have no experience with either of them (which is part of the reason
>> I'm doing this), but I would like to avoid getting into too many
>> things at once.
>>
>> Regards,
>> Rüdiger
>>
>>
>> 2011/8/1 Josh Suereth :
>> > If you'd like to start looking into it, please do!  I'm working on an
>> > SBT
>> > build for Scala right now that should ease the usage of standard scala
>> > testing frameworks for such an endeavor.  In the meantime you could do
>> > one
>> > of two things:
>> > (1) Fork scala on github and add scalacheck collections tests using the
>> > partest-scalacheck framework (little documentation, just follow examples
>> > and
>> > ask for help)
>> > (2) Create a separate project that contains all the nitty gritty
>> > collections
>> > tests.  We can fire this one off in hudson after the nightly against the
>> > nightly.
>> > I'll be more than willing to help out as I can.  Once the git migration
>> > and
>> > sbt builds are done, improving testing on Scalac is one of my personal
>> > aims.
>> > - Josh
>> >
>> > On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> in an interview earlier this year Paul Phillips said that a test suite
>> >> of correctness tests for the Scala collections framework was
>> >> desperately needed.
>> >>
>> >> Is this still the case and is still nobody working on it?
>> >>
>> >> If so, I will start looking into this. If I understand correctly, it
>> >> should run with partest which brings it's own fork of scalacheck.
>> >>
>> >> Regards,
>> >> Rüdiger
>> >
>> >
>
>

Ah, so here's where the fun comes in.  The testing framework must *build* with sbt 0.10 so that we can rebuild it with the nightly.  There's a chance the nightly could break the testing framework, but we'll have to deal with a bit of noise from these tests.  Better to have them then not.  In any case, this is slightly safer than relying on binary compatibility, since that can change on major Scala versions.
I'm currently working on the sbt 0.10 build for scala itself.  We can chat offline if you want over all the requirements I'll have to be able to pull your project in directly to a Scala nightly build.
- Josh

On Tue, Aug 2, 2011 at 8:45 AM, Ruediger Keller <ruediger.keller@rk42.de> wrote:

Yes, I'm basing this on sbt 0.10, so no worries there.

sbt 0.10 supports all three testing libs out of the box, it seems, so
that should be no problem either.

Perhaps I will go with scalacheck only at first and later I tend to
examine specs2, too.

Regards,
Rüdiger

2011/8/2 Josh Suereth <joshua.suereth@gmail.com>:
> Whichever one supports sbt 0.10 is the clincher.  For the new Scala build,
> please ensure *everything* works with sbt 0.10 and then it will be trivial
> for us to pull in the project for nightlies.  Otherwise, you may be working
> on something that would be too difficult to integrate.
> I know scalacheck is ok to use.  Not sure about any other testing lib yet.
>
> On Tue, Aug 2, 2011 at 8:00 AM, Ruediger Keller <ruediger.keller@rk42.de>
> wrote:
>>
>> Hi,
>>
>> I'm looking for some advice.
>>
>> Would you recommend using Scalacheck only, or would you additionally
>> use Specs2 or ScalaTest?
>>
>> I have no experience with either of them (which is part of the reason
>> I'm doing this), but I would like to avoid getting into too many
>> things at once.
>>
>> Regards,
>> Rüdiger
>>
>>
>> 2011/8/1 Josh Suereth <joshua.suereth@gmail.com>:
>> > If you'd like to start looking into it, please do!  I'm working on an
>> > SBT
>> > build for Scala right now that should ease the usage of standard scala
>> > testing frameworks for such an endeavor.  In the meantime you could do
>> > one
>> > of two things:
>> > (1) Fork scala on github and add scalacheck collections tests using the
>> > partest-scalacheck framework (little documentation, just follow examples
>> > and
>> > ask for help)
>> > (2) Create a separate project that contains all the nitty gritty
>> > collections
>> > tests.  We can fire this one off in hudson after the nightly against the
>> > nightly.
>> > I'll be more than willing to help out as I can.  Once the git migration
>> > and
>> > sbt builds are done, improving testing on Scalac is one of my personal
>> > aims.
>> > - Josh
>> >
>> > On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller
>> > <ruediger.keller@googlemail.com> wrote:
>> >>
>> >> Hello,
>> >>
>> >> in an interview earlier this year Paul Phillips said that a test suite
>> >> of correctness tests for the Scala collections framework was
>> >> desperately needed.
>> >>
>> >> Is this still the case and is still nobody working on it?
>> >>
>> >> If so, I will start looking into this. If I understand correctly, it
>> >> should run with partest which brings it's own fork of scalacheck.
>> >>
>> >> Regards,
>> >> Rüdiger
>> >
>> >
>
>

On 8/2/11 5:49 AM, Josh Suereth wrote:
> I'm currently working on the sbt 0.10 build for scala itself. We can
> chat offline if you want over all the requirements I'll have to be able
> to pull your project in directly to a Scala nightly build.

The challenge tends to be more in the completion of the task at hand
than in the subsequent integration. If my only problem were to be
figuring out how to integrate useful work into the build, then I would
be a happy problem sufferer. I say this in case "all the requirements"
sounds at all imposing: don't worry, this is not where the ship founders.

Ok, now I get it! :-)

Scalacheck is safe then, it builds with sbt 0.10, it seems.

I will get back to you, when I have something remotely presentable, or
with some more questions.

Regards,
Rüdiger

2011/8/2 Josh Suereth :
> Ah, so here's where the fun comes in.  The testing framework must *build*
> with sbt 0.10 so that we can rebuild it with the nightly.  There's a chance
> the nightly could break the testing framework, but we'll have to deal with a
> bit of noise from these tests.  Better to have them then not.  In any case,
> this is slightly safer than relying on binary compatibility, since that can
> change on major Scala versions.
> I'm currently working on the sbt 0.10 build for scala itself.  We can chat
> offline if you want over all the requirements I'll have to be able to pull
> your project in directly to a Scala nightly build.
> - Josh
>
> On Tue, Aug 2, 2011 at 8:45 AM, Ruediger Keller
> wrote:
>>
>> Yes, I'm basing this on sbt 0.10, so no worries there.
>>
>> sbt 0.10 supports all three testing libs out of the box, it seems, so
>> that should be no problem either.
>>
>> Perhaps I will go with scalacheck only at first and later I tend to
>> examine specs2, too.
>>
>> Regards,
>> Rüdiger
>>
>> 2011/8/2 Josh Suereth :
>> > Whichever one supports sbt 0.10 is the clincher.  For the new Scala
>> > build,
>> > please ensure *everything* works with sbt 0.10 and then it will be
>> > trivial
>> > for us to pull in the project for nightlies.  Otherwise, you may be
>> > working
>> > on something that would be too difficult to integrate.
>> > I know scalacheck is ok to use.  Not sure about any other testing lib
>> > yet.
>> >
>> > On Tue, Aug 2, 2011 at 8:00 AM, Ruediger Keller
>> >
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm looking for some advice.
>> >>
>> >> Would you recommend using Scalacheck only, or would you additionally
>> >> use Specs2 or ScalaTest?
>> >>
>> >> I have no experience with either of them (which is part of the reason
>> >> I'm doing this), but I would like to avoid getting into too many
>> >> things at once.
>> >>
>> >> Regards,
>> >> Rüdiger
>> >>
>> >>
>> >> 2011/8/1 Josh Suereth :
>> >> > If you'd like to start looking into it, please do!  I'm working on an
>> >> > SBT
>> >> > build for Scala right now that should ease the usage of standard
>> >> > scala
>> >> > testing frameworks for such an endeavor.  In the meantime you could
>> >> > do
>> >> > one
>> >> > of two things:
>> >> > (1) Fork scala on github and add scalacheck collections tests using
>> >> > the
>> >> > partest-scalacheck framework (little documentation, just follow
>> >> > examples
>> >> > and
>> >> > ask for help)
>> >> > (2) Create a separate project that contains all the nitty gritty
>> >> > collections
>> >> > tests.  We can fire this one off in hudson after the nightly against
>> >> > the
>> >> > nightly.
>> >> > I'll be more than willing to help out as I can.  Once the git
>> >> > migration
>> >> > and
>> >> > sbt builds are done, improving testing on Scalac is one of my
>> >> > personal
>> >> > aims.
>> >> > - Josh
>> >> >
>> >> > On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller
>> >> > wrote:
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> in an interview earlier this year Paul Phillips said that a test
>> >> >> suite
>> >> >> of correctness tests for the Scala collections framework was
>> >> >> desperately needed.
>> >> >>
>> >> >> Is this still the case and is still nobody working on it?
>> >> >>
>> >> >> If so, I will start looking into this. If I understand correctly, it
>> >> >> should run with partest which brings it's own fork of scalacheck.
>> >> >>
>> >> >> Regards,
>> >> >> Rüdiger
>> >> >
>> >> >
>> >
>> >
>
>

Hi,

I just checked in my first steps of the collections test suite based
on sbt 0.10 and scalacheck.

https://github.com/ruedigerk/collection-tests

Feedback is welcome.

Regards,
Rüdiger

2011/8/2 Ruediger Keller :
> Ok, now I get it! :-)
>
> Scalacheck is safe then, it builds with sbt 0.10, it seems.
>
> I will get back to you, when I have something remotely presentable, or
> with some more questions.
>
> Regards,
> Rüdiger
>
> 2011/8/2 Josh Suereth :
>> Ah, so here's where the fun comes in.  The testing framework must *build*
>> with sbt 0.10 so that we can rebuild it with the nightly.  There's a chance
>> the nightly could break the testing framework, but we'll have to deal with a
>> bit of noise from these tests.  Better to have them then not.  In any case,
>> this is slightly safer than relying on binary compatibility, since that can
>> change on major Scala versions.
>> I'm currently working on the sbt 0.10 build for scala itself.  We can chat
>> offline if you want over all the requirements I'll have to be able to pull
>> your project in directly to a Scala nightly build.
>> - Josh
>>
>> On Tue, Aug 2, 2011 at 8:45 AM, Ruediger Keller
>> wrote:
>>>
>>> Yes, I'm basing this on sbt 0.10, so no worries there.
>>>
>>> sbt 0.10 supports all three testing libs out of the box, it seems, so
>>> that should be no problem either.
>>>
>>> Perhaps I will go with scalacheck only at first and later I tend to
>>> examine specs2, too.
>>>
>>> Regards,
>>> Rüdiger
>>>
>>> 2011/8/2 Josh Suereth :
>>> > Whichever one supports sbt 0.10 is the clincher.  For the new Scala
>>> > build,
>>> > please ensure *everything* works with sbt 0.10 and then it will be
>>> > trivial
>>> > for us to pull in the project for nightlies.  Otherwise, you may be
>>> > working
>>> > on something that would be too difficult to integrate.
>>> > I know scalacheck is ok to use.  Not sure about any other testing lib
>>> > yet.
>>> >
>>> > On Tue, Aug 2, 2011 at 8:00 AM, Ruediger Keller
>>> >
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm looking for some advice.
>>> >>
>>> >> Would you recommend using Scalacheck only, or would you additionally
>>> >> use Specs2 or ScalaTest?
>>> >>
>>> >> I have no experience with either of them (which is part of the reason
>>> >> I'm doing this), but I would like to avoid getting into too many
>>> >> things at once.
>>> >>
>>> >> Regards,
>>> >> Rüdiger
>>> >>
>>> >>
>>> >> 2011/8/1 Josh Suereth :
>>> >> > If you'd like to start looking into it, please do!  I'm working on an
>>> >> > SBT
>>> >> > build for Scala right now that should ease the usage of standard
>>> >> > scala
>>> >> > testing frameworks for such an endeavor.  In the meantime you could
>>> >> > do
>>> >> > one
>>> >> > of two things:
>>> >> > (1) Fork scala on github and add scalacheck collections tests using
>>> >> > the
>>> >> > partest-scalacheck framework (little documentation, just follow
>>> >> > examples
>>> >> > and
>>> >> > ask for help)
>>> >> > (2) Create a separate project that contains all the nitty gritty
>>> >> > collections
>>> >> > tests.  We can fire this one off in hudson after the nightly against
>>> >> > the
>>> >> > nightly.
>>> >> > I'll be more than willing to help out as I can.  Once the git
>>> >> > migration
>>> >> > and
>>> >> > sbt builds are done, improving testing on Scalac is one of my
>>> >> > personal
>>> >> > aims.
>>> >> > - Josh
>>> >> >
>>> >> > On Mon, Aug 1, 2011 at 9:02 AM, Rüdiger Keller
>>> >> > wrote:
>>> >> >>
>>> >> >> Hello,
>>> >> >>
>>> >> >> in an interview earlier this year Paul Phillips said that a test
>>> >> >> suite
>>> >> >> of correctness tests for the Scala collections framework was
>>> >> >> desperately needed.
>>> >> >>
>>> >> >> Is this still the case and is still nobody working on it?
>>> >> >>
>>> >> >> If so, I will start looking into this. If I understand correctly, it
>>> >> >> should run with partest which brings it's own fork of scalacheck.
>>> >> >>
>>> >> >> Regards,
>>> >> >> Rüdiger
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>