Scala Testing Frameworks

On Wed, Mar 18, 2009 at 19:55, Randall R Schulz wrote:
> Is there currently a clearly superior choice for a unit testing
> framework for Scala? If so, what is it? If not, what are the tradeoffs
> to consider?

Specs. Upcoming release has a cool option to output test results to
the terminal in color :)

Specs is almost perfect — simple and easy to use. The one problem is
package name org.specs that I hope will be renamed to something more
appropriate one day.

S.

Randall,

On Wed, 2009-03-18 at 09:55 -0700, Randall R Schulz wrote:
> Hi,
>
> Is there currently a clearly superior choice for a unit testing
> framework for Scala? If so, what is it? If not, what are the tradeoffs
> to consider?

Bizarrely (or not), I asked effectively this same question here a few
weeks back. Of the many excellent frameworks, and one or two not so
good ones, the majority view seemed to be the excellent ScalaTest and
equally excellent, but slightly different, Specs, in some combination.
I am sure the other excellent ones are deserving of use but these two
seemed to get people's votes.

I tried scalacheck, and setting aside the issue of whether it is a good testing framework, I would say it is definitely not for unittesting. On my code it just hangs (meaning, by the morning it's still not there yet). Plain junit-style testing, otoh, is okay.

2009/3/18 Randall R Schulz <rschulz@sonic.net>

Hi,

Is there currently a clearly superior choice for a unit testing
framework for Scala? If so, what is it? If not, what are the tradeoffs
to consider?


Randall Schulz

--
Thanks,
-Vlad

2009/3/18 Vlad Patryshev :
> I tried scalacheck, and setting aside the issue of whether it is a good
> testing framework, I would say it is definitely not for unittesting. On my
> code it just hangs (meaning, by the morning it's still not there yet). Plain
> junit-style testing, otoh, is okay.

I tried writing while(true){} println("hello world"); and it never did
anything. It just sat there. :-( Clearly Scala is not suitable for
production use.

Or in other words: Perhaps you did something wrong? You're the only
person I've ever heard reporting this problem, so I suspect the
chances are much better that you introduced an infinite loop somewhere
than that it's actually something wrong with scalacheck. Did you try
getting other people to look at your code and see what's wrong?

Okay; let me go ahead and reproduce it, and either confess I screwed it or send you the code. :)

2009/3/18 David MacIver <david.maciver@gmail.com>

2009/3/18 Vlad Patryshev <vpatryshev@gmail.com>:
> I tried scalacheck, and setting aside the issue of whether it is a good
> testing framework, I would say it is definitely not for unittesting. On my
> code it just hangs (meaning, by the morning it's still not there yet). Plain
> junit-style testing, otoh, is okay.

I tried writing while(true){} println("hello world"); and it never did
anything. It just sat there. :-( Clearly Scala is not suitable for
production use.

Or in other words: Perhaps you did something wrong? You're the only
person I've ever heard reporting this problem, so I suspect the
chances are much better that you introduced an infinite loop somewhere
than that it's actually something wrong with scalacheck. Did you try
getting other people to look at your code and see what's wrong?

--
Thanks,
-Vlad

I've been using Specs for a while, which I really like since I use RSpec when I write Ruby code (heresy, I know!).  The "XSpec" frameworks support the Behavior-Driven Development style of testing.
However, the new BDD syntax option in ScalaTest looks very good and in some ways has a more natural feel. Which tool you use and which "style" in the tool you pick will probably come down to personal preferences.  Don't overlook the integration with your existing infrastructure, as they offer different options, such as driving your tests with JUnit.
Assuming there isn't a real problem with ScalaCheck ;), it is a very intriguing approach to testing that's worth a look. I'm still evolving my testing strategy, but I think I'm heading toward a combination of Specs or ScalaTest + ScalaCheck. 
dean

On Wed, Mar 18, 2009 at 12:02 PM, Stepan Koltsov <stepan.koltsov@gmail.com> wrote:

On Wed, Mar 18, 2009 at 19:55, Randall R Schulz <rschulz@sonic.net> wrote:
> Is there currently a clearly superior choice for a unit testing
> framework for Scala? If so, what is it? If not, what are the tradeoffs
> to consider?

Specs. Upcoming release has a cool option to output test results to
the terminal in color :)

Specs is almost perfect — simple and easy to use. The one problem is
package name org.specs that I hope will be renamed to something more
appropriate one day.

S.

--
Dean Wampler
twitter: @deanwampler, @chicagoscala
Chicago-Area Scala Enthusiasts (CASE):
-  http://groups.google.com/group/chicagoscala
-  http://www.meetup.com/chicagoscala/ (Meetings)
http://www.objectmentor.com
http://www.polyglotprogramming.com
http://www.aspectprogramming.com
http://aquarium.rubyforge.org
http://www.contract4j.org

Hi Randall,

On Wed, Mar 18, 2009 at 9:55 AM, Randall R Schulz wrote:
> Hi,
>
> Is there currently a clearly superior choice for a unit testing
> framework for Scala? If so, what is it? If not, what are the tradeoffs
> to consider?
>
I think that everyone should take a look at ScalaCheck, but that for
most people it won't be sufficient. Some people may want to only write
properties, but I think most people will want to combine ScalaCheck
property-based tests with traditional assertion-based tests. And for
that you can use any of ScalaTest, Specs, JUnit, or TestNG. ScalaTest
and Specs both have nice integration with ScalaCheck, so baring other
considerations I think most people will probably pick one of those.

Specs and ScalaTest differ in several ways. They are both good tools,
so it is not that one is better or worse than the other. But they are
different. This question keeps coming up so I'll try and capture some
of the differences that I know of. One main difference between the
tools is that Specs is designed for Behavior-Driven Development (BDD),
whereas ScalaTest is more general. ScalaTest provides traits that you
can mix together to get whatever behavior you prefer in your test
classes, and you can also easily define your own behavior if you want
something different.

ScalaTest does support BDD though its Spec trait, and also has traits
that you can mix in to get a nice matcher syntax. If you like
"should", you mix in ShouldMatchers. If you like "must", you mix in
MustMatchers. But if you like BDD but don't like matcher syntax, you
can just use ScalaTest's Spec trait without mixing in a matchers
trait. Specs has a Specification class that you extend, and you must
use the word "must" in your matcher expressions.

The matcher syntax is also different between ScalaTest and Specs. In
ScalaTest I tried to see how far I could go with operator notation,
and ended up with matcher expressions that read very much like English
sentences, with spaces between the words. Specs matcher syntax runs
words together more with camel case.

ScalaTest also has two dynamic matchers that I don't believe Specs
has, which is a convenient way to check properties using Scala symbol
literals. Specs does have more matchers than ScalaTest, and that
reflects possibly a difference in design attitude. I actually cut
probably 2/3 of the matcher syntax I built and considered for release.
I will add more matchers in future releases, but wanted to be sure I
knew users actually wanted something before I added it. However
ScalaTest's dynamic property matcher syntax takes up some of that
slack. For example in Specs you can write on a java.io.File:

file must beDirectory

This will invoke the isDirectory and make sure it is true. ScalaTest
does not have any special matchers for java.io.Files currently, but in
ScalaTest, you could just use a dynamic check like this:

file must be a ('directory)

Anytime you pass a symbol in after "be", it will use reflection to
look for (in this case) a method or field named "directory" or a
method named "isDirectory". There's also a way to make this static, by
defining a BePropertyMatcher (which requires only 2 or 3 lines of code
usually). So basically in ScalaTest I try to provide more
functionality with less API.

The syntax of Spec in ScalaTest looks more like Ruby's RSpec, using
"describe" to specify the system under test (SUT) and "it" to specify
an example. Specs Specification class declares implicit conversions on
string to specify the SUT and examples. One difference here is that
you can nest describe clauses in a ScalaTest Spec, as you can in
RSpec, but I don't think that's possible in Specs (Eric, can you
verify this?)

On that subject, I think implicit conversions constitutes a general
design attitude difference between Specs and ScalaTest. By default you
get only one implicit conversion when you use ScalaTest, which is the
one that puts the === operator on everything. (If you need to, you can
"turn off" this implicit conversion with one line of code. The only
reason you would need to do that is if you were trying to test
something that has its own === operator, and you get a conflict.)
ScalaTest defines many other implicit conversions, but to use them you
need to explicitly "invite" them into your code by mixing in a trait
or doing an import. When you extend class Specification in Specs I
think you pretty much get dozens of implicit conversion by default.
I'm not sure how much that will matter in practice, but I figure
people will want to test code that uses their own implicits, and
sometimes there may be a conflict between the test framwork's
implicits and those of the production code. When that happens I think
it may be easier to work around the problem in ScalaTest than Specs.

Another difference in design attitude that I've noticed is comfort
with operators. One goal I had was that any programmer looking at
someone else's test code that uses ScalaTest would be able to guess
what the meaning was without looking anything up in the ScalaTest
documentation. I wanted ScalaTest client code to be drop dead obvious.
One way that goal manifested itself is that ScalaTest is very
conservative about operators. I only define five operators in
ScalaTest: ===, which means equals, >, which means greater than, <,
less than, >=, greater than or equal, and <=, less than or equal.
That's it. So these things pretty much look like what mean. If you see
in someone else's code:

result should be <= 7

My hope is that you won't need to run to the API documentation to
guess what that <= means. By contrast, Eric has been much freer with
operators in Specs. Nothing wrong with that, but it is a difference.
Operators can make code more concise, but the tradeoff is you may have
to run to the documentation when you find things like ->-, >>, |, |>,
!, or ^^^ (which all have special meanings in Specs) in your
colleague's test code.

Specs has a few features that ScalaTest doesn't have. The ->- operator
for example is used to bring in shared behavior, I think. (I may be
confused as to how that works.) In ScalaTest, you can bring in shared
behavior, but there's no special syntax sugar right now. (Actually a
slicker "should behave like" syntax is private in Should/MustMatchers
in 0.9.5, because I need to split it off into another trait. So that
is something that will come out in a future release.) Specs has an
"aka" operator which I think is very well named. I didn't put anything
like that into ScalaTest because I'm not sure it is needed. Users
could convince me, but so far I've found most objects in Scala have
pretty good toString messages. Specs also has a data table feature,
which ScalaTest doesn't have. I frankly in all my years of testing
have never really needed that. So unless lots of users scream that it
is useful it probably won't go in, and besides, I think a ScalaCheck
property check would in most cases be better. However if users want it
they will get it. Specs also has a lightweight mocking library, and
some syntax sugar for JMock. In either ScalaTest or Specs you can use
any of the Java mocking libraries: Mockito, JMock, EasyMock,
PowerMock, etc. I don't currently offer any sugar for any of these in
ScalaTest. Sugar is something that can be added, potentially, but
currently all these mocking frameworks work just fine in ScalaTest by
just importing and using them. But that's also a difference. By the
way I have found that Scala is so concise that often I just write
mocks by hand in Scala.

One other difference is that I do try and make it just slightly easier
in ScalaTest to use a functional style when you need to share a
fixture, whereas Specs by default continues the tradition of the setUp
and tearDown approach popularized by JUnit, in which you reassign vars
before each test. However if you want to test that way, it is also
very easy in ScalaTest. You just need to mix in the BeforeAndAfter
trait.

Lastly, ScalaTest has a GUI, which as of 0.9.5 displays specification
like output. Specs does not have a GUI, though with a bit of extra
code you can use JUnit's GUI. However our plan is that ScalaTest will
support Specs very well. What Eric and I have been working on together
is that when Specs is run through ScalaTest, you will get the nice
Spec-like GUI output. Basically my plan has been to support
ScalaCheck, Specs, JUnit, and TestNG as first-class citizens in
ScalaTest. Eric and I have already done some of that integration, and
have more work to do. But in the next release or two of both tools I
expect we'll have it all done.

So those are some differences to consider. Perhaps Eric can add more insight.

Thanks.

Bill

Hi Bill,

It would be great if you captured this design rationale and placed it on a page somewhere on the ScalaTest website.  As a framework user, it's always good to understand the motivations of the designer(s) to better assimilate framework itself.

alex


On Wed, Mar 18, 2009 at 2:23 PM, Bill Venners <bill@artima.com> wrote:

Hi Randall,

On Wed, Mar 18, 2009 at 9:55 AM, Randall R Schulz <rschulz@sonic.net> wrote:
> Hi,
>
> Is there currently a clearly superior choice for a unit testing
> framework for Scala? If so, what is it? If not, what are the tradeoffs
> to consider?
>
I think that everyone should take a look at ScalaCheck, but that for
most people it won't be sufficient. Some people may want to only write
properties, but I think most people will want to combine ScalaCheck
property-based tests with traditional assertion-based tests. And for
that you can use any of ScalaTest, Specs, JUnit, or TestNG. ScalaTest
and Specs both have nice integration with ScalaCheck, so baring other
considerations I think most people will probably pick one of those.

Specs and ScalaTest differ in several ways. They are both good tools,
so it is not that one is better or worse than the other. But they are
different. This question keeps coming up so I'll try and capture some
of the differences that I know of. One main difference between the
tools is that Specs is designed for Behavior-Driven Development (BDD),
whereas ScalaTest is more general. ScalaTest provides traits that you
can mix together to get whatever behavior you prefer in your test
classes, and you can also easily define your own behavior if you want
something different.

ScalaTest does support BDD though its Spec trait, and also has traits
that you can mix in to get a nice matcher syntax. If you like
"should", you mix in ShouldMatchers. If you like "must", you mix in
MustMatchers. But if you like BDD but don't like matcher syntax, you
can just use ScalaTest's Spec trait without mixing in a matchers
trait. Specs has a Specification class that you extend, and you must
use the word "must" in your matcher expressions.

The matcher syntax is also different between ScalaTest and Specs. In
ScalaTest I tried to see how far I could go with operator notation,
and ended up with matcher expressions that read very much like English
sentences, with spaces between the words. Specs matcher syntax runs
words together more with camel case.

ScalaTest also has two dynamic matchers that I don't believe Specs
has, which is a convenient way to check properties using Scala symbol
literals. Specs does have more matchers than ScalaTest, and that
reflects possibly a difference in design attitude. I actually cut
probably 2/3 of the matcher syntax I built and considered for release.
I will add more matchers in future releases, but wanted to be sure I
knew users actually wanted something before I added it. However
ScalaTest's dynamic property matcher syntax takes up some of that
slack. For example in Specs you can write on a java.io.File:

file must beDirectory

This will invoke the isDirectory and make sure it is true. ScalaTest
does not have any special matchers for java.io.Files currently, but in
ScalaTest, you could just use a dynamic check like this:

file must be a ('directory)

Anytime you pass a symbol in after "be", it will use reflection to
look for (in this case) a method or field named "directory" or a
method named "isDirectory". There's also a way to make this static, by
defining a BePropertyMatcher (which requires only 2 or 3 lines of code
usually). So basically in ScalaTest I try to provide more
functionality with less API.

The syntax of Spec in ScalaTest looks more like Ruby's RSpec, using
"describe" to specify the system under test (SUT) and "it" to specify
an example. Specs Specification class declares implicit conversions on
string to specify the SUT and examples. One difference here is that
you can nest describe clauses in a ScalaTest Spec, as you can in
RSpec, but I don't think that's possible in Specs (Eric, can you
verify this?)

On that subject, I think implicit conversions constitutes a general
design attitude difference between Specs and ScalaTest. By default you
get only one implicit conversion when you use ScalaTest, which is the
one that puts the === operator on everything. (If you need to, you can
"turn off" this implicit conversion with one line of code. The only
reason you would need to do that is if you were trying to test
something that has its own === operator, and you get a conflict.)
ScalaTest defines many other implicit conversions, but to use them you
need to explicitly "invite" them into your code by mixing in a trait
or doing an import. When you extend class Specification in Specs I
think you pretty much get dozens of implicit conversion by default.
I'm not sure how much that will matter in practice, but I figure
people will want to test code that uses their own implicits, and
sometimes there may be a conflict between the test framwork's
implicits and those of the production code. When that happens I think
it may be easier to work around the problem in ScalaTest than Specs.

Another difference in design attitude that I've noticed is comfort
with operators. One goal I had was that any programmer looking at
someone else's test code that uses ScalaTest would be able to guess
what the meaning was without looking anything up in the ScalaTest
documentation. I wanted ScalaTest client code to be drop dead obvious.
One way that goal manifested itself is that ScalaTest is very
conservative about operators. I only define five operators in
ScalaTest: ===, which means equals, >, which means greater than, <,
less than, >=, greater than or equal, and <=, less than or equal.
That's it. So these things pretty much look like what mean. If you see
in someone else's code:

result should be <= 7

My hope is that you won't need to run to the API documentation to
guess what that <= means. By contrast, Eric has been much freer with
operators in Specs. Nothing wrong with that, but it is a difference.
Operators can make code more concise, but the tradeoff is you may have
to run to the documentation when you find things like  ->-, >>, |, |>,
!, or ^^^ (which all have special meanings in Specs) in your
colleague's test code.

Specs has a few features that ScalaTest doesn't have. The ->- operator
for example is used to bring in shared behavior, I think. (I may be
confused as to how that works.) In ScalaTest, you can bring in shared
behavior, but there's no special syntax sugar right now. (Actually a
slicker "should behave like" syntax is private in Should/MustMatchers
in 0.9.5, because I need to split it off into another trait. So that
is something that will come out in a future release.) Specs has an
"aka" operator which I think is very well named. I didn't put anything
like that into ScalaTest because I'm not sure it is needed. Users
could convince me, but so far I've found most objects in Scala have
pretty good toString messages. Specs also has a data table feature,
which ScalaTest doesn't have. I frankly in all my years of testing
have never really needed that. So unless lots of users scream that it
is useful it probably won't go in, and besides, I think a ScalaCheck
property check would in most cases be better. However if users want it
they will get it. Specs also has a lightweight mocking library, and
some syntax sugar for JMock. In either ScalaTest or Specs you can use
any of the Java mocking libraries: Mockito, JMock, EasyMock,
PowerMock, etc. I don't currently offer any sugar for any of these in
ScalaTest. Sugar is something that can be added, potentially, but
currently all these mocking frameworks work just fine in ScalaTest by
just importing and using them. But that's also a difference. By the
way I have found that Scala is so concise that often I just write
mocks by hand in Scala.

One other difference is that I do try and make it just slightly easier
in ScalaTest to use a functional style when you need to share a
fixture, whereas Specs by default continues the tradition of the setUp
and tearDown approach popularized by JUnit, in which you reassign vars
before each test. However if you want to test that way, it is also
very easy in ScalaTest. You just need to mix in the BeforeAndAfter
trait.

Lastly, ScalaTest has a GUI, which as of 0.9.5 displays specification
like output. Specs does not have a GUI, though with a bit of extra
code you can use JUnit's GUI. However our plan is that ScalaTest will
support Specs very well. What Eric and I have been working on together
is that when Specs is run through ScalaTest, you will get the nice
Spec-like GUI output. Basically my plan has been to support
ScalaCheck, Specs, JUnit, and TestNG as first-class citizens in
ScalaTest. Eric and I have already done some of that integration, and
have more work to do. But in the next release or two of both tools I
expect we'll have it all done.

So those are some differences to consider. Perhaps Eric can add more insight.

Thanks.

Bill

definitely. also, if i might: it would be good for there to be a
simple bullet-point-style summary of this very informative, but long,
text. e.g.

* Specs is more focused on BDD; STest is more inclusive/general.
* Matcher syntax differs: Specs is more camelCase, STest is more operators.
* STest has dynamic matchers.
* STest looks a bit more like RSpec than does Specs (?)
* Specs uses implicits more than STest.
* Specs has more features out-of-the-box in the sense of operators/fns
to test / make assertions.
* Specs follows the setUp/tearDown model; STest tries to be more functional (?)
* STest has a GUI.

sincerely.

On Wed, Mar 18, 2009 at 3:15 PM, Alex Boisvert wrote:
> Hi Bill,
>
> It would be great if you captured this design rationale and placed it on a
> page somewhere on the ScalaTest website.  As a framework user, it's always
> good to understand the motivations of the designer(s) to better assimilate
> framework itself.
>
> alex
>
>
> On Wed, Mar 18, 2009 at 2:23 PM, Bill Venners wrote:
>>
>> Hi Randall,
>>
>> On Wed, Mar 18, 2009 at 9:55 AM, Randall R Schulz
>> wrote:
>> > Hi,
>> >
>> > Is there currently a clearly superior choice for a unit testing
>> > framework for Scala? If so, what is it? If not, what are the tradeoffs
>> > to consider?
>> >
>> I think that everyone should take a look at ScalaCheck, but that for
>> most people it won't be sufficient. Some people may want to only write
>> properties, but I think most people will want to combine ScalaCheck
>> property-based tests with traditional assertion-based tests. And for
>> that you can use any of ScalaTest, Specs, JUnit, or TestNG. ScalaTest
>> and Specs both have nice integration with ScalaCheck, so baring other
>> considerations I think most people will probably pick one of those.

Hi Raoul and Alex,

Thanks for the suggestion. I'll get together with Eric and put up some
kind of comparison somewhere of ScalaTest and Specs, so people can
more quickly understand the differences.

One other difference that occurred to me after sending my long, wordy
email was that ScalaTest has a PrivateMethodTester trait that
facilitates testing private methods. I don't think Specs has anything
like this, but if you want you can just mix ScalaTest's
PrivateMethodTester trait into your Specs Specification and it should
work just fine.

Another unrelated goal I had in mind as I designed ScalaTest is that I
wanted to use as a source of specific examples when I'm trying to
explain things about Scala in general. For example in my Devoxx talk
from last December, which will come up hopefully soon on Parleys, I
gave several examples from ScalaTest. And I just published a blog that
attempts to get across the point that Scala lets you work at a higher
level using examples from ScalaTest:

http://www.artima.com/weblogs/viewpost.jsp?thread=252702

My hope is that because most programmers are familiar with the domain
of testing, this will be an effective way to explain general concepts
about Scala.

Thanks.

Bill

On Wed, Mar 18, 2009 at 3:54 PM, Raoul Duke wrote:
> definitely. also, if i might: it would be good for there to be a
> simple bullet-point-style summary of this very informative, but long,
> text. e.g.
>
> * Specs is more focused on BDD; STest is more inclusive/general.
> * Matcher syntax differs: Specs is more camelCase, STest is more operators.
> * STest has dynamic matchers.
> * STest looks a bit more like RSpec than does Specs (?)
> * Specs uses implicits more than STest.
> * Specs has more features out-of-the-box in the sense of operators/fns
> to test / make assertions.
> * Specs follows the setUp/tearDown model; STest tries to be more functional (?)
> * STest has a GUI.
>
> sincerely.
>
>
> On Wed, Mar 18, 2009 at 3:15 PM, Alex Boisvert wrote:
>> Hi Bill,
>>
>> It would be great if you captured this design rationale and placed it on a
>> page somewhere on the ScalaTest website.  As a framework user, it's always
>> good to understand the motivations of the designer(s) to better assimilate
>> framework itself.
>>
>> alex
>>
>>
>> On Wed, Mar 18, 2009 at 2:23 PM, Bill Venners wrote:
>>>
>>> Hi Randall,
>>>
>>> On Wed, Mar 18, 2009 at 9:55 AM, Randall R Schulz
>>> wrote:
>>> > Hi,
>>> >
>>> > Is there currently a clearly superior choice for a unit testing
>>> > framework for Scala? If so, what is it? If not, what are the tradeoffs
>>> > to consider?
>>> >
>>> I think that everyone should take a look at ScalaCheck, but that for
>>> most people it won't be sufficient. Some people may want to only write
>>> properties, but I think most people will want to combine ScalaCheck
>>> property-based tests with traditional assertion-based tests. And for
>>> that you can use any of ScalaTest, Specs, JUnit, or TestNG. ScalaTest
>>> and Specs both have nice integration with ScalaCheck, so baring other
>>> considerations I think most people will probably pick one of those.
>

Hi all,

Let me add some points on top of Bill's message (sorry, looooong post):

> The matcher syntax is also different between ScalaTest and Specs. In
ScalaTest I tried to see how far I could go with operator notation,
and ended up with matcher expressions that read very much like English
sentences, with spaces between the words. Specs matcher syntax runs
words together more with camel case.

{{{==================

This is indeed a noticeable difference. The syntax in ScalaTest flows more
naturally with things like:

"hello world" must (not include ("G'day") and have size (11))

whereas you would write in specs:

"hello world" must (notInclude("G'day") and haveSize(11))

The syntax of ScalaTest is nicer but it comes to the expense of pre-defining
in the library implementation a lot of possible combinations like: and not
be, and not have, or not be, or not have,...

I don't necessarily disagree with this choice since the end result is clean
client code but this is not a trade-off I was willing to make for specs.

==================}}}

> ScalaTest also has two dynamic matchers that I don't believe Specs
has, which is a convenient way to check properties using Scala symbol
literals.

> Specs does have more matchers than ScalaTest, and that
reflects possibly a difference in design attitude. I actually cut
probably 2/3 of the matcher syntax I built and considered for release.
I will add more matchers in future releases, but wanted to be sure I
knew users actually wanted something before I added it. However
ScalaTest's dynamic property matcher syntax takes up some of that
slack. For example in Specs you can write on a java.io.File:

> file must beDirectory

> This will invoke the isDirectory and make sure it is true. ScalaTest
does not have any special matchers for java.io.Files currently, but in
ScalaTest, you could just use a dynamic check like this:

> file must be a ('directory)

> Anytime you pass a symbol in after "be", it will use reflection to
look for (in this case) a method or field named "directory" or a
method named "isDirectory". There's also a way to make this static, by
defining a BePropertyMatcher (which requires only 2 or 3 lines of code
usually). So basically in ScalaTest I try to provide more
functionality with less API.

{{{==================

In specs I wanted this to stay as "statically-typed" as possible, but I
acknowledge that this is convenient and can be ok for testing.

==================}}}

> The syntax of Spec in ScalaTest looks more like Ruby's RSpec, using
"describe" to specify the system under test (SUT) and "it" to specify
an example. Specs Specification class declares implicit conversions on
string to specify the SUT and examples.

{{{==================

There are indeed implicit conversions because I felt that using "describe"
everywhere was just adding unnecessary redundancy in the specs. However, you
can still use the implicit def:

specify("my system") should {
forExample("doThis") in {
...
}
}

==================}}}

> One difference here is that
you can nest describe clauses in a ScalaTest Spec, as you can in
RSpec, but I don't think that's possible in Specs (Eric, can you
verify this?)

{{{==================

In specs you can compose specifications and you can nest Examples
(subExamples). I think that does the trick in terms of nesting (but feedback
is welcome to show me otherwise, especially from long-time rspec
afficionados).

==================}}}

> On that subject, I think implicit conversions constitutes a general
design attitude difference between Specs and ScalaTest. By default you
get only one implicit conversion when you use ScalaTest, which is the
one that puts the === operator on everything. (If you need to, you can
"turn off" this implicit conversion with one line of code. The only
reason you would need to do that is if you were trying to test
something that has its own === operator, and you get a conflict.)
ScalaTest defines many other implicit conversions, but to use them you
need to explicitly "invite" them into your code by mixing in a trait
or doing an import. When you extend class Specification in Specs I
think you pretty much get dozens of implicit conversion by default.
I'm not sure how much that will matter in practice, but I figure
people will want to test code that uses their own implicits, and
sometimes there may be a conflict between the test framwork's
implicits and those of the production code. When that happens I think
it may be easier to work around the problem in ScalaTest than Specs.

{{{==================

Yes, that is a difference in the sense that I want in specs to have
everything at hand for usual specifications with the simplest declaration.
So, when you extends the Specification class, you get a lot of additional
traits.

However, in the case of a conflict I'm assuming that you can remove/override
some traits and compose your own "Specification class". I haven't found such
issues, but that doesn't prove anything!

==================}}}

> Another difference in design attitude that I've noticed is comfort
with operators. One goal I had was that any programmer looking at
someone else's test code that uses ScalaTest would be able to guess
what the meaning was without looking anything up in the ScalaTest
documentation. I wanted ScalaTest client code to be drop dead obvious.
One way that goal manifested itself is that ScalaTest is very
conservative about operators. I only define five operators in
ScalaTest: ===, which means equals, >, which means greater than, <,
less than, >=, greater than or equal, and <=, less than or equal.
That's it. So these things pretty much look like what mean. If you see
in someone else's code:

> result should be <= 7

> My hope is that you won't need to run to the API documentation to
guess what that <= means. By contrast, Eric has been much freer with
operators in Specs. Nothing wrong with that, but it is a difference.
Operators can make code more concise, but the tradeoff is you may have
to run to the documentation when you find things like ->-, >>, |, |>,
!, or ^^^ (which all have special meanings in Specs) in your
colleague's test code.

{{{==================

True! I've been much more liberal with operators. However the operators
you're mentioning are used in very specific (you could say "advanced")
situations.

For example ->- is used to share "Contexts" (which are a common set of
setUp/tearDown operations to setup the context of a system under
specification).

Similarly "|" and "|>" are used when specifying DataTables (see below).

==================}}}

> Specs has a few features that ScalaTest doesn't have. The ->- operator
for example is used to bring in shared behavior, I think. (I may be
confused as to how that works.) In ScalaTest, you can bring in shared
behavior, but there's no special syntax sugar right now. (Actually a
slicker "should behave like" syntax is private in Should/MustMatchers
in 0.9.5, because I need to split it off into another trait. So that
is something that will come out in a future release.) Specs has an
"aka" operator which I think is very well named. I didn't put anything
like that into ScalaTest because I'm not sure it is needed. Users
could convince me, but so far I've found most objects in Scala have
pretty good toString messages. Specs also has a data table feature,
which ScalaTest doesn't have. I frankly in all my years of testing
have never really needed that. So unless lots of users scream that it
is useful it probably won't go in, and besides, I think a ScalaCheck
property check would in most cases be better. However if users want it
they will get it.

{{{==================

The DataTables feature started from a perceived need of filling in the gap
of creating lots of small examples and using ScalaCheck. It also started as
a test of how far I could go with Scala syntax: can we embedded
statically-type tables in the code?!

I've used it for example in the xmlRunner unit tests, as I wanted to specify
a few possible combinations:

"create an xml file in the specified output directory, handling file
separators" in {
"output dir" | "spec name" | "file path" |>
"" ! "spec1" ! "./spec1.xml" |
"result" ! "spec1" ! "./result/spec1.xml" |
"result/" ! "spec1" ! "./result/spec1.xml" |
"result\\" ! "spec1" ! "./result/spec1.xml" |
"/result" ! "spec1" ! "/result/spec1.xml" |
"\\result" ! "spec1" ! "/result/spec1.xml" |
"result/xml" ! "spec1" ! "./result/xml/spec1.xml" | { (dir,
spec, result) =>
xRunner.reset
xRunner.setOutputDir(dir)
spec1.name = spec
xRunner.reportSpecs
xRunner.files.keySet must contain(result)
}
}

The syntax is both concise and statically typed (the types of the elements
in the first row after the header determine the types of the other rows).

Of course, this is a very specific usage so the DataTables trait is not
provided by default in a Specification.

==================}}}

> Specs also has a lightweight mocking library, and
some syntax sugar for JMock. In either ScalaTest or Specs you can use
any of the Java mocking libraries: Mockito, JMock, EasyMock,
PowerMock, etc. I don't currently offer any sugar for any of these in
ScalaTest. Sugar is something that can be added, potentially, but
currently all these mocking frameworks work just fine in ScalaTest by
just importing and using them. But that's also a difference. By the
way I have found that Scala is so concise that often I just write
mocks by hand in Scala.

{{{==================

The next version will have an integration with Mockito (just finished
it,...). It's not a big thing against the direct use of the library but it
is "integrated" and has additional syntactic sugar:

object spec extends Specification with Mockito {

val m = mock[java.util.List[String]] // a concrete class would be mocked
with: mock(new java.util.LinkedList[String])

// stub a method call with a return value
m.get(0) returns "one"

// call the method
m.get(0)

// verify that the call happened, this is an expectation which will
throw a FailureException if that is not the case
m.get(0) was called
m.get(1) wasnt called // we can also check that another call did not
occur
}

==================}}}

> One other difference is that I do try and make it just slightly easier
in ScalaTest to use a functional style when you need to share a
fixture, whereas Specs by default continues the tradition of the setUp
and tearDown approach popularized by JUnit, in which you reassign vars
before each test. However if you want to test that way, it is also
very easy in ScalaTest. You just need to mix in the BeforeAndAfter
trait.

{{{==================

One thing I like in specs is that the creation of before / after methods is
very simple:

object s extends Specification {

"my system" should {
before { doThis() } // will be executed before each example
"have one example" in { ... }
}
}

However this has all the drawbacks of using mutable and shared variables. If
you want more encapsulation you can use "System contexts". Here is a full
example:

object systemcontext extends Specification {

/** Computer and Order definitions removed
*/

// this is my system
case class ComputerOrder(order: Order)

// this is one of the contexts (a "fixture")
def laptopOrder = new SystemContext[ComputerOrder] {
def newSystem = ComputerOrder(Order(Laptop()))
}
// this is another contexts
def desktopOrder = new SystemContext[ComputerOrder] {
def newSystem = ComputerOrder(Order(Desktop()))
}

// this system under specification is defined as being in one context
// the system can be passed to each example. It is setup with the
newSystem method
"A laptop order".definedAs(laptopOrder) should {
"have a mini mouse when assembled" in { order: ComputerOrder =>
order.assemble.accessories must contain(MiniMouse())
}
"have a case when assembled" in { order: ComputerOrder =>
order.assemble.accessories must contain(Case())
}
}

// this system under specification is defined as being in another context
"A desktop order".definedAs(desktopOrder) should {
"have a printer when assembled" in { order: ComputerOrder =>
order.assemble.accessories must contain(Printer())
}
"have a webcam when assembled" in { order: ComputerOrder =>
order.assemble.accessories must contain(Webcam())
}
}

}

==================}}}

> Lastly, ScalaTest has a GUI, which as of 0.9.5 displays specification
like output. Specs does not have a GUI, though with a bit of extra
code you can use JUnit's GUI.

{{{==================

Yes, "with JUnit" ;-) (in 1.4.4-SNAPSHOT,...)

==================}}}

> However our plan is that ScalaTest will
support Specs very well. What Eric and I have been working on together
is that when Specs is run through ScalaTest, you will get the nice
Spec-like GUI output. Basically my plan has been to support
ScalaCheck, Specs, JUnit, and TestNG as first-class citizens in
ScalaTest. Eric and I have already done some of that integration, and
have more work to do. But in the next release or two of both tools I
expect we'll have it all done.

[From Bill's last post]
> One other difference that occurred to me after sending my long, wordy
email was that ScalaTest has a PrivateMethodTester trait that
facilitates testing private methods. I don't think Specs has anything
like this, but if you want you can just mix ScalaTest's
PrivateMethodTester trait into your Specs Specification and it should
work just fine.

{{{==================

I was planning of using Mockito and Powermock more generally to do that.

==================}}}

> So those are some differences to consider. Perhaps Eric can add more
> insight.

{{{==================

Well I tried to incorporate as much feedback as possible in specs (but I'm
still resisting changing the packaging to something longer to type than
org.specs._ ;-) ).

Yet I have no doubt that some features could be dropped or reworked. For
example I provided a light-weight Mock library which I feel won't be
necessary at all when Mockito and Powermock will be available. The only
"interesting" feature was a way to declare "protocols" to check that an API
was called with the proper sequence of events:

val protocol = expect(inAnyOrder) {
expect(oneOf){mock.on; mock.off; mock.on}
expect(oneOf){mock.off}
}

I also added some features that I needed from time to time:

- detailedDiffs() to highlight the differences when 2 strings are not
matching (that one is very useful!)

- tags: you can tags systems and examples to get a more "tranversal" view of
your specifications and execute only some tags while excluding others

- matchers can be composed with functions to create matchers for object
graphs:
http://code.google.com/p/specs/wiki/MatchersGuide#Matchers_composition_f...

- literate specs. That's the next big thing I'm working on. I would like to
produce nice readable executable specifications, as a replacement for
Fitnesse. The basic idea to avoid the separation between text and code that
you have in Fitnesse (and use scala xml + embedded code + wiki markup for
that). The other idea is to think of a specification as some text where you
would highlight some fragments that you want as "examples" and that you back
up with executable expectations. When the specification is executed the test
is hightlighted in green if the example passes and in yellow if it doesn't
(with the error message as a pop-up).

I'm not releasing this feature yet as there are many rough edges and more
experimentation needs to be done before you can use it on "entreprise"
specifications. So I need to eat my own dog food first,... Still, I attach
an example of the Mockito specification I recently created (which also uses
Snippets, a trait allowing to capture and execute Scala code snippets) so
everyone can have a peek at it.

==================}}}

Eric.

http://www.nabble.com/file/p22592560/literateSpecs%2Bexample.zip
literateSpecs+example.zip

Hi Eric,

On Wed, Mar 18, 2009 at 7:10 PM, Eric Torreborre wrote:
>
>
>> The syntax of Spec in ScalaTest looks more like Ruby's RSpec, using
> "describe" to specify the system under test (SUT) and "it" to specify
> an example. Specs Specification class declares implicit conversions on
> string to specify the SUT and examples.
>
> {{{==================
>
> There are indeed implicit conversions because I felt that using "describe"
> everywhere was just adding unnecessary redundancy in the specs. However, you
> can still use the implicit def:
>
> specify("my system") should {
>  forExample("doThis") in {
>    ...
>  }
> }
>
> ==================}}}
>
That's interesting. I hasn't thought of that syntax as a way around
the implicit conversions because the implicits are still in scope even
if you use specify and forExample. In ScalaTest, actually you could
get rid of the "describe" and "it" by mixing in a trait that adds
implicit conversions on strings. So you could use Specs style syntax
in a ScalaTest Spec, for example, by mixing in something like:

trait SpecsStyle { this: Spec =>

class Wrapper(text: String) {
def should(descFun: => Unit) {
describe(text)(descFun)
}
def in(testFun: => Unit) {
it(text)(testFun)
}
}

implicit def stringToWrapper(s: String) = new Wrapper(s)
}

This way the implicits actually need to be invited into your code by
mixing in the SpecsStyle trait. If you don't mix it in, you don't get
them at all.

>> One difference here is that
> you can nest describe clauses in a ScalaTest Spec, as you can in
> RSpec, but I don't think that's possible in Specs (Eric, can you
> verify this?)
>
> {{{==================
>
> In specs you can compose specifications and you can nest Examples
> (subExamples). I think that does the trick in terms of nesting (but feedback
> is welcome to show me otherwise, especially from long-time rspec
> afficionados).
>
> ==================}}}
>
I looked into subexamples. They seem useful, but are different than
nested describe clauses if I understand subexamples correctly.
Basically what you can do in ScalaTest (and RSpec) is something like
this:

describe("A Stack") {
describe("(when empty)" {
it("should complain when popped") {}
// more examples for an empty stack
}
describe("(when full)") {
it("should complain when pushed") {}
// more examples for a full stack
}
}

Then when this executes you'll see:

A Stack (when empty)
- should complain when popped
...
A Stack (when full)
- should complain when pushed
...

Anyway this is a minor difference.

>> On that subject, I think implicit conversions constitutes a general
> design attitude difference between Specs and ScalaTest. By default you
> get only one implicit conversion when you use ScalaTest, which is the
> one that puts the === operator on everything. (If you need to, you can
> "turn off" this implicit conversion with one line of code. The only
> reason you would need to do that is if you were trying to test
> something that has its own === operator, and you get a conflict.)
> ScalaTest defines many other implicit conversions, but to use them you
> need to explicitly "invite" them into your code by mixing in a trait
> or doing an import. When you extend class Specification in Specs I
> think you pretty much get dozens of implicit conversion by default.
> I'm not sure how much that will matter in practice, but I figure
> people will want to test code that uses their own implicits, and
> sometimes there may be a conflict between the test framwork's
> implicits and those of the production code. When that happens I think
> it may be easier to work around the problem in ScalaTest than Specs.
>
> {{{==================
>
> Yes, that is a difference in the sense that I want in specs to have
> everything at hand for usual specifications with the simplest declaration.
> So, when you extends the Specification class, you get a lot of additional
> traits.
>
I actually don't expect folks will mix everything together every time.
More likely people will make a Suite or a Spec trait that has
everything they like, or perhaps has the stuff the team agreed on, and
then everyone will just use that. If you worked at Acme corp on
project "X", you might write something like:

package com.acme.projectx.testing

import org.scalatest._
import org.scalatest.matchers._

// The chosen style for tests for project X
trait XSpec extends Spec with ShouldMatchers with PrivateMethodTester
with BeforeAndAfter

Then in their tests they would just say:

import com.acme.projectx.testing.XSpec

class MySpec extends XSpec {
// ...
}

> However, in the case of a conflict I'm assuming that you can remove/override
> some traits and compose your own "Specification class". I haven't found such
> issues, but that doesn't prove anything!
>
I doubt it would happen very often, but I wanted to minimize the
number of implicits in scope by default to make it easier on people if
it does happen. And besides, implicits are not explicit. I think it is
good to minimize their use in general.

> ==================}}}
>
>> Another difference in design attitude that I've noticed is comfort
> with operators. One goal I had was that any programmer looking at
> someone else's test code that uses ScalaTest would be able to guess
> what the meaning was without looking anything up in the ScalaTest
> documentation. I wanted ScalaTest client code to be drop dead obvious.
> One way that goal manifested itself is that ScalaTest is very
> conservative about operators. I only define five operators in
> ScalaTest: ===, which means equals, >, which means greater than, <,
> less than, >=, greater than or equal, and <=, less than or equal.
> That's it. So these things pretty much look like what mean. If you see
> in someone else's code:
>
>> result should be <= 7
>
>> My hope is that you won't need to run to the API documentation to
> guess what that <= means. By contrast, Eric has been much freer with
> operators in Specs. Nothing wrong with that, but it is a difference.
> Operators can make code more concise, but the tradeoff is you may have
> to run to the documentation when you find things like  ->-, >>, |, |>,
> !, or ^^^ (which all have special meanings in Specs) in your
> colleague's test code.
>
> {{{==================
>
> True! I've been much more liberal with operators. However the operators
> you're mentioning are used in very specific (you could say "advanced")
> situations.
>
> For example ->- is used to share "Contexts" (which are a common set of
> setUp/tearDown operations to setup the context of a system under
> specification).
>
> Similarly "|" and "|>" are used when specifying DataTables (see below).
>
While that may be true, still if I run into someone else's code that's
using ->-, I'll have to look it up. That's the tradeoff with
operators.

>
>> So those are some differences to consider. Perhaps Eric can add more
>> insight.
>
> {{{==================
>
> Well I tried to incorporate as much feedback as possible in specs (but I'm
> still resisting changing the packaging to something longer to type than
> org.specs._ ;-) ).
>
> Yet I have no doubt that some features could be dropped or reworked. For
> example I provided a light-weight Mock library which I feel won't be
> necessary at all when Mockito and Powermock will be available. The only
> "interesting" feature was a way to declare "protocols" to check that an API
> was called with the proper sequence of events:
>
>  val protocol = expect(inAnyOrder) {
>    expect(oneOf){mock.on; mock.off; mock.on}
>    expect(oneOf){mock.off}
>  }
>
> I also added some features that I needed from time to time:
>
> - detailedDiffs() to highlight the differences when 2 strings are not
> matching (that one is very useful!)
>
ScalaTest does this as well.

> - tags: you can tags systems and examples to get a more "tranversal" view of
> your specifications and execute only some tags while excluding others
>
ScalaTest does this as well too. Though it is called groups, as it was
called in TestNG. Oddly enough this is one of the features I have
wondered if I should have cut. I never used it, but I saw it in TestNG
and thought it was a good idea. Regardless it's in. Though I think I
like the name "tags" better. I noticed RSpec's Cucumber project just
released a new version this week with the feature, and they call it
tags too. I may change the name.

> - matchers can be composed with functions to create matchers for object
> graphs:
> http://code.google.com/p/specs/wiki/MatchersGuide#Matchers_composition_f...
>
I never could understand this. Could you explain how this works?

Thanks.

Bill

Bill Venners schrieb:
> Lastly, ScalaTest has a GUI, which as of 0.9.5 displays specification
> like output.

What I am really missing in ScalaTest (as in right now, five minutes
before writing this mail) is working ant integration. I can call ScalaTest
from ant, but it will always tell ant that everything is fine, even if it
can't find any test suite at all or all tests fail.
What I'd love to see for the GUI if called from ant would be to open, display
the green bar and close itself again if no test failed. That would give
immediate, unobtrusive feedback.

Right now my options are:
* Use the graphic reporter, then I will always see test failures, but have to
perform additional mouse clicks to complete every single build, even if
no tests fail.
* Use the file reporter. Then I will never see any test failures at all,
unless I happen look into the log file
* Use the stdout or stderr reporter. Then I will see failures, but the whole
build process output is swamped in test log output even if no test fails.
And the failures are far down in the *compilation* buffer with the exit status
signaling no error even in the presence of test failures.

- Florian

Hi Florian,

On Thu, Mar 19, 2009 at 1:13 AM, Florian Hars wrote:
> Bill Venners schrieb:
>> Lastly, ScalaTest has a GUI, which as of 0.9.5 displays specification
>> like output.
>
> What I am really missing in ScalaTest (as in right now, five minutes
> before writing this mail) is working ant integration. I can call ScalaTest
> from ant, but it will always tell ant that everything is fine, even if it
> can't find any test suite at all or all tests fail.
> What I'd love to see for the GUI if called from ant would be to open, display
> the green bar and close itself again if no test failed. That would give
> immediate, unobtrusive feedback.
>
It is actually already on our list to give you a way to say in the ant
task to fail the build if a test fails. We'll try to get that done for
the next release. I'm not sure about killing the GUI if a test fails
if called from Ant. A better option is probably to configure the
standard out reporter to only show failures, which I'll explain below.

> Right now my options are:
> * Use the graphic reporter, then I will always see test failures, but have to
>  perform additional mouse clicks to complete every single build, even if
>  no tests fail.
> * Use the file reporter. Then I will never see any test failures at all,
>  unless I happen look into the log file
> * Use the stdout or stderr reporter. Then I will see failures, but the whole
>  build process output is swamped in test log output even if no test fails.
>  And the failures are far down in the *compilation* buffer with the exit status
>  signaling no error even in the presence of test failures.
>
You can configure what reports are shown by a reporter. So instead of
-o, which says you want a standard out reporter on the Runner command
line, you would say -oYFABR. The meaning of all the config params is
explained in the Runner scaladocs:

http://www.artima.com/scalatest/doc-0.9.5/org/scalatest/tools/Runner$object.html

The ones I suggest you use mean:

Y - show run starting reports
F - show test failed reports
A - show run aborted reports
B - show suite aborted reports
R - show run completed reports

I suspect that will give you just the info you want without any
clutter you don't. In the Ant task you say:

...

Sorry this repoprter configuration feature was a bit buried in the
documentation.

Bill

Bill Venners schrieb:
> I'm not sure about killing the GUI if a test fails if called from Ant.

No, I want it the other way round: keep the GUI open if there is a failure,
but close it automatically if there is not failure if called from ant, so I can
just admire my green bar without having to explicitly interact with the GUI.

>

Yes, thanks, that is better.

But there seems to be a counting and buffer flushing issue left:

- test is executed as a dependency of jar
- build.xml contains:

- Console output:
| test:
| [scalatest] Run starting. Expected test count is: 27
|
| jar:
| [jar] Run completed. Total number of tests run was: 0
| [jar] All tests passed.
- tail -3 test.out
| Suite Completed - TestSuite: The execute method of a nested suite returned normally.
| Run completed. Total number of tests run was: 27
| All tests passed.

- Florian.

Hi Florian,

On Thu, Mar 19, 2009 at 2:18 AM, Florian Hars wrote:
> Bill Venners schrieb:
>> I'm not sure about killing the GUI if a test fails if called from Ant.
>
> No, I want it the other way round: keep the GUI open if there is a failure,
> but close it automatically if there is not failure if called from ant, so I can
> just admire my green bar without having to explicitly interact with the GUI.
>
OK. This does make sense as a usage mode, but if we add it it would be
an option in the ant task. It would make sense to do this at the same
time we add an option to kill the build if there's a test failure.

>>    
>
> Yes, thanks, that is better.
>
> But there seems to be a counting and buffer flushing issue left:
>
> - test is executed as a dependency of jar
> - build.xml contains:
>      
>      
> - Console output:
>  | test:
>  | [scalatest] Run starting. Expected test count is: 27
>  |
>  | jar:
>  |       [jar] Run completed. Total number of tests run was: 0
>  |       [jar] All tests passed.
> - tail -3 test.out
>  | Suite Completed - TestSuite: The execute method of a nested suite returned normally.
>  | Run completed. Total number of tests run was: 27
>  | All tests passed.
>
Ah, that's a bug. I'll add fixing that to the list. Where it came from
is that back in SuiteRunner, the Java tool from which ScalaTest
sprang, you could configure reporters directly. This was implemented
as a "setConfiguration" method, which means that any reporter you
create would by definition be a mutable object. So in ScalaTest I
dropped that method and just filter the reports out before they get to
the reporter, so reporters can be immutable. But the PrintReporters
still count the number of tests themselves, which worked fine when
reporters were doing their own filtering. But in 0.9.5 you'll need to
include testSucceeded reports to get an accurate count at the end
(config="YFABRT"), which moves you back to a bit more verbosity in
your report. We'll fix that for the next release.

Bill

On Wednesday March 18 2009, Randall R Schulz wrote:
> Hi,
>
> Is there currently a clearly superior choice for a unit testing
> framework for Scala? If so, what is it? If not, what are the
> tradeoffs to consider?

Thanks to everyone who replied for such a great discussion.
I've only skimmed so far, but will re-read it carefully.

Randall Schulz

2009/3/18 Bill Venners <bill@artima.com>

<snip/>

, but in
ScalaTest, you could just use a dynamic check like this:

file must be a ('directory)

Anytime you pass a symbol in after "be", it will use reflection to
look for (in this case) a method or field named "directory" or a
method named "isDirectory". There's also a way to make this static, by
defining a BePropertyMatcher (which requires only 2 or 3 lines of code
usually). So basically in ScalaTest I try to provide more
functionality with less API.

Hi,

I never used any of these frameworks but I'd rather not use this feature in my tests. Reflection defeats IDE's and refactoring tools. If somebody modifies the name of a field he can't see immediately what he breaks until he runs the test which may be a while later. It will become quickly tedious if I have to write 2 or 3 lines of wrapper code for each field.

I understand assert(file.isDirectory) is a bit dull but it is very straightforward for a programmer. What is the advantage of using a DSL in this case?

Thanks,
Sebastien

Hi Sébastien,

On Mon, Apr 6, 2009 at 3:32 AM, Sébastien Bocq wrote:
>
> 2009/3/18 Bill Venners
>
>>
>>
>> , but in
>> ScalaTest, you could just use a dynamic check like this:
>>
>> file must be a ('directory)
>>
>> Anytime you pass a symbol in after "be", it will use reflection to
>> look for (in this case) a method or field named "directory" or a
>> method named "isDirectory". There's also a way to make this static, by
>> defining a BePropertyMatcher (which requires only 2 or 3 lines of code
>> usually). So basically in ScalaTest I try to provide more
>> functionality with less API.
>
> Hi,
>
> I never used any of these frameworks but I'd rather not use this feature in
> my tests. Reflection defeats IDE's and refactoring tools. If somebody
> modifies the name of a field he can't see immediately what he breaks until
> he runs the test which may be a while later. It will become quickly tedious
> if I have to write 2 or 3 lines of wrapper code for each field.
>
That's reasonable.

> I understand assert(file.isDirectory) is a bit dull but it is very
> straightforward for a programmer. What is the advantage of using a DSL in
> this case?
>
The matchers DSL is for people who prefer the DSL, and I've met a lot
of people who do. I often just use asserts myself, but I also use the
DSL sometimes. Moreover, the DSL isn't only dynamic. There's also a
static version of that same syntax that would give you the refactoring
you (and I) like, with the cost of only a couple more lines of code.
But those lines of code could be easily automatically generated as
well, so you may not have to write them. Then it would look like:

file must be a (directory)

All statically checked. It's called a BePropertyMatcher, and is documented here:

http://www.artima.com/scalatest/doc-0.9.5/org/scalatest/matchers/BePrope...

In ScalaTest, if you don't like matchers, you simply don't mix a
matchers trait into your test classes. By default, ScalaTest gives you
this minimal, easy to learn, easy to read set of assertions:

assert(a === b)

expect(3) {
1 + 2
}

intercept[StringIndexOutOfBoundsException] {
"hi".charAt(-1)
}

Bill

Hey Bill.
I think you are doing a fantastic job with ScalaTest.
0.9.5 looks very very good. I love the BDD support.
Thanks, Jonas

2009/4/6 Bill Venners :
> Hi Sébastien,
>
> On Mon, Apr 6, 2009 at 3:32 AM, Sébastien Bocq wrote:
>>
>> 2009/3/18 Bill Venners
>>
>>>
>>>
>>> , but in
>>> ScalaTest, you could just use a dynamic check like this:
>>>
>>> file must be a ('directory)
>>>
>>> Anytime you pass a symbol in after "be", it will use reflection to
>>> look for (in this case) a method or field named "directory" or a
>>> method named "isDirectory". There's also a way to make this static, by
>>> defining a BePropertyMatcher (which requires only 2 or 3 lines of code
>>> usually). So basically in ScalaTest I try to provide more
>>> functionality with less API.
>>
>> Hi,
>>
>> I never used any of these frameworks but I'd rather not use this feature in
>> my tests. Reflection defeats IDE's and refactoring tools. If somebody
>> modifies the name of a field he can't see immediately what he breaks until
>> he runs the test which may be a while later. It will become quickly tedious
>> if I have to write 2 or 3 lines of wrapper code for each field.
>>
> That's reasonable.
>
>> I understand assert(file.isDirectory) is a bit dull but it is very
>> straightforward for a programmer. What is the advantage of using a DSL in
>> this case?
>>
> The matchers DSL is for people who prefer the DSL, and I've met a lot
> of people who do. I often just use asserts myself, but I also use the
> DSL sometimes. Moreover, the DSL isn't only dynamic. There's also a
> static version of that same syntax that would give you the refactoring
> you (and I) like, with the cost of only a couple more lines of code.
> But those lines of code could be easily automatically generated as
> well, so you may not have to write them. Then it would look like:
>
> file must be a (directory)
>
> All statically checked. It's called a BePropertyMatcher, and is documented here:
>
> http://www.artima.com/scalatest/doc-0.9.5/org/scalatest/matchers/BePrope...
>
> In ScalaTest, if you don't like matchers, you simply don't mix a
> matchers trait into your test classes. By default, ScalaTest gives you
> this minimal, easy to learn, easy to read set of assertions:
>
> assert(a === b)
>
> expect(3) {
>  1 + 2
> }
>
> intercept[StringIndexOutOfBoundsException] {
>  "hi".charAt(-1)
> }
>
> Bill
>

Sorry for the late reply. I like very much the simple assertions! This way of testing is still unknown territory for me but I feel reassured now that I know the good old assert is still there :)

Thanks for your time,
Sebastien

2009/4/9 Jonas Bonér <lists@jonasboner.com>

Hey Bill.
I think you are doing a fantastic job with ScalaTest.
0.9.5 looks very very good. I love the BDD support.
Thanks, Jonas

2009/4/6 Bill Venners <bill@artima.com>:
> Hi Sébastien,
>
> On Mon, Apr 6, 2009 at 3:32 AM, Sébastien Bocq <sebastien.bocq@gmail.com> wrote:
>>
>> 2009/3/18 Bill Venners <bill@artima.com>
>>
>>> <snip/>
>>>
>>> , but in
>>> ScalaTest, you could just use a dynamic check like this:
>>>
>>> file must be a ('directory)
>>>
>>> Anytime you pass a symbol in after "be", it will use reflection to
>>> look for (in this case) a method or field named "directory" or a
>>> method named "isDirectory". There's also a way to make this static, by
>>> defining a BePropertyMatcher (which requires only 2 or 3 lines of code
>>> usually). So basically in ScalaTest I try to provide more
>>> functionality with less API.
>>
>> Hi,
>>
>> I never used any of these frameworks but I'd rather not use this feature in
>> my tests. Reflection defeats IDE's and refactoring tools. If somebody
>> modifies the name of a field he can't see immediately what he breaks until
>> he runs the test which may be a while later. It will become quickly tedious
>> if I have to write 2 or 3 lines of wrapper code for each field.
>>
> That's reasonable.
>
>> I understand assert(file.isDirectory) is a bit dull but it is very
>> straightforward for a programmer. What is the advantage of using a DSL in
>> this case?
>>
> The matchers DSL is for people who prefer the DSL, and I've met a lot
> of people who do. I often just use asserts myself, but I also use the
> DSL sometimes. Moreover, the DSL isn't only dynamic. There's also a
> static version of that same syntax that would give you the refactoring
> you (and I) like, with the cost of only a couple more lines of code.
> But those lines of code could be easily automatically generated as
> well, so you may not have to write them. Then it would look like:
>
> file must be a (directory)
>
> All statically checked. It's called a BePropertyMatcher, and is documented here:
>
> http://www.artima.com/scalatest/doc-0.9.5/org/scalatest/matchers/BePropertyMatcher.html
>
> In ScalaTest, if you don't like matchers, you simply don't mix a
> matchers trait into your test classes. By default, ScalaTest gives you
> this minimal, easy to learn, easy to read set of assertions:
>
> assert(a === b)
>
> expect(3) {
>  1 + 2
> }
>
> intercept[StringIndexOutOfBoundsException] {
>  "hi".charAt(-1)
> }
>
> Bill
>



--
Jonas Bonér | Crisp AB

http://jonasboner.com
http://crisp.se

> I feel reassured now that I know the good old assert is still there :)

This is why there is also a xUnit trait in specs :-) :

http://code.google.com/p/specs/wiki/MatchersGuide#xUnit_assertions

Eric.