Re: Re : Scala Testing Frameworks

Hi Eric,

On Wed, Mar 18, 2009 at 9:28 PM, Eric Torreborre wrote:
> Hi Bill,
>>> - detailedDiffs() to highlight the differences when 2 strings are not
>>> matching (that one is very useful!)
>>>
>> ScalaTest does this as well.
>
> Yes, but are you using the Levenshtein distance? ;-). Joke aside that was an
> interesting coding exercise.
>
No, I just use the Venners distance. It wasn't as interesting to
implement but seems to work OK.

>> ScalaTest does this as well too. Though it is called groups, as it was
> called in TestNG. Oddly enough this is one of the features I have
> wondered if I should have cut. I never used it, but I saw it in TestNG
> and thought it was a good idea.
>
> I actually have the same doubts as you. I've been using tags to run only
> some examples until I added some command line options to run examples based
> on their names (in the next version). I also used TestNG groups in my
> previous company and found them to add a supplementary maintainance burden.
>
I also use groups to implement the ScalaTest's ignore feature. So
"ignore" is like a group or tag in ScalaTest, which is excluded from
execution by default. That's one thing that tipped the balance for me,
because ignore is a feature that gets used a lot, and groups was a
generalization of that.

>>> - matchers can be composed with functions to create matchers for object
>> graphs:
>> http://code.google.com/p/specs/wiki/MatchersGuide#Matchers_composition_for_object_graphs
>>
>> I never could understand this. Could you explain how this works?
>
> I thought the example was self-explanatory. Obviously not ;-)
> The idea is the following. You have an object of type Foo, referencing
> another Bar object:
>
Well although I think I get the general concept, I can't quite grok
the details yet. Can you perhaps explain what this expression does:

((beEqual(_:Long)) ^^^ ((b: Bar) => b.id)))

It looks like you're ^^^-ing a partially applied function (which is a
matcher) with a function that transforms a Bar to its id, which is
probably of the type needed by the partially applied function. Maybe
I'm figuring this out. So what results from ^^^ is a matcher that
takes a Bar perhaps and matches if its ID is equal to the passed Bar's
id field? Is that correct?

Bill

> case class Foo(val name:String) {
>     var singlebar: Bar = _
>     val bars = mutable.Set[Bar]()
>   }
>
>   case class Bar(val id: Long)
>
> You want to be able to create a Matcher for Foo object which will precisely
> tell you when 2 Foo objects don't match. And you want to do this by reusing
> as much as possible the existing matchers you have at hand. So you create
> aMatcher[Foo]:
>
> case class matchFoo(foo: Foo) extends Matcher[Foo] {
>     def apply(other: => Foo) = {
>       ((beEqual(_:String)) ^^^ ((f:Foo) => f.name) and
>        (matchBar(_)) ^^^ ((f: Foo) => foo.singlebar) and
>        (matchBar(_)).toSet ^^^ ((f: Foo) => foo.bars))(foo)(other)
>     }
>   }
>
> That matcher says that 2 Foo objects are matching if:
> 1. their name is matching
> 2. their singlebar attribute is matching
> 3. their bars attributes is matching
> Let's examine point 1. If you want to check that their name is matching by
> reusing the beEqual matcher you need to compose that matcher with a function
> extracting the name attribute from a Foo object.
> We're doing the same thing for the other Foo attributes, with the additional
> transformation of a Bar matcher to a Matcher of Set of Bars with the toSet()
> method for the Foo.bars attribute.
> So you can construct, from a complex graph of objects, a corresponding graph
> of Matchers reusing as many matchers as possible.
> This stuff arose from the needs of a regular project but I don't know how
> far they went with this.
> Eric.
> ---------------------------------------------------------------------------
> Eric TORREBORRE
> tel: +81 (0)90 5580 3280
> e-mail: etorreborre@yahoo.com / etorreborre@docomo.ne.jp
> blog: http://etorreborre.blogspot.com
> oss project: http://code.google.com/p/specs
> ---------------------------------------------------------------------------
>
> ________________________________
> De : Bill Venners
> À : Eric Torreborre
> Cc : scala-user@listes.epfl.ch
> Envoyé le : Jeudi, 19 Mars 2009, 12h39mn 58s
> Objet : Re: [scala-user] Scala Testing Frameworks
>
> Hi Eric,
>
> On Wed, Mar 18, 2009 at 7:10 PM, Eric Torreborre
> wrote:
>>
>>
>>> The syntax of Spec in ScalaTest looks more like Ruby's RSpec, using
>> "describe" to specify the system under test (SUT) and "it" to specify
>> an example. Specs Specification class declares implicit conversions on
>> string to specify the SUT and examples.
>>
>> {{{==================
>>
>> There are indeed implicit conversions because I felt that using "describe"
>> everywhere was just adding unnecessary redundancy in the specs. However,
>> you
>> can still use the implicit def:
>>
>> specify("my system") should {
>>  forExample("doThis") in {
>>    ...
>>  }
>> }
>>
>> ==================}}}
>>
> That's interesting. I hasn't thought of that syntax as a way around
> the implicit conversions because the implicits are still in scope even
> if you use specify and forExample. In ScalaTest, actually you could
> get rid of the "describe" and "it" by mixing in a trait that adds
> implicit conversions on strings. So you could use Specs style syntax
> in a ScalaTest Spec, for example, by mixing in something like:
>
> trait SpecsStyle { this: Spec =>
>
>   class Wrapper(text: String) {
>     def should(descFun: => Unit) {
>       describe(text)(descFun)
>     }
>     def in(testFun: => Unit) {
>       it(text)(testFun)
>     }
>   }
>
>   implicit def stringToWrapper(s: String) = new Wrapper(s)
> }
>
> This way the implicits actually need to be invited into your code by
> mixing in the SpecsStyle trait. If you don't mix it in, you don't get
> them at all.
>
>>> One difference here is that
>> you can nest describe clauses in a ScalaTest Spec, as you can in
>> RSpec, but I don't think that's possible in Specs (Eric, can you
>> verify this?)
>>
>> {{{==================
>>
>> In specs you can compose specifications and you can nest Examples
>> (subExamples). I think that does the trick in terms of nesting (but
>> feedback
>> is welcome to show me otherwise, especially from long-time rspec
>> afficionados).
>>
>> ==================}}}
>>
> I looked into subexamples. They seem useful, but are different than
> nested describe clauses if I understand subexamples correctly.
> Basically what you can do in ScalaTest (and RSpec) is something like
> this:
>
> describe("A Stack") {
>   describe("(when empty)" {
>     it("should complain when popped") {}
>     // more examples for an empty stack
>   }
>   describe("(when full)") {
>     it("should complain when pushed") {}
>     // more examples for a full stack
>   }
> }
>
> Then when this executes you'll see:
>
> A Stack (when empty)
> - should complain when popped
> ...
> A Stack (when full)
> - should complain when pushed
> ...
>
> Anyway this is a minor difference.
>
>>> On that subject, I think implicit conversions constitutes a general
>> design attitude difference between Specs and ScalaTest. By default you
>> get only one implicit conversion when you use ScalaTest, which is the
>> one that puts the === operator on everything. (If you need to, you can
>> "turn off" this implicit conversion with one line of code. The only
>> reason you would need to do that is if you were trying to test
>> something that has its own === operator, and you get a conflict.)
>> ScalaTest defines many other implicit conversions, but to use them you
>> need to explicitly "invite" them into your code by mixing in a trait
>> or doing an import. When you extend class Specification in Specs I
>> think you pretty much get dozens of implicit conversion by default.
>> I'm not sure how much that will matter in practice, but I figure
>> people will want to test code that uses their own implicits, and
>> sometimes there may be a conflict between the test framwork's
>> implicits and those of the production code. When that happens I think
>> it may be easier to work around the problem in ScalaTest than Specs.
>>
>> {{{==================
>>
>> Yes, that is a difference in the sense that I want in specs to have
>> everything at hand for usual specifications with the simplest declaration.
>> So, when you extends the Specification class, you get a lot of additional
>> traits.
>>
> I actually don't expect folks will mix everything together every time.
> More likely people will make a Suite or a Spec trait that has
> everything they like, or perhaps has the stuff the team agreed on, and
> then everyone will just use that. If you worked at Acme corp on
> project "X", you might write something like:
>
> package com.acme.projectx.testing
>
> import org.scalatest._
> import org.scalatest.matchers._
>
> // The chosen style for tests for project X
> trait XSpec extends Spec with ShouldMatchers with PrivateMethodTester
> with BeforeAndAfter
>
> Then in their tests they would just say:
>
> import com.acme.projectx.testing.XSpec
>
> class MySpec extends XSpec {
>   // ...
> }
>
>> However, in the case of a conflict I'm assuming that you can
>> remove/override
>> some traits and compose your own "Specification class". I haven't found
>> such
>> issues, but that doesn't prove anything!
>>
> I doubt it would happen very often, but I wanted to minimize the
> number of implicits in scope by default to make it easier on people if
> it does happen. And besides, implicits are not explicit. I think it is
> good to minimize their use in general.
>
>> ==================}}}
>>
>>> Another difference in design attitude that I've noticed is comfort
>> with operators. One goal I had was that any programmer looking at
>> someone else's test code that uses ScalaTest would be able to guess
>> what the meaning was without looking anything up in the ScalaTest
>> documentation. I wanted ScalaTest client code to be drop dead obvious.
>> One way that goal manifested itself is that ScalaTest is very
>> conservative about operators. I only define five operators in
>> ScalaTest: ===, which means equals, >, which means greater than, <,
>> less than, >=, greater than or equal, and <=, less than or equal.
>> That's it. So these things pretty much look like what mean. If you see
>> in someone else's code:
>>
>>> result should be <= 7
>>
>>> My hope is that you won't need to run to the API documentation to
>> guess what that <= means. By contrast, Eric has been much freer with
>> operators in Specs. Nothing wrong with that, but it is a difference.
>> Operators can make code more concise, but the tradeoff is you may have
>> to run to the documentation when you find things like  ->-, >>, |, |>,
>> !, or ^^^ (which all have special meanings in Specs) in your
>> colleague's test code.
>>
>> {{{==================
>>
>> True! I've been much more liberal with operators. However the operators
>> you're mentioning are used in very specific (you could say "advanced")
>> situations.
>>
>> For example ->- is used to share "Contexts" (which are a common set of
>> setUp/tearDown operations to setup the context of a system under
>> specification).
>>
>> Similarly "|" and "|>" are used when specifying DataTables (see below).
>>
> While that may be true, still if I run into someone else's code that's
> using ->-, I'll have to look it up. That's the tradeoff with
> operators.
>
>>
>>> So those are some differences to consider. Perhaps Eric can add more
>>> insight.
>>
>> {{{==================
>>
>> Well I tried to incorporate as much feedback as possible in specs (but I'm
>> still resisting changing the packaging to something longer to type than
>> org.specs._ ;-) ).
>>
>> Yet I have no doubt that some features could be dropped or reworked. For
>> example I provided a light-weight Mock library which I feel won't be
>> necessary at all when Mockito and Powermock will be available. The only
>> "interesting" feature was a way to declare "protocols" to check that an
>> API
>> was called with the proper sequence of events:
>>
>>  val protocol = expect(inAnyOrder) {
>>    expect(oneOf){mock.on; mock.off; mock.on}
>>    expect(oneOf){mock.off}
>>  }
>>
>> I also added some features that I needed from time to time:
>>
>> - detailedDiffs() to highlight the differences when 2 strings are not
>> matching (that one is very useful!)
>>
> ScalaTest does this as well.
>
>> - tags: you can tags systems and examples to get a more "tranversal" view
>> of
>> your specifications and execute only some tags while excluding others
>>
> ScalaTest does this as well too. Though it is called groups, as it was
> called in TestNG. Oddly enough this is one of the features I have
> wondered if I should have cut. I never used it, but I saw it in TestNG
> and thought it was a good idea. Regardless it's in. Though I think I
> like the name "tags" better. I noticed RSpec's Cucumber project just
> released a new version this week with the feature, and they call it
> tags too. I may change the name.
>
>> - matchers can be composed with functions to create matchers for object
>> graphs:
>>
>> http://code.google.com/p/specs/wiki/MatchersGuide#Matchers_composition_f...
>>
> I never could understand this. Could you explain how this works?
>
> Thanks.
>
> Bill
>
>