Re: Testing compile-time constraints

Maxime,

Thanks, your "specs" hint led me to [1] and [2]. Does anyone know
whether this functionality made it into specs2 in some form?

[1] http://stackoverflow.com/questions/1857999/static-testing-for-scala
[2] http://code.google.com/p/specs/source/browse/trunk/src/main/scala/org/sp...

2011/6/8 Maxime Lévesque :
>
> If there was a test framework that did a "non compilation" tests, I would
> use it today,
> instead I write a bunch of code that should not compile, verify that it
> doesn't,
> and then comment the code. When I do some refactoring, I have to remember
> to test "non compilation", i.e. uncomment and compile the code... This
> approach
> is clearly suboptimal.  Apparently the specs framework has such a feature.
>
> On Wed, Jun 8, 2011 at 5:58 PM, Aaron Novstrup
> wrote:
>>
>> Right, but I don't think that's true in general.  Here's an example,
>> to ground the discussion a bit more.  Consider an API that allows
>> users to make asynchronous remote procedure calls by coding in a
>> direct-style (rather than with callbacks):
>>
>> field.setText("Starting up...")
>> async {
>>  val response = service.asyncCall(arg1, arg2)
>>  field.setText(response)
>> }
>> field.setText("Waiting for response")
>>
>> Some desirable constraints for this API might include:
>> * It's not possible to make an RPC outside of an `async` or `asyncOnce`
>> block
>> * In an `asyncOnce` block, the user must make exactly one RPC
>> * Even in an `async` block, it's not possible to accidentally call a
>> method that makes an RPC -- you have to do something explicit to
>> indicate your intention
>> * It's not easy to accidentally call `shift` for a different
>> continuations-based API, without wrapping it in a reset, in an `async`
>> block
>>
>> Now, it was pretty easy to state these constraints, and users should
>> have no trouble following them.  But, at least for a half-wit such as
>> myself, it is not easy to encode these constraints in the type system.
>>  Even if it were easy, I would want some assurance that I don't break
>> the constraints as I add new features to the API.
>>
>> ~Aaron
>>
>> P.S. Sorry if the threading is messed up. I forgot to use "reply-all".
>>
>> On Wed, Jun 8, 2011 at 2:31 PM, Lex wrote:
>> > I was trying to say that if you cant easily encode the constraints,
>> > then your users will have very hard time using these constraints.
>> >
>> >
>> > On Wed, Jun 8, 2011 at 4:25 PM, Aaron Novstrup
>> > wrote:
>> >> The problem is not that the original developer can't understand the
>> >> constraints themselves (that would be silly indeed). Rather, the
>> >> problem is that encoding the constraints in the type system may be
>> >> non-trivial and it would be useful to be able to verify automatically
>> >> that the constraints are in fact enforced.  Most importantly, testing
>> >> would protect the developer from accidentally breaking constraints
>> >> while refactoring.
>> >
>
>

If there was a test framework that did a "non compilation" tests, I would use it today,
instead I write a bunch of code that should not compile, verify that it doesn't,
and then comment the code. When I do some refactoring, I have to remember
to test "non compilation", i.e. uncomment and compile the code... This approach
is clearly suboptimal.  Apparently the specs framework has such a feature.

On Wed, Jun 8, 2011 at 5:58 PM, Aaron Novstrup <aaron.novstrup@gmail.com> wrote:

Right, but I don't think that's true in general.  Here's an example,
to ground the discussion a bit more.  Consider an API that allows
users to make asynchronous remote procedure calls by coding in a
direct-style (rather than with callbacks):

field.setText("Starting up...")
async {
 val response = service.asyncCall(arg1, arg2)
 field.setText(response)
}
field.setText("Waiting for response")

Some desirable constraints for this API might include:
* It's not possible to make an RPC outside of an `async` or `asyncOnce` block
* In an `asyncOnce` block, the user must make exactly one RPC
* Even in an `async` block, it's not possible to accidentally call a
method that makes an RPC -- you have to do something explicit to
indicate your intention
* It's not easy to accidentally call `shift` for a different
continuations-based API, without wrapping it in a reset, in an `async`
block

Now, it was pretty easy to state these constraints, and users should
have no trouble following them.  But, at least for a half-wit such as
myself, it is not easy to encode these constraints in the type system.
 Even if it were easy, I would want some assurance that I don't break
the constraints as I add new features to the API.

~Aaron

P.S. Sorry if the threading is messed up. I forgot to use "reply-all".

On Wed, Jun 8, 2011 at 2:31 PM, Lex <lexn82@gmail.com> wrote:
> I was trying to say that if you cant easily encode the constraints,
> then your users will have very hard time using these constraints.
>
>
> On Wed, Jun 8, 2011 at 4:25 PM, Aaron Novstrup <aaron.novstrup@gmail.com> wrote:
>> The problem is not that the original developer can't understand the
>> constraints themselves (that would be silly indeed). Rather, the
>> problem is that encoding the constraints in the type system may be
>> non-trivial and it would be useful to be able to verify automatically
>> that the constraints are in fact enforced.  Most importantly, testing
>> would protect the developer from accidentally breaking constraints
>> while refactoring.
>

I think we are talking about different constraints here. I though you
meant type constraints, such as Trait[U <: Trait, S < U, T <: A[S with
B] with B[U]]. Now, I would never use API that requires this kind of
gymnastics. My own libs had these nasty type constraints at some
point, prompting a compete redesign.

On Wed, Jun 8, 2011 at 4:58 PM, Aaron Novstrup wrote:
> Right, but I don't think that's true in general.  Here's an example,
> to ground the discussion a bit more.  Consider an API that allows
> users to make asynchronous remote procedure calls by coding in a
> direct-style (rather than with callbacks):
>
> field.setText("Starting up...")
> async {
>  val response = service.asyncCall(arg1, arg2)
>  field.setText(response)
> }
> field.setText("Waiting for response")
>
> Some desirable constraints for this API might include:
> * It's not possible to make an RPC outside of an `async` or `asyncOnce` block
> * In an `asyncOnce` block, the user must make exactly one RPC
> * Even in an `async` block, it's not possible to accidentally call a
> method that makes an RPC -- you have to do something explicit to
> indicate your intention
> * It's not easy to accidentally call `shift` for a different
> continuations-based API, without wrapping it in a reset, in an `async`
> block
>
> Now, it was pretty easy to state these constraints, and users should
> have no trouble following them.  But, at least for a half-wit such as
> myself, it is not easy to encode these constraints in the type system.
>  Even if it were easy, I would want some assurance that I don't break
> the constraints as I add new features to the API.
>
> ~Aaron
>
> P.S. Sorry if the threading is messed up. I forgot to use "reply-all".
>
> On Wed, Jun 8, 2011 at 2:31 PM, Lex wrote:
>> I was trying to say that if you cant easily encode the constraints,
>> then your users will have very hard time using these constraints.
>>
>>
>> On Wed, Jun 8, 2011 at 4:25 PM, Aaron Novstrup wrote:
>>> The problem is not that the original developer can't understand the
>>> constraints themselves (that would be silly indeed). Rather, the
>>> problem is that encoding the constraints in the type system may be
>>> non-trivial and it would be useful to be able to verify automatically
>>> that the constraints are in fact enforced.  Most importantly, testing
>>> would protect the developer from accidentally breaking constraints
>>> while refactoring.
>>
>

2011/6/9 Aaron Novstrup :
> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
> whether this functionality made it into specs2 in some form?

Pretty sure that was dropped from Specs2, as it was hard to keep it
portable across different versions of Scalac. You directly invoke the
interpreter, as is done behind the scenes by Specs1 Snippets.

-jason

The snippets approach is limited in several ways:
* snippets are encoded in strings, so it's not possible to use
IDEs/tools to write/refactor them
* they are compiled at runtime, which slows down the test process
* as Jason pointed out, it's a difficult approach to maintain in test
frameworks because it creates a dependency on the Scala interpreter

An alternative approach would be to write the snippets in actual
source code within the test classes. Of course, the whole point is to
verify that the snippets fail to compile, which implies that the test
classes themselves would fail to compile. Would it be possible in
principle to write a compiler plugin that would isolate failure in a
given AST node and replace that node with the compilation result?

For example, I would write

@isolateFailure val snippet = List[String](1,2,3)

and the compiler plugin would turn this into something like

val snippet: CompileResult = CompileError("error: type mismatch; etc, etc")

Then the test code would just verify the expected CompileResult for
each snippet (e.g. `snippet must beAnInstanceOf[CompileError]`).

2011/6/8 Jason Zaugg :
> 2011/6/9 Aaron Novstrup :
>> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
>> whether this functionality made it into specs2 in some form?
>
> Pretty sure that was dropped from Specs2, as it was hard to keep it
> portable across different versions of Scalac. You directly invoke the
> interpreter, as is done behind the scenes by Specs1 Snippets.
>
> -jason
>

Hi All,

My gut reaction would be to put each bit of code into a file with a
name like, .badscala or something, then have some BadSuite that you
configure with a directory, and it goes in there discovering .badscala
files, tries to compile them, reporting a successful test for each one
that fails to compile, and a failed test for each one that compiles.
I'm not sure how you would invoke the compiler or find out whether the
code compiled or not, but I'm sure they must be doing this at EPFL so
it should be doable.

Backing up a bit, I think this would be useful. I try to make as many
errors compiler errors as possible, and it would be nice to have a way
to do this.

Bill

2011/6/8 Aaron Novstrup :
> The snippets approach is limited in several ways:
> * snippets are encoded in strings, so it's not possible to use
> IDEs/tools to write/refactor them
> * they are compiled at runtime, which slows down the test process
> * as Jason pointed out, it's a difficult approach to maintain in test
> frameworks because it creates a dependency on the Scala interpreter
>
> An alternative approach would be to write the snippets in actual
> source code within the test classes.  Of course, the whole point is to
> verify that the snippets fail to compile, which implies that the test
> classes themselves would fail to compile.  Would it be possible in
> principle to write a compiler plugin that would isolate failure in a
> given AST node and replace that node with the compilation result?
>
> For example, I would write
>
>   @isolateFailure val snippet = List[String](1,2,3)
>
> and the compiler plugin would turn this into something like
>
>   val snippet: CompileResult = CompileError("error: type mismatch; etc, etc")
>
> Then the test code would just verify the expected CompileResult for
> each snippet (e.g. `snippet must beAnInstanceOf[CompileError]`).
>
> 2011/6/8 Jason Zaugg :
>> 2011/6/9 Aaron Novstrup :
>>> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
>>> whether this functionality made it into specs2 in some form?
>>
>> Pretty sure that was dropped from Specs2, as it was hard to keep it
>> portable across different versions of Scalac. You directly invoke the
>> interpreter, as is done behind the scenes by Specs1 Snippets.
>>
>> -jason
>>
>

I think that's a good starting point, since it would be the simplest
approach to get up and running. It has the drawback, however, that it
forces you to scatter the test cases over separate files, and to
separate each test case from its description and any additional test
logic (i.e. logic to verify that you get a certain type of compile
error).

On Wed, Jun 8, 2011 at 6:43 PM, Bill Venners wrote:
> Hi All,
>
> My gut reaction would be to put each bit of code into a file with a
> name like, .badscala or something, then have some BadSuite that you
> configure with a directory, and it goes in there discovering .badscala
> files, tries to compile them, reporting a successful test for each one
> that fails to compile, and a failed test for each one that compiles.
> I'm not sure how you would invoke the compiler or find out whether the
> code compiled or not, but I'm sure they must be doing this at EPFL so
> it should be doable.
>
> Backing up a bit, I think this would be useful. I try to make as many
> errors compiler errors as possible, and it would be nice to have a way
> to do this.
>
> Bill
>
> 2011/6/8 Aaron Novstrup :
>> The snippets approach is limited in several ways:
>> * snippets are encoded in strings, so it's not possible to use
>> IDEs/tools to write/refactor them
>> * they are compiled at runtime, which slows down the test process
>> * as Jason pointed out, it's a difficult approach to maintain in test
>> frameworks because it creates a dependency on the Scala interpreter
>>
>> An alternative approach would be to write the snippets in actual
>> source code within the test classes.  Of course, the whole point is to
>> verify that the snippets fail to compile, which implies that the test
>> classes themselves would fail to compile.  Would it be possible in
>> principle to write a compiler plugin that would isolate failure in a
>> given AST node and replace that node with the compilation result?
>>
>> For example, I would write
>>
>>   @isolateFailure val snippet = List[String](1,2,3)
>>
>> and the compiler plugin would turn this into something like
>>
>>   val snippet: CompileResult = CompileError("error: type mismatch; etc, etc")
>>
>> Then the test code would just verify the expected CompileResult for
>> each snippet (e.g. `snippet must beAnInstanceOf[CompileError]`).
>>
>> 2011/6/8 Jason Zaugg :
>>> 2011/6/9 Aaron Novstrup :
>>>> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
>>>> whether this functionality made it into specs2 in some form?
>>>
>>> Pretty sure that was dropped from Specs2, as it was hard to keep it
>>> portable across different versions of Scalac. You directly invoke the
>>> interpreter, as is done behind the scenes by Specs1 Snippets.
>>>
>>> -jason
>>>
>>
>
>
>
> --
> Bill Venners
> Artima, Inc.
> http://www.artima.com
>

Hi Aaron,

I'm giving a Scala class this week so won't have time to code anything
up for a few days, but ScalaTest is designed to be easily customize
for just such unforeseen use cases. So if you want you can give this a
try in the meantime. I can think of two basic approaches, a custom
Suite trait or a custom matcher, and I'm not sure which you would
prefer, so I'll describe both. First the custom Suite trait approach:

ScalaTest's design is encapsulated in these lifecycle methods in trait suite:

run - override this method to define custom ways to run suites of tests.
runNestedSuites - override this method to define custom ways to run
nested suites.
runTests - override this method to define custom ways to run a suite's tests.
runTest - override this method to define custom ways to run a single named test.
testNames - override this method to specify the Suite's test names in
a custom way.
tags - override this method to specify the Suite's test tags in a custom way.
nestedSuites - override this method to specify the Suite's nested
Suites in a custom way.
suiteName - override this method to specify the Suite's name in a custom way.
expectedTestCount - override this method to count this Suite's
expected tests in a custom way.
withFixture - override this method to perform setup before and/or
cleanup after each test

So one approach is to define a trait ShouldNotCompileSuite that
overrides testNames to look in a directory or directories for files
that end in .shouldnotcompile (or some other extension) and then come
up with a testname for each such file. The test name could be based on
the filename, or grabbed from inside the file (say if a line matches
"testname: ..."), or both. Then override runTest, which gets passed
each testName, such that it somehow passes that file to the Scala
compiler, and ensures it does not compile. The directory or
directories in which to look could either be passed to the constructor
of ShouldNotCompileSuite or passed in via the "config map."

The other approach would be to make a "compile" matcher, which I'm
thinking might be nicer, but I want to know what you would prefer. You
could then write tests with any style trait, like FunSuite or
WordSpec, etc., then inside those write matcher expressions like:

"IncorrectUseOfMyDSL" should not (compile)

This matcher expression would go looking for a file named
IncorrectUseOfMyDSL.shouldnotcompile and somehow invoke the Scala
compiler on it and ensure it doesn't compile. It could be a matcher of
either Strings and/or java.io.Files, perhaps.

Trouble with the custom matcher approach is you have to explicitly
list each filename in your tests, whereas the custom Suite trait
approach can do discovery of files that should not compile if given
just a directory name or names. But the benefit of the custom matcher
approach is that you can write tests using existing style traits and
define test names or specifications like the rest of your test suite.

Either way, you'd need to figure out how to invoke the compiler and
check the result. Does anyone have a suggestion for that part?

Bill

On Wed, Jun 8, 2011 at 11:05 PM, Aaron Novstrup
wrote:
> I think that's a good starting point, since it would be the simplest
> approach to get up and running.  It has the drawback, however, that it
> forces you to scatter the test cases over separate files, and to
> separate each test case from its description and any additional test
> logic (i.e. logic to verify that you get a certain type of compile
> error).
>
> On Wed, Jun 8, 2011 at 6:43 PM, Bill Venners wrote:
>> Hi All,
>>
>> My gut reaction would be to put each bit of code into a file with a
>> name like, .badscala or something, then have some BadSuite that you
>> configure with a directory, and it goes in there discovering .badscala
>> files, tries to compile them, reporting a successful test for each one
>> that fails to compile, and a failed test for each one that compiles.
>> I'm not sure how you would invoke the compiler or find out whether the
>> code compiled or not, but I'm sure they must be doing this at EPFL so
>> it should be doable.
>>
>> Backing up a bit, I think this would be useful. I try to make as many
>> errors compiler errors as possible, and it would be nice to have a way
>> to do this.
>>
>> Bill
>>
>> 2011/6/8 Aaron Novstrup :
>>> The snippets approach is limited in several ways:
>>> * snippets are encoded in strings, so it's not possible to use
>>> IDEs/tools to write/refactor them
>>> * they are compiled at runtime, which slows down the test process
>>> * as Jason pointed out, it's a difficult approach to maintain in test
>>> frameworks because it creates a dependency on the Scala interpreter
>>>
>>> An alternative approach would be to write the snippets in actual
>>> source code within the test classes.  Of course, the whole point is to
>>> verify that the snippets fail to compile, which implies that the test
>>> classes themselves would fail to compile.  Would it be possible in
>>> principle to write a compiler plugin that would isolate failure in a
>>> given AST node and replace that node with the compilation result?
>>>
>>> For example, I would write
>>>
>>>   @isolateFailure val snippet = List[String](1,2,3)
>>>
>>> and the compiler plugin would turn this into something like
>>>
>>>   val snippet: CompileResult = CompileError("error: type mismatch; etc, etc")
>>>
>>> Then the test code would just verify the expected CompileResult for
>>> each snippet (e.g. `snippet must beAnInstanceOf[CompileError]`).
>>>
>>> 2011/6/8 Jason Zaugg :
>>>> 2011/6/9 Aaron Novstrup :
>>>>> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
>>>>> whether this functionality made it into specs2 in some form?
>>>>
>>>> Pretty sure that was dropped from Specs2, as it was hard to keep it
>>>> portable across different versions of Scalac. You directly invoke the
>>>> interpreter, as is done behind the scenes by Specs1 Snippets.
>>>>
>>>> -jason
>>>>
>>>
>>
>>
>>
>> --
>> Bill Venners
>> Artima, Inc.
>> http://www.artima.com
>>
>

Hi Aaron,

One more question. Do you think most of the syntax you would want to
make sure doesn't compile would be relatively small snippets of code?
Because if so, it seems like it might be nicer if those snippets could
just show up as strings in the tests or suites themselves. Something
like:

""" "hi".charAt(1) """ should compile
""" "hi".charAt("ho") """ should not (compile)

Another question is would you want to be able to specify a compiler version?

Bill

On Thu, Jun 9, 2011 at 7:15 AM, Bill Venners wrote:
> Hi Aaron,
>
> I'm giving a Scala class this week so won't have time to code anything
> up for a few days, but ScalaTest is designed to be easily customize
> for just such unforeseen use cases. So if you want you can give this a
> try in the meantime. I can think of two basic approaches, a custom
> Suite trait or a custom matcher, and I'm not sure which you would
> prefer, so I'll describe both. First the custom Suite trait approach:
>
> ScalaTest's design is encapsulated in these lifecycle methods in trait suite:
>
> run - override this method to define custom ways to run suites of tests.
> runNestedSuites - override this method to define custom ways to run
> nested suites.
> runTests - override this method to define custom ways to run a suite's tests.
> runTest - override this method to define custom ways to run a single named test.
> testNames - override this method to specify the Suite's test names in
> a custom way.
> tags - override this method to specify the Suite's test tags in a custom way.
> nestedSuites - override this method to specify the Suite's nested
> Suites in a custom way.
> suiteName - override this method to specify the Suite's name in a custom way.
> expectedTestCount - override this method to count this Suite's
> expected tests in a custom way.
> withFixture - override this method to perform setup before and/or
> cleanup after each test
>
> So one approach is to define a trait ShouldNotCompileSuite that
> overrides testNames to look in a directory or directories for files
> that end in .shouldnotcompile (or some other extension) and then come
> up with a testname for each such file. The test name could be based on
> the filename, or grabbed from inside the file (say if a line matches
> "testname: ..."), or both. Then override runTest, which gets passed
> each testName, such that it somehow passes that file to the Scala
> compiler, and ensures it does not compile. The directory or
> directories in which to look could either be passed to the constructor
> of ShouldNotCompileSuite or passed in via the "config map."
>
> The other approach would be to make a "compile" matcher, which I'm
> thinking might be nicer, but I want to know what you would prefer. You
> could then write tests with any style trait, like FunSuite or
> WordSpec, etc., then inside those write matcher expressions like:
>
> "IncorrectUseOfMyDSL" should not (compile)
>
> This matcher expression would go looking for a file named
> IncorrectUseOfMyDSL.shouldnotcompile and somehow invoke the Scala
> compiler on it and ensure it doesn't compile. It could be a matcher of
> either Strings and/or java.io.Files, perhaps.
>
> Trouble with the custom matcher approach is you have to explicitly
> list each filename in your tests, whereas the custom Suite trait
> approach can do discovery of files that should not compile if given
> just a directory name or names. But the benefit of the custom matcher
> approach is that you can write tests using existing style traits and
> define test names or specifications like the rest of your test suite.
>
> Either way, you'd need to figure out how to invoke the compiler and
> check the result. Does anyone have a suggestion for that part?
>
> Bill
>
> On Wed, Jun 8, 2011 at 11:05 PM, Aaron Novstrup
> wrote:
>> I think that's a good starting point, since it would be the simplest
>> approach to get up and running.  It has the drawback, however, that it
>> forces you to scatter the test cases over separate files, and to
>> separate each test case from its description and any additional test
>> logic (i.e. logic to verify that you get a certain type of compile
>> error).
>>
>> On Wed, Jun 8, 2011 at 6:43 PM, Bill Venners wrote:
>>> Hi All,
>>>
>>> My gut reaction would be to put each bit of code into a file with a
>>> name like, .badscala or something, then have some BadSuite that you
>>> configure with a directory, and it goes in there discovering .badscala
>>> files, tries to compile them, reporting a successful test for each one
>>> that fails to compile, and a failed test for each one that compiles.
>>> I'm not sure how you would invoke the compiler or find out whether the
>>> code compiled or not, but I'm sure they must be doing this at EPFL so
>>> it should be doable.
>>>
>>> Backing up a bit, I think this would be useful. I try to make as many
>>> errors compiler errors as possible, and it would be nice to have a way
>>> to do this.
>>>
>>> Bill
>>>
>>> 2011/6/8 Aaron Novstrup :
>>>> The snippets approach is limited in several ways:
>>>> * snippets are encoded in strings, so it's not possible to use
>>>> IDEs/tools to write/refactor them
>>>> * they are compiled at runtime, which slows down the test process
>>>> * as Jason pointed out, it's a difficult approach to maintain in test
>>>> frameworks because it creates a dependency on the Scala interpreter
>>>>
>>>> An alternative approach would be to write the snippets in actual
>>>> source code within the test classes.  Of course, the whole point is to
>>>> verify that the snippets fail to compile, which implies that the test
>>>> classes themselves would fail to compile.  Would it be possible in
>>>> principle to write a compiler plugin that would isolate failure in a
>>>> given AST node and replace that node with the compilation result?
>>>>
>>>> For example, I would write
>>>>
>>>>   @isolateFailure val snippet = List[String](1,2,3)
>>>>
>>>> and the compiler plugin would turn this into something like
>>>>
>>>>   val snippet: CompileResult = CompileError("error: type mismatch; etc, etc")
>>>>
>>>> Then the test code would just verify the expected CompileResult for
>>>> each snippet (e.g. `snippet must beAnInstanceOf[CompileError]`).
>>>>
>>>> 2011/6/8 Jason Zaugg :
>>>>> 2011/6/9 Aaron Novstrup :
>>>>>> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
>>>>>> whether this functionality made it into specs2 in some form?
>>>>>
>>>>> Pretty sure that was dropped from Specs2, as it was hard to keep it
>>>>> portable across different versions of Scalac. You directly invoke the
>>>>> interpreter, as is done behind the scenes by Specs1 Snippets.
>>>>>
>>>>> -jason
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Bill Venners
>>> Artima, Inc.
>>> http://www.artima.com
>>>
>>
>
>
>
> --
> Bill Venners
> Artima, Inc.
> http://www.artima.com
>

I agree that the matcher approach is more flexible. More replies inline.

On Thu, Jun 9, 2011 at 5:15 AM, Bill Venners wrote:
> It could be a matcher of either Strings and/or java.io.Files, perhaps.

I like this idea.

> Trouble with the custom matcher approach is you have to explicitly
> list each filename in your tests, whereas the custom Suite trait
> approach can do discovery of files that should not compile if given
> just a directory name or names.

If the matcher matches java.io.Files, you could check whether the file
is a directory and do automatic discovery in that case. More generally,
a DSL could specify the files. e.g.

filesMatching("*.badscala") in "path/to/files" should not (compile)

> Either way, you'd need to figure out how to invoke the compiler and
> check the result. Does anyone have a suggestion for that part?

I wonder if some logic could be borrowed from sbt.

> On Wed, Jun 8, 2011 at 11:05 PM, Aaron Novstrup
> wrote:
>> I think that's a good starting point, since it would be the simplest
>> approach to get up and running.  It has the drawback, however, that it
>> forces you to scatter the test cases over separate files, and to
>> separate each test case from its description and any additional test
>> logic (i.e. logic to verify that you get a certain type of compile
>> error).
>>
>> On Wed, Jun 8, 2011 at 6:43 PM, Bill Venners wrote:
>>> Hi All,
>>>
>>> My gut reaction would be to put each bit of code into a file with a
>>> name like, .badscala or something, then have some BadSuite that you
>>> configure with a directory, and it goes in there discovering .badscala
>>> files, tries to compile them, reporting a successful test for each one
>>> that fails to compile, and a failed test for each one that compiles.
>>> I'm not sure how you would invoke the compiler or find out whether the
>>> code compiled or not, but I'm sure they must be doing this at EPFL so
>>> it should be doable.
>>>
>>> Backing up a bit, I think this would be useful. I try to make as many
>>> errors compiler errors as possible, and it would be nice to have a way
>>> to do this.
>>>
>>> Bill

I do expect that most of the snippets would be fairly small, which is
why the specs Snippets approach seemed nice. It has its own issues,
though: its slow because it relies on the interpreter, and the
snippets are no longer accessible to IDEs as source code.

Concerning compiler version: I suppose there are a whole host of
configuration issues once you get into the realm of invoking the
compiler. I don't really have a good answer for this. One idea would
be to have an sbt/ant/maven plugin do the compilation part and leave
behind compiler output in some form that the test framework could just
read and verify (of course, this approach would be more difficult with
embedded strings).

On Thu, Jun 9, 2011 at 5:47 AM, Bill Venners wrote:
> Hi Aaron,
>
> One more question. Do you think most of the syntax you would want to
> make sure doesn't compile would be relatively small snippets of code?
> Because if so, it seems like it might be nicer if those snippets could
> just show up as strings in the tests or suites themselves. Something
> like:
>
> """  "hi".charAt(1) """ should compile
> """  "hi".charAt("ho") """ should not (compile)
>
> Another question is would you want to be able to specify a compiler version?
>
> Bill
>
> On Thu, Jun 9, 2011 at 7:15 AM, Bill Venners wrote:
>> Hi Aaron,
>>
>> I'm giving a Scala class this week so won't have time to code anything
>> up for a few days, but ScalaTest is designed to be easily customize
>> for just such unforeseen use cases. So if you want you can give this a
>> try in the meantime. I can think of two basic approaches, a custom
>> Suite trait or a custom matcher, and I'm not sure which you would
>> prefer, so I'll describe both. First the custom Suite trait approach:
>>
>> ScalaTest's design is encapsulated in these lifecycle methods in trait suite:
>>
>> run - override this method to define custom ways to run suites of tests.
>> runNestedSuites - override this method to define custom ways to run
>> nested suites.
>> runTests - override this method to define custom ways to run a suite's tests.
>> runTest - override this method to define custom ways to run a single named test.
>> testNames - override this method to specify the Suite's test names in
>> a custom way.
>> tags - override this method to specify the Suite's test tags in a custom way.
>> nestedSuites - override this method to specify the Suite's nested
>> Suites in a custom way.
>> suiteName - override this method to specify the Suite's name in a custom way.
>> expectedTestCount - override this method to count this Suite's
>> expected tests in a custom way.
>> withFixture - override this method to perform setup before and/or
>> cleanup after each test
>>
>> So one approach is to define a trait ShouldNotCompileSuite that
>> overrides testNames to look in a directory or directories for files
>> that end in .shouldnotcompile (or some other extension) and then come
>> up with a testname for each such file. The test name could be based on
>> the filename, or grabbed from inside the file (say if a line matches
>> "testname: ..."), or both. Then override runTest, which gets passed
>> each testName, such that it somehow passes that file to the Scala
>> compiler, and ensures it does not compile. The directory or
>> directories in which to look could either be passed to the constructor
>> of ShouldNotCompileSuite or passed in via the "config map."
>>
>> The other approach would be to make a "compile" matcher, which I'm
>> thinking might be nicer, but I want to know what you would prefer. You
>> could then write tests with any style trait, like FunSuite or
>> WordSpec, etc., then inside those write matcher expressions like:
>>
>> "IncorrectUseOfMyDSL" should not (compile)
>>
>> This matcher expression would go looking for a file named
>> IncorrectUseOfMyDSL.shouldnotcompile and somehow invoke the Scala
>> compiler on it and ensure it doesn't compile. It could be a matcher of
>> either Strings and/or java.io.Files, perhaps.
>>
>> Trouble with the custom matcher approach is you have to explicitly
>> list each filename in your tests, whereas the custom Suite trait
>> approach can do discovery of files that should not compile if given
>> just a directory name or names. But the benefit of the custom matcher
>> approach is that you can write tests using existing style traits and
>> define test names or specifications like the rest of your test suite.
>>
>> Either way, you'd need to figure out how to invoke the compiler and
>> check the result. Does anyone have a suggestion for that part?
>>
>> Bill
>>
>> On Wed, Jun 8, 2011 at 11:05 PM, Aaron Novstrup
>> wrote:
>>> I think that's a good starting point, since it would be the simplest
>>> approach to get up and running.  It has the drawback, however, that it
>>> forces you to scatter the test cases over separate files, and to
>>> separate each test case from its description and any additional test
>>> logic (i.e. logic to verify that you get a certain type of compile
>>> error).
>>>
>>> On Wed, Jun 8, 2011 at 6:43 PM, Bill Venners wrote:
>>>> Hi All,
>>>>
>>>> My gut reaction would be to put each bit of code into a file with a
>>>> name like, .badscala or something, then have some BadSuite that you
>>>> configure with a directory, and it goes in there discovering .badscala
>>>> files, tries to compile them, reporting a successful test for each one
>>>> that fails to compile, and a failed test for each one that compiles.
>>>> I'm not sure how you would invoke the compiler or find out whether the
>>>> code compiled or not, but I'm sure they must be doing this at EPFL so
>>>> it should be doable.
>>>>
>>>> Backing up a bit, I think this would be useful. I try to make as many
>>>> errors compiler errors as possible, and it would be nice to have a way
>>>> to do this.
>>>>
>>>> Bill
>>>>
>>>> 2011/6/8 Aaron Novstrup :
>>>>> The snippets approach is limited in several ways:
>>>>> * snippets are encoded in strings, so it's not possible to use
>>>>> IDEs/tools to write/refactor them
>>>>> * they are compiled at runtime, which slows down the test process
>>>>> * as Jason pointed out, it's a difficult approach to maintain in test
>>>>> frameworks because it creates a dependency on the Scala interpreter
>>>>>
>>>>> An alternative approach would be to write the snippets in actual
>>>>> source code within the test classes.  Of course, the whole point is to
>>>>> verify that the snippets fail to compile, which implies that the test
>>>>> classes themselves would fail to compile.  Would it be possible in
>>>>> principle to write a compiler plugin that would isolate failure in a
>>>>> given AST node and replace that node with the compilation result?
>>>>>
>>>>> For example, I would write
>>>>>
>>>>>   @isolateFailure val snippet = List[String](1,2,3)
>>>>>
>>>>> and the compiler plugin would turn this into something like
>>>>>
>>>>>   val snippet: CompileResult = CompileError("error: type mismatch; etc, etc")
>>>>>
>>>>> Then the test code would just verify the expected CompileResult for
>>>>> each snippet (e.g. `snippet must beAnInstanceOf[CompileError]`).
>>>>>
>>>>> 2011/6/8 Jason Zaugg :
>>>>>> 2011/6/9 Aaron Novstrup :
>>>>>>> Thanks, your "specs" hint led me to [1] and [2].  Does anyone know
>>>>>>> whether this functionality made it into specs2 in some form?
>>>>>>
>>>>>> Pretty sure that was dropped from Specs2, as it was hard to keep it
>>>>>> portable across different versions of Scalac. You directly invoke the
>>>>>> interpreter, as is done behind the scenes by Specs1 Snippets.
>>>>>>
>>>>>> -jason
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Bill Venners
>>>> Artima, Inc.
>>>> http://www.artima.com
>>>>
>>>
>>
>>
>>
>> --
>> Bill Venners
>> Artima, Inc.
>> http://www.artima.com
>>
>
>
>
> --
> Bill Venners
> Artima, Inc.
> http://www.artima.com
>

HI,

On 9 June 2011 15:45, Aaron Novstrup <aaron.novstrup@gmail.com> wrote:

I do expect that most of the snippets would be fairly small, which is
why the specs Snippets approach seemed nice.  

Additionally, it keeps the entire test case in one place, rather than shattering it over various files and directories. If something fails, you know exactly where to look.  

It has its own issues,
though: its slow because it relies on the interpreter,

This one, yes - although would be a great use-case to drive making the interpreter super-lean. 

and the
snippets are no longer accessible to IDEs as source code.

In IntellijIDEA you can use 'language injection' to designate a string as being source in some language understood by the IDE. I don't know if eclipse and netbeans have a similar ability.  Matthew
--
Matthew Pocockmailto: turingatemyhamster@gmail.comgchat: turingatemyhamster@gmail.com msn: matthew_pocock@yahoo.co.ukirc.freenode.net: drdozer(0191) 2566550

Replies inline.

On Thu, Jun 9, 2011 at 7:57 AM, Matthew Pocock
wrote:
> On 9 June 2011 15:45, Aaron Novstrup wrote:
>> It has its own issues,
>> though: its slow because it relies on the interpreter,
>
> This one, yes - although would be a great use-case to drive making the
> interpreter super-lean.

Right. Actually, the use case is different enough from the REPL use
case that it would probably be desirable to have a special-purpose
interpreter that avoids a lot of the overhead of the REPL interpreter.

>> and the
>> snippets are no longer accessible to IDEs as source code.
>
> In IntellijIDEA you can use 'language injection' to designate a string as
> being source in some language understood by the IDE. I don't know if eclipse
> and netbeans have a similar ability.

Oooh, that's awesome! I doubt Eclipse does that, but I prefer IntelliJ
anyway. How do you designate it as being source? It would be great if
it uses an annotation. Actually, I think that would make the compiler
plugin approach a whole lot more feasible. I'm imagining something
like:

class source(lang: String) extends StaticAnnotation // this is the
annotation used to designate a string as source
class snippet extends source("scala") // this
is a special annotation used by the compiler plugin

sealed class CompileResult
case object CompileSuccess
case class CompileFailure(error: String)

Tests would then look like:

class Test extends WhateverSuite {
@snippet val s1: CompileResult = "List[String](1,2,3)"
@snippet val s2: CompileResult = """ withResource { r =>
r.put[Int]("1")
} """

s1 must not (compile)
s2 must not (compile)
}

The compiler plugin could look for the @snippet annotation and invoke
another instance of the compiler to compile each snippet, then
statically put the result in a CompileResult object. At runtime, the
tests would just check for the expected CompileResult.

>
> Matthew
> --
> Matthew Pocock
> mailto: turingatemyhamster@gmail.com
> gchat: turingatemyhamster@gmail.com
> msn: matthew_pocock@yahoo.co.uk
> irc.freenode.net: drdozer
> (0191) 2566550
>

On Thu, Jun 9, 2011 at 9:00 PM, Aaron Novstrup wrote:
>> In IntellijIDEA you can use 'language injection' to designate a string as
>> being source in some language understood by the IDE. I don't know if eclipse
>> and netbeans have a similar ability.
>
> Oooh, that's awesome! I doubt Eclipse does that, but I prefer IntelliJ
> anyway.  How do you designate it as being source? It would be great if
> it uses an annotation.  Actually, I think that would make the compiler
> plugin approach a whole lot more feasible.  I'm imagining something
> like:

Here's an example of this used, in the unit tests for the IDEA Scala
plugin itself.

http://bit.ly/l26Jjb
http://bit.ly/jfgg5G
http://imgur.com/5gU46

The code snippet is standalone (other than the header and footer). So
it can't reference the code defined in the 'host' file/module. Perhaps
that could be made to work, if the IntelliLang plugin allows for that
sort of thing.

-jason

Maybe this is a candidate for an sbt plugin.All source files in src/test/badscala must not compile, or testing fails!

****or***** partest could be made an sbt plugin, since it already supports negative compile tests.

On Jun 16, 2011 7:34 AM, "Naftoli Gugenheim" <naftoligug@gmail.com> wrote:
> Maybe this is a candidate for an sbt plugin.
> All source files in src/test/badscala must not compile, or testing fails!

Hi Josh,

Can you point me to partest? Or more generally, can you or someone
point me to how to checkout the whole Scala shebang and build and run
partest? I'd like to see how it does this kind of ensuring something
does not compile test.

Thanks.

Bill

On Thu, Jun 16, 2011 at 5:07 AM, Josh Suereth wrote:
> ****or***** partest could be made an sbt plugin, since it already supports
> negative compile tests.
>
> On Jun 16, 2011 7:34 AM, "Naftoli Gugenheim" wrote:
>> Maybe this is a candidate for an sbt plugin.
>> All source files in src/test/badscala must not compile, or testing fails!
>

If you checkout Scala itself and build it, you'll see partest if you run "ant test".  More than that I'm not sure there's very good documentation, besides looking at the examples.
- Josh

On Thu, Jun 16, 2011 at 11:01 AM, Bill Venners <bill@artima.com> wrote:

Hi Josh,

Can you point me to partest? Or more generally, can you or someone
point me to how to checkout the whole Scala shebang and build and run
partest? I'd like to see how it does this kind of ensuring something
does not compile test.

Thanks.

Bill

On Thu, Jun 16, 2011 at 5:07 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> ****or***** partest could be made an sbt plugin, since it already supports
> negative compile tests.
>
> On Jun 16, 2011 7:34 AM, "Naftoli Gugenheim" <naftoligug@gmail.com> wrote:
>> Maybe this is a candidate for an sbt plugin.
>> All source files in src/test/badscala must not compile, or testing fails!
>



--
Bill Venners
Artima, Inc.
http://www.artima.com

Hi Josh,

OK. I'll give that a try. Checking it out now. Paul Phillips described
firing up the interpreter for testing through partest. I wonder if
that might be a good way to approach this for others. I'll ask Paul to
point me to some examples.

Bill

On Thu, Jun 16, 2011 at 8:06 AM, Josh Suereth wrote:
> If you checkout Scala itself and build it, you'll see partest if you run
> "ant test".  More than that I'm not sure there's very good documentation,
> besides looking at the examples.
> - Josh
>
> On Thu, Jun 16, 2011 at 11:01 AM, Bill Venners wrote:
>>
>> Hi Josh,
>>
>> Can you point me to partest? Or more generally, can you or someone
>> point me to how to checkout the whole Scala shebang and build and run
>> partest? I'd like to see how it does this kind of ensuring something
>> does not compile test.
>>
>> Thanks.
>>
>> Bill
>>
>> On Thu, Jun 16, 2011 at 5:07 AM, Josh Suereth
>> wrote:
>> > ****or***** partest could be made an sbt plugin, since it already
>> > supports
>> > negative compile tests.
>> >
>> > On Jun 16, 2011 7:34 AM, "Naftoli Gugenheim"
>> > wrote:
>> >> Maybe this is a candidate for an sbt plugin.
>> >> All source files in src/test/badscala must not compile, or testing
>> >> fails!
>> >
>>
>>
>>
>> --
>> Bill Venners
>> Artima, Inc.
>> http://www.artima.com
>
>

On Thu, Jun 16, 2011 at 11:01 AM, Bill Venners wrote:
> Can you point me to partest? Or more generally, can you or someone
> point me to how to checkout the whole Scala shebang and build and run
> partest? I'd like to see how it does this kind of ensuring something
> does not compile test.

After building Scala, running ./test/partest will print a usage message.
See also src/partest/README.

To run the "shouldn't compile" tests only you do "test/partest --neg".

On Thu, Jun 16, 2011 at 17:48, Seth Tisue wrote:
> On Thu, Jun 16, 2011 at 11:01 AM, Bill Venners wrote:
>> Can you point me to partest? Or more generally, can you or someone
>> point me to how to checkout the whole Scala shebang and build and run
>> partest? I'd like to see how it does this kind of ensuring something
>> does not compile test.
>
> After building Scala, running ./test/partest will print a usage message.
> See also src/partest/README.

Oh. It does. I tried partest -help, and that doesn't show anything,
nor does it mention the possibility of running without parameters to
get help.