Forty Six and Two: FsCheck 0.7: Pleasure and Pain

I just released FsCheck 0.7 on the codeplex page. Whereas previous releases have been mostly incremental, this one is quite different. With the official release of F# 2.0 and all that it entails, I’ve refactored the FsCheck API from a usability perspective.

The previous API was inherited from Haskell QuickCheck, which means that it wasn’t very .NET like. For example, once you opened the FsCheck namespace, all the relevant functions just came into scope. So you, dear user, had to rely mostly on your memory to find the function you need, and also you were not empowered to discover new functions. The new API is a much needed improvement on that situation: it is consistent with .NET idioms, works well with intellisense and the documentation should tell you all you need to know.

I’ve also tried to improve FsCheck’s output, tried to improve the story with registering generators by type (which was a real pain before, and I hope it has now got at least a bit better), removed concepts that nobody seemed to understand (Hi, coarbitrary!) and generally improved the documentation and names of functions.

That’s the good news – the bad news is that this release is not backwards compatible. I’m afraid that was not possible given the breadth and depth of the refactoring that I wanted to do. I really believe that FsCheck is in a much better place now, also as far as plans for the future are concerned.

Short attention span?

Get a visually appealing (but much less detailed) overview in this prezi.

The whole truth

Well, at least as far as I can remember.

Factored API into three type and module pairs – cfr e.g. the collection types in FSharp.Core. So there is now:

Gen<’a> type and the Gen module that contains utility functions to define your own random generators. This is essentially the same as before, except there have been many renames to make everything consistent both internally and with .NET, and understandable (e.g. vectorOf has been renamed to listOfLength, and fmapGen has been renamed to Gen.map). Also, where functions used to take a list as argument, they now take a more general and .NET-like seq instead.
Arbitrary<’a> type and Arb module. Arbitrary, as before, packages up a generator and a shrinker. Coarbitrary was removed (without removing FsCheck’s functionality to generate functions). The Arb module is new, and plays a fairly central role in FsCheck now – a few useful functions that operate on a generator and a shrinker at the same time were added.

Arb.Default type contains the default generators and shrinkers that ship with FsCheck. I’ve cleaned up most of the mess, but am not completely finished yet. Anyway, you should find many useful Arbitrary instances here, and they are now easily accessible (I believe in the previous version you had to type something like Arbitrary.Arbitrary.String().Arbitrary – now it is Arb.Default.String)

Property type and Prop module. Here is where all the functions to build the actual properties live. Not much changes here, except that Prop.forAll now takes an Arbitrary instance instead of a generator – I believe this is a first step in making FsCheck more easily extendable to other kinds of generators (e.g. file based or exhaustive generators) by abstracting away the Gen type from Property. (This is currently only skin-deep though)

Similar refactoring with running tests.

Instead of quickCheck, verboseCheck, and all the other functions, there is now a Check type with relevant static methods that do the same thing. This made it possible to overload method names, which was annoying about the functions – I felt like I had to make up meaningless names. Also, the various checking types that you can do are now discoverable using intellisense by typing Check.<intellisense>.
Similarly, the default configurations have now been added as static members on the Config record type, which is were you would expect them to be.
The Runner module has had some cleanup as well, with some new names for old friends, and the IRunner interface has been extended a bit. Like the other modules, it should only expose what is useful to you, so it should be fairly easy to figure out what you need from intellisense alone.

Output has been improved in various places.

When registering generators, FsCheck now automatically overrides the previous generators, and also produces to some output where you can see what the new types that have been registered are, as well as what types were overridden. Hopefully this will help you fix those frustrating issues where types are not registered properly more easily.
When checking all the members on a type or module, and an exception is thrown, the TargetInvocationExcception is now removed and you just get to see the actual exception, which reduces much of the noise.
Various small output improvements.

Size control has been changed in the Config. Instead of a function, you can now give a begin and end size and FsCheck will increase the size linearly between those two values. I did this to make the size independent of the number of test runs that you choose – the final size is always guaranteed to be the end size you ask for. So it is easy to change the number of tests while keeping the same size. I believe this is the least surprising result, as changing the number of tests should not directly influence the test size.
Added many more tests, especially for the generators. I’m still not happy with the coverage, but we’re getting there.
Wrote a simple documentation generator. I’m not very happy with it, but at least I reviewed all the documentation and at least the output shown in the documentation is now consistent with the actual output, and it is easy to keep that way.
Cleaned up the examples a bit. The examples used in the documentation are now moved to the FsCheck.Documentation project (which also generated the codeplex formatted output).
Typeclass simulation has had a pretty much bottom up rewrite. Hopefully this cleans up some nasty issues with static state I’ve seen, and also the API is I think much improved. Various bug fixes.

That’s pretty much it. I’ve got more in store still, but not sure when it will materialize into something that I can release. My spare time is getting scarce…but do post a comment on the forum or on my blog if you get stuck in moving your old FsCheck code to the new API, or any other feedback you may have.

Happy FsChecking!