FsCheck 0.9.4

Small update – some other generators have been hardened against the new null-generating string generator. Also, FsCheck is compiled against F# 3.1 now, FSharp.Core 4.3.1.0. As always, available on NuGet.

FsCheck 0.9.3 and FsCheck.Xunit 0.4.1 released

FsCheck and Fscheck.Xunit both have a new release on NuGet.

FsCheck has a breaking change – the string and object generates by default now generate and shrink nulls.

FsCheck.Xunit has a workaround for an Xunit bug, the bug itself is also fixed in a beta version of Xunit itself but I figured the workaround is worth it.

FsCheck 0.9 Released and moved to GitHub

FsCheck has a new home on GitHub. I wanted to move to GitHub for a while, because it’s where all the cool kids hang out, and @carstkoenig gave me the necessary push. I hope the barrier for contributions is now low enough to get you to chip in if you feel like it, and simplifies the life of existing contributors. (Not that I’m complaining. You’ve been great.)

As for the long overdue release - a significant portion of contributions didn’t come from me.

Here are the highlights of the change log:

• New generators for KeyValuePair, IList, ICollection, Action, Func, decimal (@mausch) and sbyte (@jackfoxy).
• Finally compiled against FSharp.Core 4.3 (no more binding redirect. Yay.)
• FsCheck.Xunit now allows you to replay a certain test by pasting in the seed from the output in the PropertyAttribute’s Replay property. This should help with debugging:

  [<Property(Replay="54321,67584")>]

• FsCheck.Xunit also allows defining ArbitraryAttribute on an enclosing module or type now – the given Arbitrary instances are then available for all properties in the type or module. But you can still override them if you define one on a specific property. 

      [<Arbitrary(typeof<TestArbitrary2>)>]
      module ModuleWithArbitrary =
  
          [<Property>]
          let ``should use Arb instances from enclosing module``(underTest:float) =
              underTest <= 0.0
  
          [<Property( Arbitrary=[| typeof<TestArbitrary1> |] )>]
          let ``should use Arb instance on method preferentially``(underTest:float) =
              underTest >= 0.0

Hope you enjoy.

Technorati Tags: fscheck,xunit

FsCheck 0.8.3 and FsCheck.Xunit 0.3

I’ve just uploaded new NuGet packages for FsCheck and FsCheck.Xunit.

The changes are minor:

• FsCheck.Xunit now works with instance methods. NCrunch, for example, reportedly only works with instance methods, so this means you can now use NCrunch for running FsCheck properties.
• Both NuGet packages now call Add-BindingRedirect on install. This will generate an app.config file in your test project if necessary, to spell out to the .NET loader that tests compiled with F#3.0 and so in all likelihood using FSharp.Core 4.3 should indeed use 4.3 instead of the older 4.0, which FsCheck is compiled with.
• Added symbols on symbolsource.org.

Note that running “Add-BindingRedirect <project name>” in the NuGet package manager console should resolve any problems you may have running FsCheck in VS2012. Also, since xunit is strongly signed, there are similar problems with FsCheck.Xunit – Add-BindingRedirect solves that one too.

If you’re not into NuGet, add the following lines to your app.config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="FSharp.Core" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.3.0.0" newVersion="4.3.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

Technorati Tags: F#,FsCheck,xunit

Share this post :

MSDN!

Technet!

Del.icio.us!

Digg!

Dotnetkicks!

Reddit!

Technorati!

FsCheck 0.8.1: The ecosystem release

Yesterday I’ve released a new version of FsCheck. There are some minor changes, additions and bug fixes to FsCheck itself. The most important part of this release is that FsCheck is now integrated with Xunit through the new FsCheck.Xunit assembly.

For the impatient: NuGet packages for FsCheck and FsCheck.Xunit

Here goes a more detailed overview.

Generators are now applicative. Or, without the general abstract nonsense: say you’ve got three generators

let ga,gb,gc :Gen<_> * Gen<_> * Gen<_>

And you have a function of three arguments that you’d like to map over those.

let f = fun a b c ->

No problem, you can use Gen.map3 – right? But how many instances of map do you need? FsCheck currently defines 6. But now that Gen is applicative, we can write:

let result = f <!> ga <*> gb <*> gc

Hey, doesn’t that almost read like function application? That’s the idea. The weird operators in between just make sure everything is applied properly. Besides being a bit nicer to use sometimes, you can also keep chaining this indefinitely, up to any arity.

And hey, if you still prefer map, FsCheck still has those.

New generators. Mauricio Scheffer contributed int64 and TimeSpan generators. I’ve also added an int16 generator, and DontSize wrappers for all of those – the default generators for int16, 32 and 64 will generate integers smaller than the test size, which is increased by FsCheck as the test progresses. The DontSize variants ignore the size and always pick from the whole range for the type.

Xunit integration. Xunit is a very nice framework – very simple to use and extend. FsCheck integrates with Xunit by adding a PropertyAtttribute. To use it, just use Property instead of Fact, and you can now give your test arguments. These are randomly generated by FsCheck. You can also use all of the normal property combinators in the Prop namespace. A couple of examples:

[<Property>]
let ``abs(v) % k equals abs(v % k)`` v (NonZeroInt k) = 
    (abs v) % k = abs(v % k)

[<Property>]
let ``divMod should satisfy definition`` (x:int) (y:int) = 
    y <> 0 ==> lazy (let (d,m) = divMod x y in d*y + m = x)

Note that Property test methods, as opposed to Fact, don’t need to return unit – they can return anything that’s testable with FsCheck, including bool. But since a test fails in FsCheck when it throws an exception, you can still use the Assert class if you like. You can also use FsUnit or Unquote or whatever to write your assertions. FsCheck does not care – it does the hard work of generating random values and shrinking them if the test fails and generally tries to get out of your way for the rest.

You can also set various configuration parameters using the Property attribute, which then apply to that method only. Most of these are trivial (e.g. the number of tests), but an interesting one is that you can override the random generation for certain types on one test method only:

type TestArbitrary1 =
    static member PositiveDouble() =
        Arb.Default.Float()
        |> Arb.mapFilter abs (fun t -> t >= 0.0)

[<Property( Arbitrary=[| typeof<TestArbitrary1> |] )>]
let ``should register Arbitrary instances from Config in last to first order``(underTest:float) =
    underTest >= 0.0

Here the generator for float is overridden to generate only positive floats.

Finally, a special thanks to @mausch for contributing, integrating FsCheck with Fuchu, and giving me a gentle nudge to get this release out the door at last!

Technorati Tags: fscheck,f#,xunit.net

Share this post :

MSDN!

Technet!

Del.icio.us!

Digg!

Dotnetkicks!

Reddit!

Technorati!

FsCheck 0.7.1: NuGet packaging

I’ve released a minor update to FsCheck, mostly so it can now be downloaded as a NuGet package. It also comes with source server support, courtesy of SymbolSource.org.

Furthermore, there are a couple of bug fixes.

Enjoy.

Technorati Tags: fscheck,random testing

Share this post :

MSDN!

Technet!

Del.icio.us!

Digg!

Dotnetkicks!

Reddit!

Technorati!

FsCheck 0.7: Pleasure and Pain

I just released FsCheck 0.7  on the codeplex page. Whereas previous releases have been mostly incremental, this one is quite different. With the official release of F# 2.0 and all that it entails, I’ve refactored the FsCheck API from a usability perspective.

The previous API was inherited from Haskell QuickCheck, which means that it wasn’t very .NET like. For example, once you opened the FsCheck namespace, all the relevant functions just came into scope. So you, dear user, had to rely mostly on your memory to find the function you need, and also you were not empowered to discover new functions. The new API is a much needed improvement on that situation: it is consistent with .NET idioms, works well with intellisense and the documentation should tell you all you need to know.

I’ve also tried to improve FsCheck’s output, tried to improve the story with registering generators by type (which was a real pain before, and I hope it has now got at least a bit better), removed concepts that nobody seemed to understand (Hi, coarbitrary!) and generally improved the documentation and names of functions.

That’s the good news – the bad news is that this release is not backwards compatible. I’m afraid that was not possible given the breadth and depth of the refactoring that I wanted to do. I really believe that FsCheck is in a much better place now, also as far as plans for the future are concerned.

Short attention span?

Get a visually appealing (but much less detailed) overview in this prezi.

The whole truth

Well, at least as far as I can remember.

• Factored API into three type and  module pairs – cfr e.g. the collection types in FSharp.Core. So there is now:
• Gen<’a> type and the Gen module that contains utility functions to define your own random generators. This is essentially the same as before, except there have been many renames to make everything consistent both internally and with .NET, and understandable (e.g. vectorOf has been renamed to listOfLength, and fmapGen has been renamed to Gen.map). Also, where functions used to take a list as argument, they now take a more general and .NET-like seq instead.
• Arbitrary<’a> type and Arb module. Arbitrary, as before, packages up a generator and a shrinker. Coarbitrary was removed (without removing FsCheck’s functionality to generate functions). The Arb module is new, and plays a fairly central role in FsCheck now – a few useful functions that operate on a generator and a shrinker at the same time were added.
• Arb.Default type contains the default generators and shrinkers that ship with FsCheck. I’ve cleaned up most of the mess, but am not completely finished yet. Anyway, you should find many useful Arbitrary instances here, and they are now easily accessible (I believe in the previous version you had to type something like Arbitrary.Arbitrary.String().Arbitrary – now it is Arb.Default.String)
• Property type and Prop module. Here is where all the functions to build the actual properties live. Not much changes here, except that Prop.forAll now takes an Arbitrary instance instead of a generator – I believe this is a first step in making FsCheck more easily extendable to other kinds of generators (e.g. file based or exhaustive generators) by abstracting away the Gen type from Property. (This is currently only skin-deep though)
• Similar refactoring with running tests.
• Instead of quickCheck, verboseCheck, and all the other functions, there is now a Check type with relevant static methods that do the same thing. This made it possible to overload method names, which was annoying about the functions – I felt like I had to make up meaningless names. Also, the various checking types that you can do are now discoverable using intellisense by typing Check.<intellisense>.
• Similarly, the default configurations have now been added as static members on the Config record type, which is were you would expect them to be.
• The Runner module has had some cleanup as well, with some new names for old friends, and the IRunner interface has been extended a bit. Like the other modules, it should only expose what is useful to you, so it should be fairly easy to figure out what you need from intellisense alone.
• Output has been improved in various places.
• When registering generators, FsCheck now automatically overrides the previous generators, and also produces to some output where you can see what the new types that have been registered are, as well as what types were overridden. Hopefully this will help you fix those frustrating issues where types are not registered properly more easily.
• When checking all the members on a type or module, and an exception is thrown, the TargetInvocationExcception is now removed and you just get to see the actual exception, which reduces much of the noise.
• Various small output improvements.
• Size control has been changed in the Config. Instead of a function, you can now give a begin and end size and FsCheck will increase the size linearly between those two values. I did this to make the size independent of the number of test runs that you choose – the final size is always guaranteed to be the end size you ask for. So it is easy to change the number of tests while keeping the same size. I believe this is the least surprising result, as changing the number of tests should not directly influence the test size.
• Added many more tests, especially for the generators. I’m still not happy with the coverage, but we’re getting there.
• Wrote a simple documentation generator. I’m not very happy with it, but at least I reviewed all the documentation and at least the output shown in the documentation is now consistent with the actual output, and it is easy to keep that way.
• Cleaned up the examples a bit. The examples used in the documentation are now moved to the FsCheck.Documentation project (which also generated the codeplex formatted output).
• Typeclass simulation has had a pretty much bottom up rewrite. Hopefully this cleans up some nasty issues with static state I’ve seen, and also the API is I think much improved. Various bug fixes.

That’s pretty much it. I’ve got more in store still, but not sure when it will materialize into something that I can release. My spare time is getting scarce…but do post a comment on the forum or on my blog if you get stuck in moving your old FsCheck code to the new API, or any other feedback you may have.

Happy FsChecking!

Technorati: fscheck,f#,random testing

Share this post :

Technet!

del.icio.us it!

del.iri.ous!

digg it!

dotnetkicks it!

reddit!

technorati!

Moving FsCheck to Mercurial

I’ve been using Mercurial as my source control system of choice for a few years now, and I’ve been very  happy with it. The only problem I had was that codeplex, where FsCheck is hosted, only supported SVN. I used Mercurial for daily development, and when a release was imminent I just copied the final snapshot to my local SVN checkout and updated codeplex from there. Needless to say, the SVN repository was out of date and didn’t represent the actual history.

Well, no more! Codeplex now supports Mercurial as well. One email to the kind folks at codeplex support, and they wiped my SVN repository from codeplex and provided me with a shiny new hg repository. Great.

That left me with a problem: since I was used to the hg repository being private, I also dumped some, well, private stuff in it. Before you start imagining things, there is nothing special there – just a few pdfs of papers and such, but nonetheless things I didn’t want to put in a public repository. If only because some authors prefer that they alone control dissemination of their work.

And obviously Mercurial isn’t keen on letting you delete history.

Luckily it has the excellent convert extension. This is usually used to convert SVN, CVS or other repositories to a Mercurial repository. But you can also convert a Mercurial repository to a new one, and rewrite and update history along the way. One of the things the convert extension lets you do is to specify files and folders that should not be migrated. It also lets you remove branches, and even splice a certain set of changesets on a parent in the new repository.

I used the first two features to remove some superfluous history and files. I used the last because I’m used to clone my main repository to a bunch of feature repositories to work on specific features in relative isolation. I have a few of those in progress, but of course since I was converting my main repository to a new one the feature repositories couldn’t be used to push changes back to the new main. This makes them pretty much useless, unless they are converted as well.

So here’s what I did:

1. Convert the old main repository to the new main, stripping out files and branches along the way.
2. Clone the new repository for each feature I’m currently working on. Now I have a mirror image of my old folder where the old main and the old feature repositories are – except the changesets in the old feature repositories need to be migrated to the new feature repositories.
3. Using hg convert’s splice feature, and the excellent TortoiseHg repository explorer to find out the changeset numbers, I then additionally converted only the extra changesets from each old feature repository to its corresponding new feature repository. One thing to keep in mind is that hg convert uses the .hg/shamap file in the new repository to keep track of which changesets from the old repository it has already converted. This file is however not automatically copied when cloning – so I had to manually copy that over from the new main to the new feature repositories – otherwise convert would try to convert all of the changesets again.

Confused yet? Well, here’s my first Prezi to explain it all…

You can check out the result in FsCheck’s source tab – although the main result is that there is now much less friction in my development process.

Technorati: mercurial,hg convert,fscheck,codeplex

Share this post :

Technet!

del.icio.us it!

del.iri.ous!

digg it!

dotnetkicks it!

reddit!

technorati!

FsCheck 0.6.3 for Visual Studio 2010 RC and F# February 2010 CTP

A new release of F# means a new release of FsCheck.

• Removed the dependency on the F# PowerPack. This makes FsCheck easier to use, especially since the PowerPack is now  separately distributed. Besides, it was only used to pretty print functions.
• Cleaned up reflective generators a bit.
• All projects are upgraded to build with Visual Studio 2010 now. For backwards compatibility, I’ve added an FsCheck2008.sln solution which allows you to build in Visual Studio 2008 with the February 2010 CTP. You’ll get a warning from MSBuild saying that ToolsVersion 4.0 is not supported and that it is treating the build as a 3.5 build. Good – that’s exactly what you want. FsCheck is still targeting .NET3.5 until .NET4 is out.

Enjoy.

Technorati: FsCheck,random testing,F#

FsCheck 0.6.2: F# October 2009 CTP/2010 Beta 2

FsCheck is updated for the new F# release! Conversion was fairly straightforward, some name changes and indentation changes mostly. Furthermore, the following smaller changes: 

• A nice set of new default generators contributed by Howard Mansell. FsCheck can now generate any Enum instance, as well as generators for DateTime, arrays, positive and negative ints and so on. Check out the Arbitrary.fs file for a complete overview.
• Some more examples
• registerInstances and overwriteInstances now throws an exception if it cannot find any instances in the given type.

Happy FsChecking!

Announcing FsCheck 0.6.1

No functional changes, just an update as a result of the May CTP released by the F# team.

Get it here.

Change log:

• Many name changes as a result of name changes in the F# API
• Used the new formatting API, which gets rid of a bunch of obsolete warnings
• Tried to solve issues resulting from changed initialization order
• Added a standalone version of the FsCheck binary, to accommodate C#/VB users better
• Added a contributors file. Want your name in there? Then contribute ;)

Enjoy the new CTP!

Share this post :

Technet!

del.icio.us it!

del.iri.ous!

digg it!

dotnetkicks it!

reddit!

technorati!

How to test DSLs (and: FsChecking FsCheck)

There are two causes for this post.

First, I’ve been watching videos from Lang.NET and DSL DevCon, so I’m in language-mode. It occurred to me that (domain specific) languages seem difficult to test using traditional unit tests: after all, the usage possibilities are far greater for a language than for, say, a typical user interface (maybe this means that typical user interfaces suck - you decide). The word “combinator library”, which are internal DSLs avant la lettre, says it all: the usage possibilities are combinatorial. Classic, example based unit testing thus becomes combinatorially expensive as well.

Second, I’ve been saying in the last release announcement of FsCheck that I wanted to write some FsChecks for FsCheck itself. So far, this effort is about 60% complete. As a sort of challenge, I want to check as much as possible from FsCheck without any traditional example-based unit testing – in other words, I want to test FsCheck using only FsCheck (and maybe some Pex and Code Contracts later on).

FsCheck itself consists of basically two internal DSLs: one for constructing random value generators for whatever value it is your tests need, and one for constructing properties: basically assertions augmented with property combinators for adding labels, classifying generated values and such. It’s the latter DSL that I’m going to test, and draw some general conclusions that may apply to testing other DSLs or even general purpose languages as well.

To see what we’re dealing with, here’s an example of using the property DSL:

let prop_InsertCombined (x:int) xs = 
    ordered xs ==> (ordered (insert x xs))
        |> classify (ordered (x::xs)) "at-head"
        |> classify (ordered (xs @ [x])) "at-tail"
        |> collect (List.length xs)
quickCheck prop_InsertCombined

This property asserts that the insert function maintains the invariant that the list is ordered. The classify and collect property combinators are used to gather some information about the generated data. So in this test, there are three property combinators at work: implies (==>), classify and collect. Also, the arguments to the property (x and xs) are implicitly universally quantified using the forAll property combinator. These and more combinators in FsCheck can be combined an arbitrary ways, so testing this using example-based testing can be quite tedious. Can FsCheck do better?

The property DSL’s syntax tree

First, we’re going to need to generate an arbitrary property, i.e. an arbitrary combination of property combinators. To do that, I made a symbolic representation of the property language – a sort of abstract syntax tree of the language. It’s no secret that discriminated unions are great for this, and this case is not different:

type SymProp =  | Unit | Bool of bool | Exception
                | ForAll of int * SymProp
                | Implies of bool * SymProp
                | Classify of bool * string * SymProp
                | Collect of int * SymProp

So, a property can consist of, respectively:

• A function that returns the unit value;
• A function that returns a bool value;
• A function that throws an exception;
• A function that uses the implies combinator, which needs two arguments: a boolean condition and another property;
• A function that classifies the generated values, which needs three arguments: a boolean condition, the name to classify with if the condition is true and another property;
• A function that collects values, which takes two arguments: the value to collect, and another property.

Note that I’ve made some simplifications here: in fact forAll takes a generator that generates values of an arbitrary type – but since we’re not testing generators here, I’ve assumed the forAll always generates a constant int value. Also, collect would normally collect a (function of a) generated value, but it’s here constrained to collecting a constant int value.

We can build an actual property from this symbolic representation:

let rec private toProperty prop =
        match prop with
        | Unit -> property ()
        | Bool b -> property b
        | Exception -> property (lazy (raise <| InvalidOperationException()))
        | ForAll (i,prop) -> forAll (constant i) (fun i -> toProperty prop)
        | Implies (b,prop) -> b ==> (toProperty prop)
        | Classify (b,stamp,prop) -> classify b stamp (toProperty prop)
        | Collect (i,prop) -> collect i (toProperty prop)

The simplifications are plainly visible here.

Defining the semantics of properties

Now, let’s define a specification of what a symbolic property means, in terms of what it should produce as Result. Result is an internal datatype used by FsCheck to track the result of one execution of a property. This is the gist of it:

type Outcome = 
    | Timeout of int
    | Exception of exn
    | False
    | True
    | Rejected

type Result = 
    {   Outcome     : Outcome
        Stamp       : list<string>
        Labels      : Set<string>
        Arguments   : list<obj> }

So, a result tracks the outcome of a property: timed out, raised exception, returned false or true, or value rejected. The latter means a generated value did not pass the condition of an implies combinator. Furthermore, it also tracks the generated arguments, the stamps applied using classify and collect, as well as labels which are not tested here.

And here is a formal, executable specification of a subset of the FsCheck property DSL:

let rec private determineResult prop =
        let addStamp stamp res = { res with Stamp = stamp :: res.Stamp }
        let addArgument arg res = { res with Arguments = arg :: res.Arguments }
        match prop with
        | Unit -> succeeded
        | Bool true -> succeeded
        | Bool false -> failed
        | Exception  -> exc <| InvalidOperationException()
        | ForAll (i,prop) -> determineResult prop |> addArgument i
        | Implies (true,prop) -> determineResult prop
        | Implies (false,_) -> rejected
        | Classify (true,stamp,prop) -> determineResult prop |> addStamp stamp
        | Classify (false,_,prop) -> determineResult prop
        | Collect (i,prop) -> determineResult prop |> addStamp (any_to_string i)

You can check this definition with the FsCheck documentation. For example, it shows that an assertion that returns unit or true is interpreted as succeeded, false as failed and so on. The succeeded function and friends are defined in FsCheck and just construct basic Result instances:

let private result =
  { Outcome     = Rejected
  ; Stamp       = []
  ; Labels       = Set.empty
  ; Arguments   = []
  }

let internal failed = { result with Outcome = False }

The test

Armed with these functions, we can write our test:

let SymProperty (symprop:SymProp) = 
        let expected = determineResult symprop 
        let (MkRose (Common.Lazy actual,_)) = generate 1 (Random.newSeed()) (toProperty symprop) 
        areSame expected actual
        |> label (sprintf "expected = %A - actual = %A" expected actual)
        |> collect symprop

Ignore the specifics of the “actual” symbol definition – it executes an actual property and extracts the Result from it. The gist of the property is the comparison between the actual Result of the execution of the property (that was built from the symbolic property representation), and compare that with our expected semantics. Just to see what kind of properties we’re testing, I’ve also added a collect combinator, as well as a label to see what’s wrong if the test should fail. Thanks to FsCheck’s built-in discriminated union generator, running this gives:

Property.SymProperty-Ok, passed 100 tests.
22% Exception.
12% Unit.
6% Bool true.
6% Bool false.
4% Classify (false,"",Unit).
2% Implies (false,Exception).
2% Classify (true,"",Unit).
2% Classify (true,"",Exception).
2% Classify (false,"",Exception).
1% Implies (true,Exception).
1% Implies (true,Collect (2,Bool true)).
1% Implies (true,Collect (0,Unit)).
1% Implies (true,Collect (0,Collect (0,Bool false))).
(and so on)

Generating more interesting properties

This leaves something to be desired: the depth of combinations isn’t very high, and almost half of the generated properties are the trivial bool, unit or exception cases. One way of increasing this depth is by increasing the size of the generator, but that didn’t work very well: it mostly increased the length of the generated strings, without increasing the depth too much. The reason is that three of the seven options are “leaf” nodes, where generation ends. So I wrote my own generator for the SymProp type:

let rec private symPropGen =
        let rec recGen size =
            match size with
            | 0 -> oneof [constant Unit; liftGen (Bool) arbitrary; constant Exception]
            | n when n>0 ->
                let subProp = recGen (size/2)
                oneof   [ liftGen2 (curry ForAll) arbitrary (subProp)
                        ; liftGen2 (curry Implies) arbitrary (subProp)
                        ; liftGen2 (curry Collect) arbitrary (subProp)
                        ; liftGen3 (fun b s p -> Classify (b,s,p)) arbitrary arbitrary (subProp)]
            | _ -> failwith "symPropGen: size must be positive"
        sized recGen

If you’re somewhat familiar with FsCheck’s generator combinators, this shouldn’t be hard to read. The idea is that we only generate one of the leaf nodes if the size equals zero, and halve the size if we generate a “real” property combinator to ensure the generation ends at some point. Let’s change the property to incorporate this generator:

let Property = 
        forAllShrink symPropGen shrink (fun symprop ->
            let expected = determineResult symprop 
            let (MkRose (Common.Lazy actual,_)) = generate 1 (Random.newSeed()) (toProperty symprop) 
            areSame expected actual
            |> label (sprintf "expected = %A - actual = %A" expected actual)
            |> collect (depth symprop)

I added a depth function that calculates the depth of the generated symbolic property (i.e. depth of Unit, Bool and Exception is zero, and the rest adds one to the depth of it’s subproperty). Running this gives:

Property.get_Property-Ok, passed 100 tests.
26% 5.
26% 4.
18% 3.
12% 2.
7% 6.
6% 0.
4% 1.

which shows that FsCheck is now generating more properties with greater depth (4-5), and trivial properties only 6% of the time.

Conclusion

This post showed an approach for testing DSLs, combinator libraries or “language-oriented” programs, whatever you’d like to call it:

1. Write a symbolic representation of the abstract syntax tree of the language you want to test. This symbolic representation is used to abstract from details you don’t want to test, to define semantics of the language later on, and it to generate output for counter-examples that FsCheck may find.
2. Generate the actual representation in your compiler or interpreter from the symbolic representation.
3. Specify the semantics of the symbolic representation.
4. Rely on FsCheck’s built-in generators, or make your own generator to generate the symbolic representation.
5. Write an FsCheck property to compare the actual with the expected semantics.

This approach may even be usable for testing compilers of general purpose programming languages. The effort involved will be much larger, but it’s easy to start small.

Want more of this? Also check out the Checks.fs file in the FsCheck distribution for a complete test of the FsCheck property language.

Share this post :

Technet!

del.icio.us it!

del.iri.ous!

digg it!

dotnetkicks it!

reddit!

technorati!

Announcing FsCheck 0.6: Dot is the new pipe

It is about time to release a new version of FsCheck, and here it is.

What’s new in this release?

• A fluent interface for C# and VB

This is in prototype stage at the moment, and has some limitations with respect to the F# interface, but I would say it’s about 90% functional. An example of a property in C#:

Spec.ForAny<int, int[]>((x, xs) => xs.Insert(x).IsOrdered())
                .When((x, xs) => xs.IsOrdered())
                .Classify((x, xs) => new int[] { x }.Concat(xs).IsOrdered(), "at-head")
                .Classify((x, xs) => xs.Concat(new int[] { x }).IsOrdered(), "at-tail")
                .QuickCheck("InsertClassify");

This property is the equivalent of the following in F#:

let prop_InsertClassify (x:int) xs = 
    ordered xs ==> (ordered (insert x xs))
    |> classify (ordered (x::xs)) "at-head"
    |> classify (ordered (xs @ [x])) "at-tail" 
quickCheck prop_InsertClassify

As you can see, it’s about the same amount of code, but because of the fluent interface I have to add a class for each number of type arguments - currently limited to three. Ideally, I’d like to generate some code for classes up to six or nine, but for the moment it’s all copy-paste and you can only write properties with up to three parameters this way. For the rest, almost everything should be working: you can register generators, provide configuration, add shrinkers, and write generators using some combinators and LINQ:

var gen = from x in Any.OfType<int>()
          from y in Any.IntBetween(5, 10)
          where x > 5
          select new { Fst = x, Snd = y };

Pretty neat.

Incidentally, there is no C# code in FsCheck itself, and the fluent interface is all pure F# code. This was a good test to see how interoperable F# really is. Overall, the experience has been great, with just a few minor hiccups.

I translated most of the F# examples from the documentation to C# as well, you can find those in the F# source distribution.

• Possibility to replay a test run. FsCheck now displays the seed it used to start a test run when it fails. You can give this seed to a configuration parameter, so that FsCheck will repeat the run with exactly the same generated values:

check { quick with Name="Counter-replay"; Replay = Some <| Random.StdGen (395461793,1) } (asProperty spec)

• Various bug fixes and updates to xml docs.

• I’m making headway in testing FsCheck using FsCheck itself. It’s an interesting experience, and I have a blog post lined up about it.

The C#/VB “mainstream” support is something that I hope to continue to improve – I also plan to add reflective object generators and Gallio integration, mainly as a means to get some more attention for FsCheck. It’s also an interesting exercise to compare approaches. So far, if you know F#, I’d stick with it, but you can see that LINQ is occasionally very elegant for defining generators.

Happy FsChecking!

Share this post :

Technet!

del.icio.us it!

del.iri.ous!

digg it!

dotnetkicks it!

reddit!

technorati!