Haskell

Connecting Haskell HUnit tests to Cabal TestSuite

This is a quick guide that should explain how to connect Haskell HUnit tests to the Cabal TestSuite. I’m going to explain the simplest possible method in which you could do so and then leave it up to you to take the method further. For this post I am going to be using the following versions of Haskell software:

We are going to go through this guide in the following order:

  • Understand the data structures in Cabal TestSuite and their expected use cases.
  • Understand the data structures in Test.HUnit
  • Walk through a simple way to combine the two.

After that is done then you should be able mesh the two together easily. If you already feel that you understand the first two sections then feel free to skip straight to the final section to see how the two are joined together.

Understanding the structure of Cabal TestSuite

I am going to attempt to connect them together for the sake of running all of my test cases easily using Cabal. It also means that other people reading my project for the first time will be able to follow conventions and run my test cases with ease. The first thing that you have to understand is the structure of the Cabal TestSuite. Here is the source code for Distribution.TestSuite:

data Test
    = Test TestInstance
    | Group
        { groupName     :: String
        , concurrently  :: Bool
            -- ^ If true, then children of this group may be run in parallel.
            -- Note that this setting is not inherited by children. In
            -- particular, consider a group F with "concurrently = False" that
            -- has some children, including a group T with "concurrently =
            -- True". The children of group T may be run concurrently with each
            -- other, as long as none are run at the same time as any of the
            -- direct children of group F.
        , groupTests    :: [Test]
        }
    | ExtraOptions [OptionDescr] Test

You will notice that this data structure has three constructors. The first constructor Test expects a single test instance, the Group constructor gathers a number of TestSuite test cases under the same name and also, interestingly, gives us the option to run the test cases concurrently. The final constructor ExtraOptions lets us pass extra options to test cases. For the sake of making the integration very simple so that you can get up and running in minutes we will be ignoring everything except the Test constructor. Now you may have noticed that the Test constructor expects a TestInstance but what does that even look like. Here is the code that makes up a TestInstance:

data TestInstance = TestInstance
    { run       :: IO Progress      -- ^ Perform the test.
    , name      :: String           -- ^ A name for the test, unique within a
                                    -- test suite.
    , tags      :: [String]         -- ^ Users can select groups of tests by
                                    -- their tags.
    , options   :: [OptionDescr]    -- ^ Descriptions of the options recognized
                                    -- by this test.
    , setOption :: String -> String -> Either String TestInstance
        -- ^ Try to set the named option to the given value. Returns an error
        -- message if the option is not supported or the value could not be
        -- correctly parsed; otherwise, a 'TestInstance' with the option set to
        -- the given value is returned.
    }

So a TestInstance expects to be able to run something that will return a progress, it expects to have a name, a list of zero or more tags, a list of zero or more options and a method that allows you to set new options and get back a modified test. The name is pretty self explanatory, it lets you name the test. The list of tags to selectively run different test cases is extremely useful and large companies that choose to write a ton of test cases and have them run in automated builds (like in Bamboo) will use this feature a massive amount to be very selective in the test cases that they run. The options are partially useful too but we will be ignoring them for the sake of speed of development. Out of all of these fields the run is perhaps the most interesting. Lets take a look at the Progress data structure that is supposed to be the result of the IO action:

data Progress = Finished Result
              | Progress String (IO Progress)

data Result = Pass
            | Fail String
            | Error String
  deriving (Eq, Read, Show)

Now the Progress data structure is actually quite interesting. It has one constructor that tells us that a test case has Finished and that the Result of the test was a Pass, Fail or Error. That is the easy part to understand, the next constructor seems to say that the progress of this test case is that it is still in Progress; it also provides a message and gives an IO action to continue the progress. This is a way of reporting progress of the test cases before the test case finishes; this would be really useful if you have some very long running test cases and you wanted to get them to give you messages sooner rather than later. For the sake of our simple test cases we will only be using the Finished constructor and the Result data structure to show success.

Now you understand how the Cabal Distribution expects Test cases to be structured. You have Test cases that have TestInstances. The TestInstances are capable of being run and reporting their Progress. Eventually the Progress will be Finished and you will be able to get a result which will be reported back to you on the screen. Now we have to learn the structure of the HUnit module so that we can figure out how to put the two together.

Understanding the structure of the HUnit Module

The Test.HUnit module provides some similar structures to write HUnit test cases. Specifically it has a Test data structure that looks like this:

-- | The basic structure used to create an annotated tree of test cases.
data Test
    -- | A single, independent test case composed.
    = TestCase Assertion
    -- | A set of @Test@s sharing the same level in the hierarchy.
    | TestList [Test]
    -- | A name or description for a subtree of the @Test@s.
    | TestLabel String Test

As you can see we have a tree based data structure with a TestCase being the smallest node that can be defined. Each test case asserts something. You can also have a list of tests that make up one larger test and you can use labels to give names to other tests or groups of tests. I’m going to just assume that you know how to write HUnit test cases (as that is not the point of this document) but how do you run a test case?

You could use the performTest function but it is quite low level and would take to long to wire up. So instead we can use the runTestTT function to do the job for us with a much simpler return type:

-- | Provides the "standard" text-based test controller. Reporting is made to
--   standard error, and progress reports are included. For possible
--   programmatic use, the final counts are returned.
--
--   The "TT" in the name suggests "Text-based reporting to the Terminal".

runTestTT :: Test -> IO Counts
runTestTT t = do (counts', 0)                  return counts'

As you can see we just give this function our test cases and it gives us the results in the form of a Counts object that looks like this:

-- | A data structure that hold the results of tests that have been performed
-- up until this point.
data Counts = Counts { cases, tried, errors, failures :: Int }
  deriving (Eq, Show, Read)

From this Counts object we can derive the number of test cases that we tried to run as well as any failures or errors that occurred. We will use that in the next section to join the two together.

Combining HUnit and Cabal TestSuite together

Now that we know how it all works we can, quite easily, join the two together. Now because both Cabal TestSuite and HUnit define a Test data structure we need to have qualified imports and that makes the code a little harder to read but I am sure that you will see straight through it. Therefore I am just going to show you the final results and then just highlight the important sections:

module Test where

import qualified Distribution.TestSuite as TS
import qualified Test.HUnit as HU

test1 = HU.TestCase (HU.assertEqual "one equals three" 1 3)

hunitTests = HU.TestList [HU.TestLabel "Test 1" test1]

runHUnitTests :: HU.Test -> IO TS.Progress
runHUnitTests tests = do
   (HU.Counts cases tried errors failures) <- HU.runTestTT tests return $ if errors > 0
      then TS.Finished $ TS.Error "There were errors in the HUnit tests"
      else if failures > 0
         then TS.Finished $ TS.Fail "There were failures in the HUnit tests"
         else TS.Finished TS.Pass

tests :: IO [TS.Test]
tests = return [ TS.Test hunit ]
  where
    hunit = TS.TestInstance
        { TS.run = runHUnitTests hunitTests
        , TS.name = "HUnit Test Cases"
        , TS.tags = ["hunit"]
        , TS.options = []
        , TS.setOption = _ _ -> Right hunit
        }

Breaking down the code segment above:

  • The first two highlights are lines 3 and 4 which show us importing the TestSuite as TS and the HUnit test classes as HU. You’ll have to remember that for the remainder of the tests.
  • The next highlight is the definition of the hunitTests which are HUnit Test classes. You can provide this wherever you want and, to make your code nicer to read, you should probably split this off into a separate test module and then import it back in.
  • The next highlight shows the function that does the majority of the work. It runs the test cases using runTestTT and then inspects the result counts to figure out what Result type to return back to the Cabal TestSuite.
  • In the next highlight you can see that we are using the runHUnitTests function to provide a runner for the TestInstance. At this point in time we have fully connected our HUnit test cases and our Cabal TestSuite and you should be able to “cabal install –enable-tests” and watch your HUnit test cases run.

That is all for this guide and I hope that it help you get your test cases up and running quickly.

Further reading: This is obviously a very simple integration of the two and you have plenty of room to improve that integration dramatically. However, I have spotted a library that seems to be in development (and not on Hackage) called cabal-test-hunit that you might want to look at for more information.