C++ Unit Testing Frameworks

Unit testing is focused on the smallest testable part of an application. A unit test exercises an individual portion of code, to determine if it functions properly.

Unit testing:

  • reduces the cost of software – defects are discovered during the development stage and can be fixed prior to the release to the test team;
  • promotes interface-based approach of implementing components. The interfaces are adding abstraction, that makes the code flexible;
  • ensures that the software functions as specified by the requirements.

Ideally the unit tests should be developed prior to the development of the unit that’s a subject of the test. This technique is known as test-driven development. Under this approach, the public API is defined first (based on the requirements), then skeleton classes and unit tests are developed. This puts the developer in the position of a client of his/her own code, giving him/her the opportunity to asses and correct the API (if needed).

It’s recommended that individual modules are tested in isolation, to avoid an impact from other modules. The modules, required by the unit under test, are stubbed. The stubs have predefined behavior and their logic does not interfere with the unit under test. Unfortunately writing stubs can be difficult, time consuming and sometimes even impossible. That’s why some people prefer to use another approach to unit testing, that does not isolate the unit under test from the rest of the system. Of course the lower level units used, need to have proper unit tests as well. This form of unit testing serves as integration testing as well, because the higher level layers invoke the lower levels modules in order to do their work.

The unit tests should be executed as a part of an automated build process. Thus if the code does not fulfill the contract or the unit tests are out of date, the build will fail and the developer(s), responsible for the failure will correct the errors. This ensures that the software is in a testable state, and does not contain major errors that can be caught by the unit tests.

In my opinion unit tests need to be executed before each check-in in the source control system. This way the developer will make sure that his/her changes will not break the build.

Unit Testing Frameworks

A unit testing framework simplifies the development of unit tests. It allows the developer to focus on the unit tests and not bother about supporting logic (for execution, error handling, UI, etc). The framework is configured which tests to run, then executes them in sequence. The unit tests should reside in the workspace of the module, that is being tested .

Most (if not all) unit testing frameworks are based on Kent Beck‘s SUnit. The C++ unit testing frameworks are not an exception from this rule.

CppUnit vs. CxxTest

CppUnit vs. CxxTest
CppUnit is described as a C++ port of JUnit. The library can be compiled for a variety of platforms
License: LGPL
CxxTest is a JUnit/CppUnit/xUnit-like framework for C++, that is designed to be as portable as possible. It requires Perl or Python.
License: LGPL

Test suites

Test cases are grouped into test suits. Each test case is a method in the test suite class. A typical naming convention is test[method], where [method] is the name of the method, being tested.

To create a test suite subclass the CppUnit::TestFixture class.

#include <TestFixture.h>

class ATest : public CppUnit::TestFixture  {
public:
   void testMethodA( )
};
vs. Test suites are written in header files. To create a test suite subclass the CxxTest::TestSuite class.

#include <cxxtest/TestSuite.h>

class ATest : public CxxTest::TestSuite
{
public:
   void testMethodA( void )
};

Test for expected results

In both cases there are several assertions available, that can be used to check return values, whether or not an exception is thrown, etc.

CPPUNIT_FAIL(message)
CPPUNIT_ASSERT(condition)
CPPUNIT_ASSERT_MESSAGE(message, condition)
CPPUNIT_ASSERT_EQUAL(expected, actual)
CPPUNIT_ASSERT_EQUAL_MESSAGE(message, expected, actual)
CPPUNIT_ASSERT_DOUBLES_EQUAL(expected, actual, delta)
CPPUNIT_ASSERT_THROW(expression, ExceptionType)
CPPUNIT_ASSERT_NO_THROW(expression)
CPPUNIT_ASSERT_ASSERTION_FAIL(assertion)
CPPUNIT_ASSERT_ASSERTION_PASS(assertion)
vs.
TS_FAIL(message)
TS_ASSERT(expr )
TS_ASSERT_EQUALS(x, y)
TS_ASSERT_SAME_DATA(x, y, size)
TS_ASSERT_DIFFERS(x, y)
TS_ASSERT_LESS_THAN(x, y)
TS_ASSERT_LESS_THAN_EQUALS(x, y)
TS_ASSERT_PREDICATE(R, x)
TS_ASSERT_RELATION(R, x, y)
TS_ASSERT_THROWS(expr, type)
TS_ASSERT_THROWS_EQUALS(expr, arg, x, y)
TS_ASSERT_THROWS_ASSERT(expr, arg, assertion)
TS_ASSERT_THROWS_ANYTHING(expr)
TS_ASSERT_THROWS_NOTHING(expr)
TS_WARN(message)
TS_TRACE(message)

Test fixtures

Usually many tests for the same module have some common code for initialisation and finalisation. This code can be creating/destroying objects, files, etc. It can be placed in the overridden methods setUp()/tearDown(), which are executed around each test case.

class ATest : public ...  {
public:
   void testMethodA( )
   void setUp() {
      m_pA = new A();
   }
   void tearDown() {
      delete m_pA;
   }
protected:
   A*   m_pA;
};

Test runners

In order to get executed your test suite needs to be registered with the test runner.

There are several ways to register your test suits with the test runner. For example you can use a macro in your test suite source file:

#include <cppunit/extensions/HelperMacros.h>
CPPUNIT_TEST_SUITE_REGISTRATION( ATest )

To run the tests add the following code to the main function of the test application:

CppUnit::TextUi::TestRunner runner;
CppUnit::TestFactoryRegistry &registry = CppUnit::TestFactoryRegistry::getRegistry();
runner.addTest( registry.makeTest() );
runner.run();
vs. Test runners are generated by a Perl or Python script.

There are various types of test runners:

  • XML output test runner;
  • Compiler-like text output (for IDE integration);
  • MFC and QT GUI test runners;

The GUI test runners allow you to choose tests to run, also they display the result from the execution of each test.

vs.
  • Compiler-like text output (for IDE integration);
  • Win32, X11 and QT GUI test runners;

The GUI test runners have limited UI. They display a progress bar only. The results are written to the standard output.

My choice

When I was doing a research for a unit testing framework, I evaluated both CppUnit and CxxTest. Personally, I like CxxTest more, because:

  • it’s lightweight;
  • you don’t need to link external libraries;
  • you don’t need to register your tests.

Please, note that you will need Perl or Python. If you are using a UNIX-compatible system, most likely you’ll have Perl already installed.
Also CppUnit has some advanced functionality that’s not present in CxxTest. The fact that it’s lightweight comes at a price.

Final thoughts

  • The unit tests should include positive (successful), negative (error ) test cases and tests for boundary conditions.
  • The unit tests need to be reviewed and updated to reflect the changes in the project’s requirements.
  • Combine unit testing with code coverage analysis, so that you can measure how well the code is exercised by a test suite.
  • Don’t rely on the environment in the tests, they can be executed anywhere.
You can leave a response, or trackback from your own site.

2 responses to “C++ Unit Testing Frameworks”

  1. Edd says:

    “Most (if not all) unit testing frameworks are based on Kent Beck‘s SUnit. The C++ unit testing frameworks are not an exception from this rule”.

    Actually, I’d say the good C++ “frameworks” are exceptions to this rule. Both the frameworks you evaluate are rather heavy on boiler plate and rather unidiomatic. For example, why do tests have to be methods on classes? And given that destructors in C++ are executed deterministically, why not use constructors and destructors for SetUp and TearDown? That’s exactly what *tors are for, after all.

    These anti-idioms are a result of directly mapping the Java incarnation on to similar constructs in C++ without actually pausing to ponder if there might be a better way. Michael Feathers, the original author of CppUnit, realised this and came up with CppUnitLite.

    Others minimalist testing libraries include UnitTest++ and test-o-matic (mine).

  2. SamVimes says:

    That’s a good point.
    Unfortunately, I’m not familiar with the frameworks you mentioned. I’ll have a look, when I have the time.
    Just today, I found about Google Test, but it is xUnit like too.

Leave a Reply