bsls_asserttest Component

Provide a test facility for assertion macros. More...

Namespaces
namespace	bsls

Detailed Description

Outline

Purpose
Classes
Macros
Description

Purpose:: Provide a test facility for assertion macros.

Classes:

bsls::AssertTest	namespace for "assert" validating functions
bsls::AssertTestHandlerGuard	guard for the negative testing assert-handler

Macros:

BSLS_ASSERTTEST_ASSERT_FAIL(EXPRESSION)	macro failure expected
BSLS_ASSERTTEST_ASSERT_FAIL_RAW(EXPRESSION)	no origination check
BSLS_ASSERTTEST_ASSERT_PASS(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_PASS_RAW(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_OPT_FAIL(EXPRESSION)	"opt" macro failure expected
BSLS_ASSERTTEST_ASSERT_OPT_FAIL_RAW(EXPRESSION)	no origination check
BSLS_ASSERTTEST_ASSERT_OPT_PASS(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_OPT_PASS_RAW(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_SAFE_FAIL(EXPRESSION)	"safe" macro failure expected
BSLS_ASSERTTEST_ASSERT_SAFE_FAIL_RAW(EXPRESSION)	no origination check
BSLS_ASSERTTEST_ASSERT_SAFE_PASS(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_SAFE_PASS_RAW(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL(EXPRESSION)	"invoke" macro expected
BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL_RAW(EXPRESSION)	no origination check
BSLS_ASSERTTEST_ASSERT_INVOKE_PASS(EXPRESSION)	macro success expected
BSLS_ASSERTTEST_ASSERT_INVOKE_PASS_RAW(EXPRESSION)	macro success expected

See also:: Component bsls_assert, Component bsls_asserttestexception

Description:: This component provides a facility to test that BSLS_ASSERT_* macros are used as intended, in the appropriate build modes, and have the expected effects. The class bsls::AssertTest provides a small set of static methods that can be used to support detailed test cases, especially in table-driven test scenarios. Additionally, a set of macros automate use of these methods to support simple testing of single expressions.

: A testing-specific handler guard, bsls::AssertTestHandlerGuard, is also provided to be used wherever the BSLS_ASSERTTEST_* macros are used.

Negative Testing:: "Negative testing" is the principle of testing for a negative result, which implies the function under test must fail in some way. Testable failures typically occur when a function is called with values outside the defined contract: a well-implemented function will validate function arguments, in appropriate build modes, using the various BSLS_ASSERT macros (see bsls_assert). When a function fails as a result of an assertion, the default behavior is to terminate the program. However, the bsls_assert facility allows a user-supplied assertion-failure handler function to be installed, which can be used to build a test facility for expected assertions.

: One important issue to be aware of with negative testing is that you are testing undefined behavior within a program. For the purpose of the test driver, the behavior of calling a function outside its contract is well- defined if it is guarded by assertions that are active in the current build mode. However, it is important that those tests are not run if the assert macros are not active, otherwise truly undefined behavior will result, with potentially disastrous consequences.

A Note On Exceptions:: It is important to note that this facility relies on throwing and catching an exception in order to identify that an assertion has been violated, cleanup any objects created on the way to that assertion, and avoid executing any of the code after that assertion with deliberately bad input. This means that this component cannot be used to test assertions in functions that are noexcept, particular destructors that are implicitly noexcept in C++11 and beyond.

: For most functions, if you have a narrow contract you should not be noexcept, as this is guaranteeing part of your behavior when your contract is violated (and actively preventing you from doing negative testing in this manner). For functions such as a destructor that are implicitly noexcept and greatly benefit from being so, it is advisable to move the checks into a separate validate method, and test destruction out of contract by just testing that the validate method asserts instead.

The Test Facility:

Installing the Assert-Failure Handler:: The function bsls::AssertTest::failTestDriver (and the parallel function bsls::AssertTest::failTestDriverByReview) is provided as the basis for a negative testing facility. It can act as an assertion-failure handler function that throws an exception, of type bsls::AssertTestException, containing the text of the failed assertion, the name of the file where it triggered, and the relevant line number within that file. The filename can be tested to ensure that the assertion was raised by the component under test, rather than by some deeper implementation detail as a consequence of the expected assertion not being present in the function under test.

: Once the function bsls::AssertTest::failTestDriver has been registered as the active assertion-failure handler, a set of testing macros automate much of the boilerplate code involved in writing a negative test, so that a test can be effectively written as a single line. This is an important quality for reading tests, to clearly see the test logic in action without being distracted by the surrounding machinery.

Basic Test Macros:

The five basic test macros are

BSLS_ASSERTTEST_ASSERT_PASS
BSLS_ASSERTTEST_ASSERT_SAFE_FAIL
BSLS_ASSERTTEST_ASSERT_FAIL
BSLS_ASSERTTEST_ASSERT_OPT_FAIL
BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL

Each of these macros takes a single expression as an argument, tests whether an assertion is raised while evaluating that expression, and, if an assertion is both raised and expected, whether that assertion was raised by the component under test.

: A test failure is indicated by invoking ASSERT(EXPRESSION), where ASSERT is either a macro or function that must be defined by the test driver, and EXPRESSION is an expression that evaluates to true or false according to whether the ASSERTTEST_ASSERT macro was expected to _PASS or _FAIL.

: For example, if we have std::vector<int> v and v is empty, then the macro test BSLS_ASSERTTEST_ASSERT_SAFE_FAIL((v.back())) will fail when the effective assertion-level is BSLS_ASSERT_LEVEL_ASSERT_SAFE unless an assertion is raised. However, if the assertion-level is not BSLS_ASSERT_LEVEL_ASSERT_SAFE, then the test will not be run.

Raw Test Macros:

The four "raw" test macros are

BSLS_ASSERTTEST_ASSERT_SAFE_FAIL_RAW
BSLS_ASSERTTEST_ASSERT_FAIL_RAW
BSLS_ASSERTTEST_ASSERT_OPT_FAIL_RAW
BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL_RAW

These testing macros perform the same test as the corresponding basic testing macros, except that there is no check to confirm that the assertion originated in the component under test.

Enabling Negative Testing:

In order to enable the negative testing facility, you must:

#include this component header, bsls_asserttest.h.
Supply an implementation of an ASSERT macro in your test driver.
Register bsls::AssertTest::failTestDriver as the active assertion-failure handler (preferably with an instance of AssertTestHandlerGuard).

Validating Disabled Macro Expressions:: An additional external macro, BSLS_ASSERTTEST_VALIDATE_DISABLED_MACROS, can be defined to control the compile time behavior of bsls_asserttest. Enabling this macro configures all disabled asserttest macros to still instantiate their expressions (in a non-evaluated context) to be sure that the expression is still syntactically valid. This can be used to ensure tests that are rarely enabled have valid expressions.

Validating Macro Testing Levels:: Another external macro, BSLS_ASSERTTEST_CHECK_LEVEL, can be used to add an additional check that the assertion that fails is of the same level or narrower than the macro testing the assertion. This will ensure that in all build modes where the assertion is enabled the test for that assertion will also be enabled.

: Note that some variations of language contracts might not support the checking of levels when testing assertions, and in those cases the macro BSLS_ASSERTTEST_CAN_CHECK_LEVELS will not be defined.

Addtional Test Pass Macros:

Seven additional PASS macros exist to parallel the remaining FAIL macros.

BSLS_ASSERTTEST_ASSERT_SAFE_PASS
BSLS_ASSERTTEST_ASSERT_SAFE_PASS_RAW
BSLS_ASSERTTEST_ASSERT_PASS_RAW
BSLS_ASSERTTEST_ASSERT_OPT_PASS
BSLS_ASSERTTEST_ASSERT_OPT_PASS_RAW
BSLS_ASSERTTEST_ASSERT_INVOKE_PASS
BSLS_ASSERTTEST_ASSERT_INVOKE_PASS_RAW

These macros are all functionally identical to BSLS_ASSERTTEST_ASSERT_PASS. They exist so that PASS checks format consistently with the corresponding negative tests they are associated with in a test driver. See Example 2, below, for how this can help formatting assertion testing code.

Usage:

Example 1: Testing Assertions In A Simple Vector Implementation:

First we will demonstrate how "negative testing" might be used to verify that the correct assertions are in place on std::vector::operator[]. We start by supplying a primitive vector-like class that offers the minimal set of operations necessary to demonstrate the test case.

  template <class T>
  class AssertTestVector {
      // This class simulates a 'std::vector' with a fixed capacity of 10
      // elements.

    private:
      // DATA
      T   d_data[10];
      int d_size;

    public:
      // CREATORS
      AssertTestVector();
          // Create an empty 'AssertTestVector' object.

      // MANIPULATORS
      void push_back(const T& value);
          // Append the specified 'value' to the back of this object.  The
          // behavior is undefined unless this method has been called fewer
          // than 10 times on this object.

      // ACCESSORS
      const T& operator[](int index) const;
          // Return a reference with non-modifiable access to the object at
          // the specified 'index' in this object.
  };

Next we implement the support functions.

  template <class T>
  AssertTestVector<T>::AssertTestVector()
  : d_data()
  , d_size()
  {
  }

  template<class T>
  void AssertTestVector<T>::push_back(const T& value)
  {
      BSLS_ASSERT_SAFE(d_size < 10);

      d_data[d_size] = value;
      ++d_size;
  }

We conclude the definition of this support type with the implementation of the operator[] overload. Note the use of BSLS_ASSERT_SAFE, which is typical for function template definitions and inline function definitions. It is most appropriate in this case as the cost of evaluating each test is significant (> ~20%) compared to simply returning a reference to the result.

  template <class T>
  const T& AssertTestVector<T>::operator[](int index) const
  {
      BSLS_ASSERT_SAFE(0 <= index);
      BSLS_ASSERT_SAFE(     index < d_size);

      return d_data[index];
  }

Finally, we can write the function to test that the BSLS_ASSERT_SAFE macros placed in operator[] work as expected. We want to validate that the assertions trigger when the function preconditions are violated; we further want to validate that the assertion macros are enabled in the build modes that we expect. We start by defining some macro aliases that will make the test driver more readable. These macro aliases are a common feature of test drivers.

  #define ASSERT_PASS(EXPR)      BSLS_ASSERTTEST_ASSERT_PASS(EXPR)
  #define ASSERT_SAFE_FAIL(EXPR) BSLS_ASSERTTEST_ASSERT_SAFE_FAIL(EXPR)
  #define ASSERT_FAIL(EXPR)      BSLS_ASSERTTEST_ASSERT_FAIL(EXPR)
  #define ASSERT_OPT_FAIL(EXPR)  BSLS_ASSERTTEST_ASSERT_OPT_FAIL(EXPR)

Then we implement the test function itself. Note that we check that exceptions are available in the current build mode, as the test macros rely on the exception facility in order to return their diagnostic results. If exceptions are not available, there is nothing for a "negative test" to do.

  void testVectorArrayAccess()
  {
  #ifdef BDE_BUILD_TARGET_EXC
      bsls::AssertTestHandlerGuard g;

      AssertTestVector<void *> mA; const AssertTestVector<void *> &A = mA;

      ASSERT_SAFE_FAIL(mA[-1]);
      ASSERT_SAFE_FAIL(mA[ 0]);
      ASSERT_SAFE_FAIL(mA[ 1]);

      ASSERT_SAFE_FAIL( A[-1]);
      ASSERT_SAFE_FAIL( A[ 0]);
      ASSERT_SAFE_FAIL( A[ 1]);

      mA.push_back(0);  // increase the length to one

      ASSERT_SAFE_FAIL(mA[-1]);
      ASSERT_PASS     (mA[ 0]);
      ASSERT_SAFE_FAIL(mA[ 1]);

      ASSERT_SAFE_FAIL( A[-1]);
      ASSERT_PASS     ( A[ 0]);
      ASSERT_SAFE_FAIL( A[ 1]);
  #else   // defined(BDE_BUILD_TARGET_EXC)

If exceptions are not available, then we write a diagnostic message to the console alerting the user that this part of the test has not run, without failing the test.

      if (globalVerbose) printf(
                         "\tDISABLED in this (non-exception) build mode.\n");

  #endif  // !defined(BDE_BUILD_TARGET_EXC)
  }

Example 2: Using PASS macros to help with formatting:

When testing the various inputs to a function to be sure that some trigger an assertion and some are in contract, it often helps to align the testing macros so that the various arguments are easily readable in relation to one another. We start by defining additional macro aliases to match the existing aliases already defined:

  #define ASSERT_SAFE_PASS(EXPR) BSLS_ASSERTTEST_ASSERT_SAFE_PASS(EXPR)
  #define ASSERT_OPT_PASS(EXPR)  BSLS_ASSERTTEST_ASSERT_OPT_PASS(EXPR)

Considering the function testVectorArrayAccess from Example 1, we could instead implement it without padded white space by using ASSERT_SAFE_PASS to replace ASSERT_PASS, matching the existing ASSERT_SAFE_FAIL tests, like this:

  void testVectorArrayAccess2()
  {
  #ifdef BDE_BUILD_TARGET_EXC
      bsls::AssertTestHandlerGuard g;

      AssertTestVector<void *> mA; const AssertTestVector<void *> &A = mA;

      ASSERT_SAFE_FAIL(mA[-1]);
      ASSERT_SAFE_FAIL(mA[ 0]);
      ASSERT_SAFE_FAIL(mA[ 1]);

      ASSERT_SAFE_FAIL( A[-1]);
      ASSERT_SAFE_FAIL( A[ 0]);
      ASSERT_SAFE_FAIL( A[ 1]);

      mA.push_back(0);  // increase the length to one

      ASSERT_SAFE_FAIL(mA[-1]);
      ASSERT_SAFE_PASS(mA[ 0]);
      ASSERT_SAFE_FAIL(mA[ 1]);

      ASSERT_SAFE_FAIL( A[-1]);
      ASSERT_SAFE_PASS( A[ 0]);
      ASSERT_SAFE_FAIL( A[ 1]);
  #endif  // defined(BDE_BUILD_TARGET_EXC)
  }

Component bsls_asserttest [Package bsls]

Namespaces

Detailed Description

Component bsls_asserttest
[Package bsls]