bsls_asserttest

Detailed Description

Provide a test facility for assertion macros.

Outline

• Purpose
• Classes
• Macros
• Description
  • Negative Testing
    • A Note On Exceptions
  • The Test Facility
    • Installing the Assert-Failure Handler
    • Basic Test Macros
    • Raw Test Macros
    • Enabling Negative Testing
    • Validating Disabled Macro Statements
    • Validating Macro Testing Levels
    • Addtional Test Pass Macros
  • Usage
    • Example 1: Testing Assertions In A Simple Vector Implementation
    • Example 2: Using PASS macros to help with formatting

Purpose

Provide a test facility for assertion macros.

Classes

• bsls::AssertTest: namespace for "assert" validating functions
• bsls::AssertTestHandlerGuard: guard for the negative testing assert-handler

Macros

• BSLS_ASSERTTEST_ASSERT_FAIL(STATEMENT): macro failure expected
• BSLS_ASSERTTEST_ASSERT_FAIL_RAW(STATEMENT): no origination check
• BSLS_ASSERTTEST_ASSERT_PASS(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_PASS_RAW(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_OPT_FAIL(STATEMENT): "opt" macro failure expected
• BSLS_ASSERTTEST_ASSERT_OPT_FAIL_RAW(STATEMENT): no origination check
• BSLS_ASSERTTEST_ASSERT_OPT_PASS(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_OPT_PASS_RAW(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_SAFE_FAIL(STATEMENT): "safe" macro failure expected
• BSLS_ASSERTTEST_ASSERT_SAFE_FAIL_RAW(STATEMENT): no origination check
• BSLS_ASSERTTEST_ASSERT_SAFE_PASS(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_SAFE_PASS_RAW(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL(STATEMENT): "invoke" macro expected
• BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL_RAW(STATEMENT): no origination check
• BSLS_ASSERTTEST_ASSERT_INVOKE_PASS(STATEMENT): macro success expected
• BSLS_ASSERTTEST_ASSERT_INVOKE_PASS_RAW(STATEMENT): macro success expected

See also

bsls_assert, bsls_asserttestexception

Description

This component provides a facility to test that BSLS_ASSERT_* macros are used as intended, in the appropriate build modes, and have the expected effects. The class bsls::AssertTest provides a small set of static methods that can be used to support detailed test cases, especially in table-driven test scenarios. Additionally, a set of macros automate use of these methods to support simple testing of single statements or expressions.

A testing-specific handler guard, bsls::AssertTestHandlerGuard, is also provided to be used wherever the BSLS_ASSERTTEST_* macros are used.

Negative Testing

"Negative testing" is the principle of testing for a negative result, which implies the function under test must fail in some way. Testable failures typically occur when a function is called with values outside the defined contract: a well-implemented function will validate function arguments, in appropriate build modes, using the various BSLS_ASSERT macros (see bsls_assert ). When a function fails as a result of an assertion, the default behavior is to terminate the program. However, the bsls_assert facility allows a user-supplied assertion-failure handler function to be installed, which can be used to build a test facility for expected assertions.

One important issue to be aware of with negative testing is that you are testing undefined behavior within a program. For the purpose of the test driver, the behavior of calling a function outside its contract is well- defined if it is guarded by assertions that are active in the current build mode. However, it is important that those tests are not run if the assert macros are not active, otherwise truly undefined behavior will result, with potentially disastrous consequences.

A Note On Exceptions

It is important to note that this facility relies on throwing and catching an exception in order to identify that an assertion has been violated, cleanup any objects created on the way to that assertion, and avoid executing any of the code after that assertion with deliberately bad input. This means that this component cannot be used to test assertions in functions that are noexcept, particular destructors that are implicitly noexcept in C++11 and beyond.

For most functions, if you have a narrow contract you should not be noexcept, as this is guaranteeing part of your behavior when your contract is violated (and actively preventing you from doing negative testing in this manner). For functions such as a destructor that are implicitly noexcept and greatly benefit from being so, it is advisable to move the checks into a separate validate method, and test destruction out of contract by just testing that the validate method asserts instead.

The Test Facility

Installing the Assert-Failure Handler

The function bsls::AssertTest::failTestDriver (and the parallel function bsls::AssertTest::failTestDriverByReview) is provided as the basis for a negative testing facility. It can act as an assertion-failure handler function that throws an exception, of type bsls::AssertTestException, containing the text of the failed assertion, the name of the file where it triggered, and the relevant line number within that file. The filename can be tested to ensure that the assertion was raised by the component under test, rather than by some deeper implementation detail as a consequence of the expected assertion not being present in the function under test.

Once the function bsls::AssertTest::failTestDriver has been registered as the active assertion-failure handler, a set of testing macros automate much of the boilerplate code involved in writing a negative test, so that a test can be effectively written as a single line. This is an important quality for reading tests, to clearly see the test logic in action without being distracted by the surrounding machinery.

Basic Test Macros

The five basic test macros are

• BSLS_ASSERTTEST_ASSERT_PASS
• BSLS_ASSERTTEST_ASSERT_SAFE_FAIL
• BSLS_ASSERTTEST_ASSERT_FAIL
• BSLS_ASSERTTEST_ASSERT_OPT_FAIL
• BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL Each of these macros takes a single statement or expression as an argument, tests whether an assertion is raised while evaluating that expression, and, if an assertion is both raised and expected, whether that assertion was raised by the component under test.

A test failure is indicated by invoking ASSERT(EXPRESSION), where ASSERT is either a macro or function that must be defined by the test driver, and EXPRESSION is an expression that evaluates to true or false according to whether the ASSERTTEST_ASSERT macro was expected to _PASS or _FAIL.

For example, if we have std::vector<int> v and v is empty, then the macro test BSLS_ASSERTTEST_ASSERT_SAFE_FAIL((v.back())) will fail when the effective assertion-level is BSLS_ASSERT_LEVEL_ASSERT_SAFE unless an assertion is raised. However, if the assertion-level is not BSLS_ASSERT_LEVEL_ASSERT_SAFE, then the test will not be run.

Raw Test Macros

The four "raw" test macros are

• BSLS_ASSERTTEST_ASSERT_SAFE_FAIL_RAW
• BSLS_ASSERTTEST_ASSERT_FAIL_RAW
• BSLS_ASSERTTEST_ASSERT_OPT_FAIL_RAW
• BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL_RAW These testing macros perform the same test as the corresponding basic testing macros, except that there is no check to confirm that the assertion originated in the component under test.

Enabling Negative Testing

In order to enable the negative testing facility, you must:

• #include this component header, bsls_asserttest.h.
• Supply an implementation of an ASSERT macro in your test driver.
• Register bsls::AssertTest::failTestDriver as the active assertion-failure handler (preferably with an instance of AssertTestHandlerGuard).

Validating Disabled Macro Statements

An additional external macro, BSLS_ASSERTTEST_VALIDATE_DISABLED_MACROS, can be defined to control the compile time behavior of bsls_asserttest . Enabling this macro configures all disabled asserttest macros to still instantiate their statements (in blocks that will not be evaluated) to be sure that the statement is still syntactically valid. This can be used to ensure tests that are rarely enabled remain syntactically valid.

Note: this behavior is currently always enabled and cannot be turned off.

Validating Macro Testing Levels

Another external macro, BSLS_ASSERTTEST_CHECK_LEVEL, can be used to add an additional check that the assertion that fails is of the same level or narrower than the macro testing the assertion. This will ensure that in all build modes where the assertion is enabled the test for that assertion will also be enabled.

Note that some variations of language contracts might not support the checking of levels when testing assertions, and in those cases the macro BSLS_ASSERTTEST_CAN_CHECK_LEVELS will not be defined.

Addtional Test Pass Macros

Seven additional PASS macros exist to parallel the remaining FAIL macros.

• BSLS_ASSERTTEST_ASSERT_SAFE_PASS
• BSLS_ASSERTTEST_ASSERT_SAFE_PASS_RAW
• BSLS_ASSERTTEST_ASSERT_PASS_RAW
• BSLS_ASSERTTEST_ASSERT_OPT_PASS
• BSLS_ASSERTTEST_ASSERT_OPT_PASS_RAW
• BSLS_ASSERTTEST_ASSERT_INVOKE_PASS
• BSLS_ASSERTTEST_ASSERT_INVOKE_PASS_RAW These macros are all functionally identical to BSLS_ASSERTTEST_ASSERT_PASS. They exist so that PASS checks format consistently with the corresponding negative tests they are associated with in a test driver. See {Example 2}, below, for how this can help formatting assertion testing code.

Usage

Example 1: Testing Assertions In A Simple Vector Implementation

First we will demonstrate how "negative testing" might be used to verify that the correct assertions are in place on std::vector::operator[]. We start by supplying a primitive vector-like class that offers the minimal set of operations necessary to demonstrate the test case.

template <class T>
class AssertTestVector {
    // This class simulates a 'std::vector' with a fixed capacity of 10
    // elements.
 
  private:
    // DATA
    T   d_data[10];
    int d_size;
 
  public:
    // CREATORS
    AssertTestVector();
        // Create an empty 'AssertTestVector' object.
 
    // MANIPULATORS
    void push_back(const T& value);
        // Append the specified 'value' to the back of this object.  The
        // behavior is undefined unless this method has been called fewer
        // than 10 times on this object.
 
    // ACCESSORS
    const T& operator[](int index) const;
        // Return a reference with non-modifiable access to the object at
        // the specified 'index' in this object.
};

Next we implement the support functions.

template <class T>
AssertTestVector<T>::AssertTestVector()
: d_data()
, d_size()
{
}
 
template<class T>
void AssertTestVector<T>::push_back(const T& value)
{
    BSLS_ASSERT_SAFE(d_size < 10);
 
    d_data[d_size] = value;
    ++d_size;
}

We conclude the definition of this support type with the implementation of the operator[] overload. Note the use of BSLS_ASSERT_SAFE, which is typical for function template definitions and inline function definitions. It is most appropriate in this case as the cost of evaluating each test is significant (> ~20%) compared to simply returning a reference to the result.

template <class T>
const T& AssertTestVector<T>::operator[](int index) const
{
    BSLS_ASSERT_SAFE(0 <= index);
    BSLS_ASSERT_SAFE(     index < d_size);
 
    return d_data[index];
}

Finally, we can write the function to test that the BSLS_ASSERT_SAFE macros placed in operator[] work as expected. We want to validate that the assertions trigger when the function preconditions are violated; we further want to validate that the assertion macros are enabled in the build modes that we expect. We start by defining some macro aliases that will make the test driver more readable. These macro aliases are a common feature of test drivers.

#define ASSERT_PASS(EXPR)      BSLS_ASSERTTEST_ASSERT_PASS(EXPR)
#define ASSERT_SAFE_FAIL(EXPR) BSLS_ASSERTTEST_ASSERT_SAFE_FAIL(EXPR)
#define ASSERT_FAIL(EXPR)      BSLS_ASSERTTEST_ASSERT_FAIL(EXPR)
#define ASSERT_OPT_FAIL(EXPR)  BSLS_ASSERTTEST_ASSERT_OPT_FAIL(EXPR)

Then we implement the test function itself. Note that we check that exceptions are available in the current build mode, as the test macros rely on the exception facility in order to return their diagnostic results. If exceptions are not available, there is nothing for a "negative test" to do. We also avoid warnings from UBSAN builds.

void testVectorArrayAccess()
{
#if defined(BDE_BUILD_TARGET_EXC) && !defined(BDE_BUILD_TARGET_UBSAN)
    bsls::AssertTestHandlerGuard g;
 
    AssertTestVector<void *> mA; const AssertTestVector<void *> &A = mA;
 
    ASSERT_SAFE_FAIL(mA[-1]);
    ASSERT_SAFE_FAIL(mA[ 0]);
    ASSERT_SAFE_FAIL(mA[ 1]);
 
    ASSERT_SAFE_FAIL( A[-1]);
    ASSERT_SAFE_FAIL( A[ 0]);
    ASSERT_SAFE_FAIL( A[ 1]);
 
    mA.push_back(0);  // increase the length to one
 
    ASSERT_SAFE_FAIL(mA[-1]);
    ASSERT_PASS     (mA[ 0]);
    ASSERT_SAFE_FAIL(mA[ 1]);
 
    ASSERT_SAFE_FAIL( A[-1]);
    ASSERT_PASS     ( A[ 0]);
    ASSERT_SAFE_FAIL( A[ 1]);
#else  // defined(BDE_BUILD_TARGET_EXC) && !defined(BDE_BUILD_TARGET_UBSAN)

If exceptions are not available, then we write a diagnostic message to the console alerting the user that this part of the test has not run, without failing the test.

    if (globalVerbose) printf(
                       "\tDISABLED in this (non-exception) build mode.\n");
 
#endif  // defined(BDE_BUILD_TARGET_EXC) && !defined(BDE_BUILD_TARGET_UBSAN)
}

Example 2: Using PASS macros to help with formatting

When testing the various inputs to a function to be sure that some trigger an assertion and some are in contract, it often helps to align the testing macros so that the various arguments are easily readable in relation to one another. We start by defining additional macro aliases to match the existing aliases already defined:

#define ASSERT_SAFE_PASS(EXPR) BSLS_ASSERTTEST_ASSERT_SAFE_PASS(EXPR)
#define ASSERT_OPT_PASS(EXPR)  BSLS_ASSERTTEST_ASSERT_OPT_PASS(EXPR)

Considering the function testVectorArrayAccess from {Example 1}, we could instead implement it without padded white space by using ASSERT_SAFE_PASS to replace ASSERT_PASS, matching the existing ASSERT_SAFE_FAIL tests, like this:

void testVectorArrayAccess2()
{
#if defined(BDE_BUILD_TARGET_EXC) && !defined(BDE_BUILD_TARGET_UBSAN)
    bsls::AssertTestHandlerGuard g;
 
    AssertTestVector<void *> mA; const AssertTestVector<void *> &A = mA;
 
    ASSERT_SAFE_FAIL(mA[-1]);
    ASSERT_SAFE_FAIL(mA[ 0]);
    ASSERT_SAFE_FAIL(mA[ 1]);
 
    ASSERT_SAFE_FAIL( A[-1]);
    ASSERT_SAFE_FAIL( A[ 0]);
    ASSERT_SAFE_FAIL( A[ 1]);
 
    mA.push_back(0);  // increase the length to one
 
    ASSERT_SAFE_FAIL(mA[-1]);
    ASSERT_SAFE_PASS(mA[ 0]);
    ASSERT_SAFE_FAIL(mA[ 1]);
 
    ASSERT_SAFE_FAIL( A[-1]);
    ASSERT_SAFE_PASS( A[ 0]);
    ASSERT_SAFE_FAIL( A[ 1]);
#endif  // defined(BDE_BUILD_TARGET_EXC) && !defined(BDE_BUILD_TARGET_UBSAN)
}

Typedefs
typedef bsls::AssertTestHandlerGuard	bsls_AssertTestHandlerGuard
typedef bsls::AssertTest	bsls_AssertTest

Macro Definition Documentation

BSLS_ASSERTTEST_ASSERT_ACTIVE_FLAG

#define BSLS_ASSERTTEST_ASSERT_ACTIVE_FLAG   false

BSLS_ASSERTTEST_ASSERT_FAIL

#define BSLS_ASSERTTEST_ASSERT_FAIL

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_FAIL_RAW

#define BSLS_ASSERTTEST_ASSERT_FAIL_RAW

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL

#define BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL_RAW

#define BSLS_ASSERTTEST_ASSERT_INVOKE_FAIL_RAW

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_INVOKE_PASS

#define BSLS_ASSERTTEST_ASSERT_INVOKE_PASS

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_INVOKE_PASS_RAW

#define BSLS_ASSERTTEST_ASSERT_INVOKE_PASS_RAW

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_OPT_ACTIVE_FLAG

#define BSLS_ASSERTTEST_ASSERT_OPT_ACTIVE_FLAG   false

BSLS_ASSERTTEST_ASSERT_OPT_FAIL

#define BSLS_ASSERTTEST_ASSERT_OPT_FAIL

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_OPT_FAIL_RAW

#define BSLS_ASSERTTEST_ASSERT_OPT_FAIL_RAW

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_OPT_PASS

#define BSLS_ASSERTTEST_ASSERT_OPT_PASS

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_OPT_PASS_RAW

#define BSLS_ASSERTTEST_ASSERT_OPT_PASS_RAW

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_PASS

#define BSLS_ASSERTTEST_ASSERT_PASS

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_PASS_RAW

#define BSLS_ASSERTTEST_ASSERT_PASS_RAW

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_SAFE_ACTIVE_FLAG

#define BSLS_ASSERTTEST_ASSERT_SAFE_ACTIVE_FLAG   false

BSLS_ASSERTTEST_ASSERT_SAFE_FAIL

#define BSLS_ASSERTTEST_ASSERT_SAFE_FAIL

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_SAFE_FAIL_RAW

#define BSLS_ASSERTTEST_ASSERT_SAFE_FAIL_RAW

(

STATEMENT_UNDER_TEST

)

BSLS_ASSERTTEST_DISABLED_IMP(STATEMENT_UNDER_TEST)

BSLS_ASSERTTEST_ASSERT_SAFE_PASS

#define BSLS_ASSERTTEST_ASSERT_SAFE_PASS

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_ASSERT_SAFE_PASS_RAW

#define BSLS_ASSERTTEST_ASSERT_SAFE_PASS_RAW

(

STATEMENT_UNDER_TEST

)

{ BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST) }

BSLS_ASSERTTEST_BRUTE_FORCE_IMP

#define BSLS_ASSERTTEST_BRUTE_FORCE_IMP	(		RESULT,
			LVL,
			STATEMENT_UNDER_TEST
	)

Value:

    {  \
    try {                                                                     \
        BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST)              \
                                                                              \
        ASSERT(BloombergLP::bsls::AssertTest::tryProbe(RESULT, LVL));         \
    }                                                                         \
    catch (const BloombergLP::bsls::AssertTestException& e) {                 \
        ASSERT(BloombergLP::bsls::AssertTest::catchProbe(                     \
                                            RESULT,                           \
                                            BSLS_ASSERTTEST_CHECK_LEVEL_ARG,  \
                                            LVL,                              \
                                            e,                                \
                                            __FILE__));                       \
    }                                                                         \
}

BSLS_ASSERTTEST_CAN_CHECK_LEVELS

#define BSLS_ASSERTTEST_CAN_CHECK_LEVELS

BSLS_ASSERTTEST_CHECK_LEVEL_ARG

#define BSLS_ASSERTTEST_CHECK_LEVEL_ARG   false

BSLS_ASSERTTEST_DISABLED_IMP

#define BSLS_ASSERTTEST_DISABLED_IMP

(

STATEMENT_UNDER_TEST

)

Value:

        while (false) do {                                                    \
            BSLS_ASSERTTEST_EVAL_STATEMENT_IMP(STATEMENT_UNDER_TEST)          \
        } while (false);

BSLS_ASSERTTEST_EVAL_STATEMENT_IMP

#define BSLS_ASSERTTEST_EVAL_STATEMENT_IMP

(

STATEMENT_UNDER_TEST

)

STATEMENT_UNDER_TEST;

BSLS_ASSERTTEST_IS_ACTIVE

#define BSLS_ASSERTTEST_IS_ACTIVE

(

TYPE

)

Value:

    (                                     \
    (   '\0' == TYPE[1]                                                       \
     || BSLS_ASSERTTEST_SAFE_2_BUILD_FLAG                                     \
    )                                                                         \
    &&                                                                        \
    (   ('S' == TYPE[0] && BSLS_ASSERTTEST_ASSERT_SAFE_ACTIVE_FLAG)           \
     || ('A' == TYPE[0] && BSLS_ASSERTTEST_ASSERT_ACTIVE_FLAG)                \
     || ('O' == TYPE[0] && BSLS_ASSERTTEST_ASSERT_OPT_ACTIVE_FLAG)            \
    )                                                                         \
)

BSLS_ASSERTTEST_RECURSIVELY_INCLUDED_TESTDRIVER_GUARD

#define BSLS_ASSERTTEST_RECURSIVELY_INCLUDED_TESTDRIVER_GUARD

BSLS_ASSERTTEST_SAFE_2_BUILD_FLAG

#define BSLS_ASSERTTEST_SAFE_2_BUILD_FLAG   false

BSLS_ASSERTTEST_VALIDATE_DISABLED_MACROS

#define BSLS_ASSERTTEST_VALIDATE_DISABLED_MACROS

Typedef Documentation

bsls_AssertTest

typedef bsls::AssertTest bsls_AssertTest

This alias is defined for backward compatibility.

bsls_AssertTestHandlerGuard

typedef bsls::AssertTestHandlerGuard bsls_AssertTestHandlerGuard

This alias is defined for backward compatibility.