bsls_fuzztest

Detailed Description

Provide macros for use in fuzz testing narrow-contract functions.

Outline

• Purpose
• Classes
• Macros
• Description
  • Usage
    • Example: Basic Usage of Macros

Purpose

Provide macros for use in fuzz testing narrow-contract functions.

Classes

• bsls::FuzzTestPreconditionTracker: utility for tracking assert violations
• bsls::FuzzTestHandlerGuard: guard for fuzz testing assert-/review-handler

Macros

• BSLS_FUZZTEST_EVALUATE(EXPRESSION): wrapper for narrow contract function
• BSLS_FUZZTEST_EVALUATE_RAW(EXPRESSION): wrapper with no origination check

See also

bsls_preconditions

Description

This component provides two macros, BSLS_FUZZTEST_EVALUATE and BSLS_FUZZTEST_EVALUATE_RAW, that can be used in fuzz testing narrow contract functions. They are intended to be used in conjunction with bsls::FuzzTestHandlerGuard as well as BSLS_PRECONDITIONS_BEGIN and BSLS_PRECONDITIONS_END.

When fuzzing narrow contract functions, if we do not wish to "massage" the data we pass to the function (as this may be error-prone and might introduce bias into the tested input) we must address the issue that we will often invoke the function out of contract, and this will cause the function to assert/review, and the test to end prematurely. The macros defined in this component solve this issue by detecting the location of precondition violations. Functions with narrow contracts that are to be tested must be decorated with the BSLS_PRECONDITIONS_BEGIN and BSLS_PRECONDITIONS_END macros. These macros must be placed just before and after the function whose preconditions are checked.

All these macros are intended to be used in fuzzing builds in which BDE_ACTIVATE_FUZZ_TESTING is defined. For our purposes, those preconditions that fail in the function under test (i.e., the one invoked by BSLS_FUZZTEST_EVALUATE) are treated differently from all other precondition failures. We refer to these preconditions as "top-level" preconditions. If a top-level precondition fails – and the assertion/review is not from another component – the execution will continue: we do not wish to stop the fuzz test if we simply invoked the narrow contract function under test out of contract. We wish to detect only subsequent assertions/reviews (i.e., not in the top-level), or assertions/reviews from other components.

The BSLS_FUZZTEST_EVALUATE_RAW macro does not check if the assertion/review originates from another component, though, like the non-RAW version, it ignores only top-level assertions/reviews. This behavior is desirable in cases in which a function delegates its implementation and associated precondition checks to a different component. In such cases, a precondition failure ought not cause the fuzz test to end.

Usage

This section illustrates intended use of this component.

Example: Basic Usage of Macros

The macros in this component rely upon the presence of related macros from bsls_preconditions . The fuzzing macros are typically used in a fuzzing build, in which case the entry point is LLVMFuzzerTestOneInput.

In this example, we illustrate the intended usage of two macros: BSLS_FUZZTEST_EVALUATE and BSLS_FUZZTEST_EVALUATE_RAW.

First, in order to illustrate the use of BSLS_FUZZTEST_EVALUATE, we define two functions that implement the sqrt function, both decorated with the precondition BEGIN and END macros. mySqrt forwards its argument to newtonsSqrt, which has a slightly more restrictive precondition: mySqrt accepts 0, while newtonsSqrt does not.

/// Return the square root of the specified `x` according to Newton's
/// method.  The behavior is undefined unless `x > 0`.
/// Return the square root of the specified `x` according to Newton's
/// method.  The behavior is undefined unless `x > 0`.
double newtonsSqrt(double x)
{
    BSLS_PRECONDITIONS_BEGIN();
    BSLS_ASSERT(x > 0);
    BSLS_PRECONDITIONS_END();
 
    double guess = 1.0;
    for (int ii = 0; ii < 100; ++ii) {
        guess = (guess + x / guess) / 2;
    }
    return guess;
}
 
/// Return the square root of the specified `x`.  The behavior is undefined
/// unless `x >= 0`.
/// Return the square root of the specified `x`.  The behavior is undefined
/// unless `x >= 0`.
double mySqrt(double x)
{
    BSLS_PRECONDITIONS_BEGIN();
    BSLS_ASSERT(x >= 0);
    BSLS_PRECONDITIONS_END();
    return newtonsSqrt(x);
}

Then, for the illustration of BSLS_FUZZTEST_EVALUATE_RAW, we define a narrow function that uses a narrow function, triggerAssert, from another component, bsls::FuzzTest_TestUtil. This function, triggerAssert, always triggers an assertion failure.

 /// Invoke `triggerAssert` from `bsls::FuzzTest_TestUtil`.  The behavior is
 /// undefined if `triggerAssert` fails.
 void invokeTriggerAssert()
 {
     bsls::FuzzTest_TestUtil::triggerAssert();
     //...
 }

Next, implement LLVMFuzzerTestOneInput. We first select the test case number based on the supplied fuzz data.

/// Use the specified `data` array of `size` bytes as input to methods of
/// this component and return zero.
extern "C"
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
    int         test;
    if (data && size) {
        test = static_cast<unsigned char>(*data) % 100;
        ++data;
        --size;
    }
    else {
        test = 0;
    }
 
    switch (test) { case 0:  // Zero is always the leading case.

Then, we implement the test case to illustrate the use of BSLS_FUZZTEST_EVALUATE.

      case 2: {
        // ----------------------------------------------------------------
        // `mySqrt`
        //
        // Concerns:
        //  1. That `mySqrt` does not invoke the original assertion handler
        //     for any `input` value.
        //
        // Testing: double mySqrt(double x);
        // ----------------------------------------------------------------
        if (size < sizeof(double)) {
            return 0;                                             // RETURN
        }
        double input;
        memcpy(&input, data, sizeof(double));

Next, we set up the handler guard that installs the precondition handlers.

        bsls::FuzzTestHandlerGuard hg;

Now, we invoke the function under test (i.e., mySqrt) with the BSLS_FUZZTEST_EVALUATE macro.

        BSLS_FUZZTEST_EVALUATE(mySqrt(input));

If the input value obtained from the fuzz data is positive (e.g., 4.0), the mySqrt implementation generates correct results without any errors. For negative inputs (e.g., -4.0), because the precondition violation occurs in the top level, execution of the test does not halt. If 0 is passed as the input, mySqrt forwards it to newtonsSqrt where a second-level assertion occurs and execution halts, indicating a defect in the implementation of mySqrt.

      } break;

Next, we implement the test case to illustrate the use of BSLS_FUZZTEST_EVALUATE_RAW.

      case 1: {
        // ----------------------------------------------------------------
        // `invokeTriggerAssert`
        //
        // Concerns:
        //  1. That `invokeTriggerAssert`, when invoked with the `RAW`
        //     macro, does not invoke the original assertion handler.
        //
        // Testing: void invokeTriggerAssert();
        // ----------------------------------------------------------------

Now, we set up the handler guard that installs the precondition handlers.

        bsls::FuzzTestHandlerGuard hg;

Finally, we invoke the function under test with the BSLS_FUZZTEST_EVALUATE_RAW macro.

        BSLS_FUZZTEST_EVALUATE_RAW(invokeTriggerAssert());

Here a top-level assertion failure from a different component will occur. Because we have invoked invokeTriggerAssert with the RAW macro, a component name check will not be performed, and execution will continue.

      } break;
    default: {
      } break;
    }
 
    if (testStatus > 0) {
        BSLS_REVIEW_INVOKE("FUZZ TEST FAILURES");
    }
 
    return 0;
}

Note that the use of bslim::FuzzUtil and bslim::FuzzDataView can simplify the consumption of fuzz data.

Macro Definition Documentation

BSLS_FUZZTEST_EVALUATE

#define BSLS_FUZZTEST_EVALUATE

(

X

)

Value:

        do {                                        \
        X;                                                                    \
    } while (false)

BSLS_FUZZTEST_EVALUATE_IMP

#define BSLS_FUZZTEST_EVALUATE_IMP

(

X

)

Value:

        do {                                    \
        X;                                                                    \
    } while (false)

BSLS_FUZZTEST_EVALUATE_RAW

#define BSLS_FUZZTEST_EVALUATE_RAW

(

X

)

Value:

        do {                                    \
        X;                                                                    \
    } while (false)

BSLS_FUZZTEST_EVALUATE_RAW_IMP

#define BSLS_FUZZTEST_EVALUATE_RAW_IMP

(

X

)

Value:

        do {                                \
        X;                                                                    \
    } while (false)