bsls_fuzztest Component

Provide macros for use in fuzz testing narrow-contract functions. More...

Namespaces
namespace	bsls

Detailed Description

Outline

Purpose
Classes
Macros
Description
- Usage
  - Example: Basic Usage of Macros

Purpose:: Provide macros for use in fuzz testing narrow-contract functions.

Classes:

bsls::FuzzTestPreconditionTracker	utility for tracking assert violations
bsls::FuzzTestHandlerGuard	guard for fuzz testing assert-handler

Macros:

BSLS_FUZZTEST_EVALUATE(EXPRESSION)	wrapper for narrow contract function
BSLS_FUZZTEST_EVALUATE_RAW(EXPRESSION)	wrapper with no origination check

See also:: Component bsls_preconditions

Description:: This component provides two macros, BSLS_FUZZTEST_EVALUATE and BSLS_FUZZTEST_EVALUATE_RAW, that can be used in fuzz testing narrow contract functions. They are intended to be used in conjunction with bsls::FuzzTestHandlerGuard as well as BSLS_PRECONDITIONS_BEGIN and BSLS_PRECONDITIONS_END.

: When fuzzing narrow contract functions, if we do not wish to "massage" the data we pass to the function (as this may be error-prone and might introduce bias into the tested input) we must address the issue that we will often invoke the function out of contract, and this will cause the function to assert, and the test to end prematurely. The macros defined in this component solve this issue by detecting the location of precondition violations. Functions with narrow contracts that are to be tested must be decorated with the BSLS_PRECONDITIONS_BEGIN and BSLS_PRECONDITIONS_END macros. These macros must be placed just before and after the function preconditions are checked.

: All these macros are intended to be used in fuzzing builds in which BDE_ACTIVATE_FUZZ_TESTING is defined. For our purposes, those preconditions that fail in the function under test (i.e., the one invoked by BSLS_FUZZTEST_EVALUATE) are treated differently from all other precondition failures. We refer to these preconditions as "top-level" preconditions. If a top-level precondition fails -- and the assertion is not from another component -- the execution will continue: we do not wish to stop the fuzz test if we simply invoked the narrow contract function under test out of contract. We wish to detect only subsequent assertions (i.e., not in the top-level), or assertions from other components.

: The BSLS_FUZZTEST_EVALUATE_RAW macro does not check if the assertion originates from another component, though, like the non-'RAW' version, it ignores only top-level assertions. This behavior is desirable in cases in which a function delegates its implementation and associated precondition checks to a different component. In such cases, a precondition failure ought not cause the fuzz test to end.

Usage:: This section illustrates intended use of this component.

Example: Basic Usage of Macros:: The macros in this component rely upon the presence of related macros from bsls_preconditions. The fuzzing macros are typically used in a fuzzing build, in which case the entry point is LLVMFuzzerTestOneInput.

: In this example, we illustrate the intended usage of two macros: BSLS_FUZZTEST_EVALUATE and BSLS_FUZZTEST_EVALUATE_RAW.

First, in order to illustrate the use of BSLS_FUZZTEST_EVALUATE, we define two functions that implement the sqrt function, both decorated with the precondition BEGIN and END macros. mySqrt forwards its argument to newtonsSqrt, which has a slightly more restrictive precondition: mySqrt accepts 0, while newtonsSqrt does not.

  double newtonsSqrt(double x)
      // Return the square root of the specified 'x' according to Newton's
      // method.  The behavior is undefined unless 'x > 0'.
  {
      BSLS_PRECONDITIONS_BEGIN();
      BSLS_ASSERT(x > 0);
      BSLS_PRECONDITIONS_END();

      double guess = 1.0;
      for (int ii = 0; ii < 100; ++ii) {
          guess = (guess + x / guess) / 2;
      }
      return guess;
  }

  double mySqrt(double x)
      // Return the square root of the specified 'x'.  The behavior is
      // undefined unless 'x >= 0'.
  {
      BSLS_PRECONDITIONS_BEGIN();
      BSLS_ASSERT(x >= 0);
      BSLS_PRECONDITIONS_END();
      return newtonsSqrt(x);
  }

Then, for the illustration of BSLS_FUZZTEST_EVALUATE_RAW, we define a class, Timer, containing a start function that uses in its implementation a narrow contract function, setInterval, from another component, bsls::TimeInterval. This function, setInterval, has precondition checks that are surrounded by BEGIN and END.

  class Timer
      // This class implements a simple interval timer.
  {
    private:
      // DATA
      bsls::TimeInterval d_timeout;  // timeout seconds and nanoseconds

    public:
      // MANIPULATORS
      void start(bsls::Types::Int64 seconds, int nanoseconds)
          // Start the countdown with a timer having the value given by the
          // sum of the specified integral number of 'seconds' and
          // 'nanoseconds'.  The behavior is undefined unless the total
          // number of seconds in the resulting time interval can be
          // represented with a 64-bit signed integer (see
          // 'TimeInterval::isValid').
      {
          d_timeout.setInterval(seconds, nanoseconds);
          //...
      }
  };

Next, implement LLVMFuzzerTestOneInput. We first select the test case number based on the supplied fuzz data.

  extern "C"
  int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
      // Use the specified 'data' array of 'size' bytes as input to methods
      // of this component and return zero.
  {
      int         test;
      if (data && size) {
          test = static_cast<unsigned char>(*data) % 100;
          ++data;
          --size;
      }
      else {
          test = 0;
      }

      switch (test) { case 0:  // Zero is always the leading case.

Then, we implement the test case to illustrate the use of BSLS_FUZZTEST_EVALUATE.

        case 2: {
          // ----------------------------------------------------------------
          // 'mySqrt'
          //
          // Concerns:
          //: 1. That 'mySqrt' does not invoke the original assertion handler
          //:    for any 'input' value.
          //
          // Testing: double mySqrt(double x);
          // ----------------------------------------------------------------
          if (size < sizeof(double)) {
              return 0;                                             // RETURN
          }
          double input;
          memcpy(&input, data, sizeof(double));

Next, we set up the handler guard that installs the precondition handlers.

          bsls::FuzzTestHandlerGuard hg;

Now, we invoke the function under test (i.e., mySqrt) with the BSLS_FUZZTEST_EVALUATE macro.

          BSLS_FUZZTEST_EVALUATE(mySqrt(input));

If the input value obtained from the fuzz data is positive (e.g., 4.0), the mySqrt implementation generates correct results without any errors. For negative inputs (e.g., -4.0), because the precondition violation occurs in the top level, execution of the test does not halt. If 0 is passed as the input, mySqrt forwards it to newtonsSqrt where a second-level assertion occurs and execution halts, indicating a defect in the implementation of mySqrt.

        } break;

Next, we implement the test case to illustrate the use of BSLS_FUZZTEST_EVALUATE_RAW.

        case 1: {
          // ----------------------------------------------------------------
          // 'Timer::start'
          //
          // Concerns:
          //: 1 That 'start', when invoked with the 'RAW' macro, does not
          //:   invoke the original assertion handler.
          //
          // Testing:
          //   void Timer::start(Int64 seconds, int nanoseconds);
          // ----------------------------------------------------------------

          if (size < sizeof(bsls::Types::Int64) + sizeof(int)) {
              return 0;                                             // RETURN
          }
          bsls::Types::Int64 seconds;
          int                nanoseconds;
          memcpy(&seconds, data, sizeof(bsls::Types::Int64));
          memcpy(&nanoseconds,
                 data + sizeof(bsls::Types::Int64),
                 sizeof(int));

Now, we set up the handler guard that installs the precondition handlers.

          bsls::FuzzTestHandlerGuard hg;

Finally, we invoke the function under test with the BSLS_FUZZTEST_EVALUATE_RAW macro.

          Timer t;
          BSLS_FUZZTEST_EVALUATE_RAW(t.start(seconds, nanoseconds));

If the total number of seconds resulting from the sum of seconds and nanoseconds cannot be represented with a 64-bit signed integer, a top-level assertion failure from a different component will occur. Because we have invoked start with the RAW macro, a component name check will not be performed, and execution will continue.

        } break;
        default: {
        } break;
      }

      if (testStatus > 0) {
          BSLS_ASSERT_INVOKE("FUZZ TEST FAILURES");
      }

      return 0;
  }

Note that the use of bslim::FuzzUtil and bslim::FuzzDataView can simplify the consumption of fuzz data.

Component bsls_fuzztest [Package bsls]

Namespaces

Detailed Description

Component bsls_fuzztest
[Package bsls]