Component of the Week #36: bdlsb - I/O streambufs for memory buffers

Summary:

  • Provides stream buffer classes for memory-based I/O operations.

  • Offers both fixed-size and dynamic memory buffer solutions.

  • Includes high-performance non-virtual stream buffer alternatives.

  • Supports both input and output operations on memory buffers.

The bdlsb package provides a collection of stream buffer classes that implement the bsl::basic_streambuf protocol using memory buffers instead of files or other I/O devices. This package is particularly useful for testing, serialization, and high-performance streaming operations where data is processed entirely in memory.

Package Components

This package contains components that fall into two categories. Components with names ending in “streambuf” define classes derived from bsl::streambuf. The other components define classes with similar interfaces but without inheritance from bsl::streambuf. These non-derived classes can be passed as template parameters, making member function calls faster since they avoid virtual function overhead. The package contains the following components:

  • bdlsb_fixedmeminput: Basic input stream buffer using a client-supplied buffer. Does not derive from bsl::streambuf for better performance.

  • bdlsb_fixedmeminstreambuf: Input stream buffer using a client-supplied buffer, deriving from bsl::streambuf for compatibility with standard streams.

  • bdlsb_fixedmemoutput: Basic output stream buffer using a client-supplied buffer. Does not derive from bsl::streambuf for better performance.

  • bdlsb_fixedmemoutstreambuf: Output stream buffer using a client-supplied buffer, deriving from bsl::streambuf for compatibility with standard streams.

  • bdlsb_memoutstreambuf: Output stream buffer using allocator-managed memory that can grow dynamically.

  • bdlsb_overflowmemoutput: Output stream buffer that starts with a client buffer but overflows to an allocator-managed buffer when needed.

  • bdlsb_overflowmemoutstreambuf: Stream buffer version of bdlsb_overflowmemoutput that derives from bsl::streambuf.

Usage Examples

Fixed Memory Input Stream Buffer

bdlsb::FixedMemInStreamBuf is designed for reading from a fixed memory buffer with full bsl::streambuf compatibility. Here we show using it with a standard bsl::istream:

#include <bdlsb_fixedmeminstreambuf.h>

#include <bdlb_doublecompareutil.h>

#include <bsl_algorithm.h>
#include <bsl_cmath.h>
#include <bsl_iostream.h>
#include <bsl_string.h>

#include <cassert>

using namespace BloombergLP;
using bsl::cout;
using bsl::endl;

int main() {
    typedef bdlb::DoubleCompareUtil CompUtil;

    // First, prepare some text data in a buffer

    const char testData[] = "42 3.14159 Hello World!\n"
                            "100 2.71828 Testing 123\n";

    // Create a fixed memory input stream buffer from the test data

    bdlsb::FixedMemInStreamBuf streamBuffer(testData,
                                            sizeof(testData) - 1);
    bsl::istream inputStream(&streamBuffer);

    // Read structured data from the buffer

    int intValue1, intValue2;
    double doubleValue1, doubleValue2;
    bsl::string text1, text2, text3, text4;

    inputStream >> intValue1 >> doubleValue1 >> text1 >> text2;
    inputStream >> intValue2 >> doubleValue2 >> text3 >> text4;

    if (inputStream.fail()) {
        cout << "Error reading from stream" << endl;
        return -1;
    }

    // Verify the parsed data

    assert(intValue1 == 42);
    assert(0 == CompUtil::fuzzyCompare(doubleValue1, 3.14159, 0.01));
    assert(text1 == "Hello");
    assert(text2 == "World!");

    assert(intValue2 == 100);
    assert(0 == CompUtil::fuzzyCompare(doubleValue2, 2.71828, 0.01));
    assert(text3 == "Testing");
    assert(text4 == "123");

    cout << "All values parsed and verified successfully!" << endl;

    return 0;
}

Fixed Memory Output Stream Buffer

bdlsb::FixedMemOutStreamBuf is designed for writing to a fixed memory buffer with full bsl::streambuf compatibility. Here we show using it with a standard bsl::ostream:

#include <bdlsb_fixedmemoutstreambuf.h>

#include <bslma_defaultallocatorguard.h>
#include <bslma_testallocator.h>

#include <bsl_iostream.h>
#include <bsl_string.h>

#include <cassert>

using namespace BloombergLP;
using bsl::cout;
using bsl::endl;
using bsl::ends;

int main() {
    // Set up test allocator as default to verify no default allocations

    bslma::TestAllocator defaultAllocator("default", false);
    bslma::DefaultAllocatorGuard guard(&defaultAllocator);

    enum { k_BUFFER_SIZE = 200 };
    char buffer[k_BUFFER_SIZE];

    // Create a fixed memory output stream buffer

    bdlsb::FixedMemOutStreamBuf streamBuf(buffer, k_BUFFER_SIZE);
    bsl::ostream outputStream(&streamBuf);

    // Write formatted text and numerical data

    outputStream << "Test Results Summary" << endl;
    outputStream << "===================" << endl;
    outputStream << "Tests run: " << 150 << endl;
    outputStream << "Success rate: " << 98.67 << "%" << endl;
    outputStream << "Average time: " << 2.345 << " seconds" << endl;
    outputStream << ends;        // null-terminate the buffer

    if (outputStream.fail()) {
        cout << "Error writing to stream" << endl;
        return -1;
    }

    // Check what was written

    cout << "Bytes written: " << streamBuf.length() << endl;
    cout << "Buffer capacity: " << streamBuf.capacity() << endl;
    cout << endl;

    // Display the formatted output

    cout << "Generated output:" << endl;
    cout << "=================" << endl;

    // Output the buffer directly

    cout << buffer;

    // Verify no default allocations occurred

    assert(0 == defaultAllocator.numAllocations());

    return 0;
}

Expected output:

Bytes written: 85
Buffer capacity: 200

Generated output:
=================
Test Results Summary
===================
Tests run: 150
Success rate: 98.67%
Average time: 2.345 seconds

Dynamic Memory Output Stream Buffer

bdlsb::MemOutStreamBuf provides a stream buffer that can grow dynamically and is particularly useful for testing stream operations:

#include <bdlsb_memoutstreambuf.h>

#include <bslma_defaultallocatorguard.h>
#include <bslma_testallocator.h>

#include <bsl_iostream.h>
#include <bsl_string.h>

#include <cassert>

using namespace BloombergLP;
using bsl::cout;
using bsl::endl;
using bsl::ends;

int main() {
    // Set up test allocator as default to verify no default allocations

    bslma::TestAllocator defaultAllocator("default", false);
    bslma::DefaultAllocatorGuard guard(&defaultAllocator);

    bslma::TestAllocator ta("test", false);

    // Create a memory output stream buffer
    // One advantage: doesn't use default allocator like
    // `bsl::ostringstream`.

    bdlsb::MemOutStreamBuf streamBuf(&ta);
    bsl::ostream os(&streamBuf);

    // Write formatted output directly to the stream

    os << "=== Test Results ===" << endl;
    os << "Tests Passed: " << 42 << endl;
    os << "Tests Failed: " << 3 << endl;
    os << ends;                                    // Add terminating null

    // Get the formatted output without using default allocator

    cout << "Formatted output:" << endl;
    cout << streamBuf.data() << endl;

    cout << "Allocator used: " << ta.numBytesInUse() << " bytes" << endl;

    // Reading the contents of a `bsl::ostringstream` with the 'str()'
    // accessor would unavoidably use the default allocator.

    cout << "Note: `bsl::ostringstream.str()` would use the default\n"
         << "allocator, but `bdlsb::MemOutStreamBuf` allows us to\n"
         << "use an allocator of our choice." << endl;

    // Verify no default allocations occurred

    assert(0 == defaultAllocator.numAllocations());

    return 0;
}

Expected output:

Formatted output:
=== Test Results ===
Tests Passed: 42
Tests Failed: 3

Allocator used: 256 bytes
Note: `bsl::ostringstream.str()` would use the default
allocator, but `bdlsb::MemOutStreamBuf` allows us to
use an allocator of our choice.

Key Features

Performance Optimization * Non-virtual versions (FixedMemInput, FixedMemOutput, OverFlowMemOutput) avoid virtual function call overhead, useful in calls that take a ‘streambuf-like’ class as a template arguement rather than as a streambuf * * Designed for high-performance scenarios where minimal overhead is crucial

Memory Management * Fixed-size buffers for predictable memory usage * Dynamic buffers that grow as needed * Overflow buffers that start fixed but expand when necessary * Control over allocator usage (unlike bsl::ostringstream)

Testing Benefits * Output classes that inherit from bsl::streambuf can be used to initialize bsl::ostream`s and test output functions without using the default allocator, which is not possible with `bsl::ostringstream. * Direct access to raw buffer contents for verification * No file I/O overhead during testing

Flexibility * Both bsl::streambuf-derived and non-derived versions * Support for both input and output operations * Compatible with standard C++ streams and text/numerical I/O

Common Use Cases

The bdlsb package is particularly useful for:

  • Text and numerical I/O: Parsing and formatting data in memory buffers

  • Testing: Capturing stream output for verification without file I/O

  • Memory-constrained environments: Fixed-size buffers with predictable memory usage

  • Performance-critical code: Non-virtual versions for minimal overhead

  • Allocator control: Avoiding default allocator usage in output stream testing

For more details, see: