bdld::Datum Quick Introduction

The bdld_datum component defines a mechanism, bdld::Datum, that provides a space-efficient discriminated union (i.e., a variant) that can hold values of many different types - from scalars like integers and doubles to aggregates like arrays and maps. Unlike typical BDE types, Datum is not a value-semantic type but behaves more like a pointer, requiring explicit memory management similar to raw pointers. However, Datum does provide equality comparison operators (operator== and operator!=) that compare the values to which the Datum objects refer, not just the addresses.

This design makes Datum particularly well-suited for applications that need to create and copy very large numbers of variant values, such as:

  • Interfacing with dynamic languages like JavaScript

  • Creating scripting languages like Bloomberg’s HSL (Hypersheet Scripting Language)

  • Spreadsheet implementations where cells can contain different types

  • Data interchange and serialization scenarios

The key characteristic that distinguishes Datum from other BDE types is its pointer-like behavior: copying a Datum performs a shallow copy (like copying a pointer), and memory must be explicitly managed using static Datum::create* and Datum::destroy functions. However, when deep copying is needed, Datum provides a clone method that creates independent, self-contained copies where even external references are copied as owned entities.

Note

Datum is a POD (Plain Old Data) type, meaning instances are bitwise copyable and have trivial initialization, assignment, and destruction. This makes Datum extremely efficient but requires careful attention to memory management.

Understanding Datum’s Memory Model

The most important aspect of Datum is understanding its memory model, which is fundamentally different from typical BDE value-semantic types:

#include <bdld_datum.h>
#include <bslma_testallocator.h>
#include <bsl_iostream.h>

using namespace BloombergLP;

int main()
{
    bslma::TestAllocator ta;

    // Values that fit in the Datum footprint require no allocation
    bdld::Datum integer = bdld::Datum::createInteger(42);
    bdld::Datum boolean = bdld::Datum::createBoolean(true);
    bdld::Datum floating = bdld::Datum::createDouble(3.14159);

    bsl::cout << "Integer: " << integer.theInteger() << bsl::endl;
    bsl::cout << "Boolean: " << boolean.theBoolean() << bsl::endl;
    bsl::cout << "Double:  " << floating.theDouble() << bsl::endl;

    // Destroy is not mandatory - these values are stored inline

    // Create a long string that exceeds short-string optimization
    const char* longText = "This is a long string that will definitely "
                           "exceed the short string optimization limit "
                           "on 64-bit platforms and require allocation";

    bdld::Datum str = bdld::Datum::copyString(longText, &ta);
    bsl::cout << "String:  " << str.theString() << bsl::endl;
    bsl::cout << "Address: "
              << static_cast<const void*>(str.theString().data())
              << bsl::endl;

    // This is a shallow copy - both point to the same string data!
    bdld::Datum str2 = str;
    bsl::cout << "Shallow copy: " << str2.theString() << bsl::endl;
    bsl::cout << "Address:      "
              << static_cast<const void*>(str2.theString().data())
              << bsl::endl;
    bsl::cout << "Shallow copy equal? "
              << (str == str2 ? "yes" : "no") << bsl::endl;
    bsl::cout << "Same address? "
              << (str.theString().data() == str2.theString().data()
                  ? "yes" : "no") << bsl::endl;

    // Only destroy once (destroying twice would be an error)
    bdld::Datum::destroy(str, &ta);

    // str2 now refers to deallocated memory - do not use it!
}

The analogy to raw pointers is instructive: Datum::create* is like new, Datum::destroy is like delete, and copying a Datum is like copying a pointer (both point to the same memory).

Creating Datum Values

Datum provides a rich set of factory functions for creating values. Some types never allocate memory, while others may or may not depending on the value:

bslma::TestAllocator ta;

// Scalar types that never require allocation
bdld::Datum nil       = bdld::Datum::createNull();
bdld::Datum integer   = bdld::Datum::createInteger(123);
bdld::Datum real      = bdld::Datum::createDouble(2.718);
bdld::Datum boolean   = bdld::Datum::createBoolean(false);
bdld::Datum date      = bdld::Datum::createDate(bdlt::Date(2025, 11, 27));
bdld::Datum time      = bdld::Datum::createTime(bdlt::Time(14, 30, 0));

bsl::cout << "Integer:          " << integer.theInteger() << bsl::endl;
bsl::cout << "Real:             " << real.theDouble()     << bsl::endl;
bsl::cout << "Date:             " << date.theDate()       << bsl::endl;

// Types that may require allocation
bdld::Datum int64     = bdld::Datum::createInteger64(1LL << 40, &ta);
bdld::Datum datetime  = bdld::Datum::createDatetime(
                            bdlt::Datetime(2025, 11, 27, 14, 30), &ta);
bdld::Datum interval  = bdld::Datum::createDatetimeInterval(
                            bdlt::DatetimeInterval(24, 0, 0, 0), &ta);
bdld::Datum decimal   = bdld::Datum::createDecimal64(
                            bdldfp::Decimal64(123.45), &ta);

bsl::cout << "Int64:            " << int64.theInteger64()   << bsl::endl;
bsl::cout << "Datetime:         " << datetime.theDatetime() << bsl::endl;

// Strings - owned vs. referenced
bdld::Datum ownedStr  = bdld::Datum::copyString("owned", &ta);
bdld::Datum refStr    = bdld::Datum::createStringRef("literal", &ta);

bsl::cout << "Owned string:     " << ownedStr.theString() << bsl::endl;
bsl::cout << "String reference: " << refStr.theString()   << bsl::endl;

// Clean up owned values
bdld::Datum::destroy(int64, &ta);
bdld::Datum::destroy(datetime, &ta);
bdld::Datum::destroy(interval, &ta);
bdld::Datum::destroy(decimal, &ta);
bdld::Datum::destroy(ownedStr, &ta);
// Note: refStr references external data, but may have allocated metadata

External References vs. Owned Data

A key feature of Datum is its ability to reference external data without taking ownership, which is crucial for performance when data will outlive the Datum:

const char* literal = "This string exists for the program lifetime";

// Create a reference to the external string - no copy made
bdld::Datum refDatum = bdld::Datum::createStringRef(literal, &ta);

assert(refDatum.isString());
assert(refDatum.isExternalReference());  // true!
bsl::cout << "External reference: " << refDatum.theString()
          << " (isExternal: " << refDatum.isExternalReference()
          << ")" << bsl::endl;
bsl::cout << "Address:            "
          << static_cast<const void*>(refDatum.theString().data())
          << bsl::endl;

// Destroy will not free the string itself
bdld::Datum::destroy(refDatum, &ta);
// literal is still valid here

// Compare with an owned string
bdld::Datum ownedDatum = bdld::Datum::copyString("copied", &ta);

assert(ownedDatum.isString());
assert(!ownedDatum.isExternalReference());  // false - Datum owns it
bsl::cout << "Owned string: " << ownedDatum.theString()
          << " (isExternal: "
          << (ownedDatum.isExternalReference() ? "true" : "false")
          << ")" << bsl::endl;
bsl::cout << "Address:      "
          << static_cast<const void*>(ownedDatum.theString().data())
          << bsl::endl;

// Destroy will free the copied string
bdld::Datum::destroy(ownedDatum, &ta);

This distinction is essential: factory methods like createStringRef and createArrayReference create references to external data (where isExternalReference() returns true), while methods like copyString and adoptArray create owned data.

Working with Arrays

Arrays of Datum objects can be created in multiple ways. The bdld package provides builder components to simplify array construction:

#include <bdld_datum.h>
#include <bdld_datumarraybuilder.h>
#include <bslma_testallocator.h>
#include <bsl_cassert.h>
#include <bsl_iostream.h>

bslma::TestAllocator ta;

// Using DatumArrayBuilder (recommended approach)
bdld::DatumArrayBuilder builder(&ta);

builder.pushBack(bdld::Datum::createInteger(1));
builder.pushBack(bdld::Datum::createInteger(2));
builder.pushBack(bdld::Datum::createDouble(3.14));
builder.pushBack(bdld::Datum::copyString("four", &ta));

bdld::Datum array = builder.commit();

// Access array elements
assert(array.isArray());
assert(4    == array.theArray().length());
assert(1    == array.theArray()[0].theInteger());
assert(3.14 == array.theArray()[2].theDouble());

bsl::cout << "Array length: " << array.theArray().length()       << bsl::endl;
bsl::cout << "  Element 0: " << array.theArray()[0].theInteger() << bsl::endl;
bsl::cout << "  Element 2: " << array.theArray()[2].theDouble()  << bsl::endl;
bsl::cout << "  Element 3: " << array.theArray()[3].theString()  << bsl::endl;

// Destroy recursively frees all elements
bdld::Datum::destroy(array, &ta);

The builder pattern handles all the complexity of memory allocation and exception safety, making it the preferred approach for constructing arrays.

Working with Maps

Maps provide key-value storage where keys are strings and values are Datum objects:

#include <bdld_datum.h>
#include <bdld_datummapbuilder.h>
#include <bslma_testallocator.h>
#include <bsl_cassert.h>
#include <bsl_iostream.h>

bslma::TestAllocator ta;

// Using DatumMapBuilder (recommended approach)
bdld::DatumMapBuilder builder(&ta);

builder.pushBack("name",    bdld::Datum::copyString("Alice", &ta));
builder.pushBack("age",     bdld::Datum::createInteger(30));
builder.pushBack("active",  bdld::Datum::createBoolean(true));
builder.pushBack("balance", bdld::Datum::createDouble(1234.56));

bdld::Datum record = builder.commit();

// Access map elements
assert(record.isMap());
assert(4 == record.theMap().size());

bsl::cout << "Map size: " << record.theMap().size() << bsl::endl;

// Find by key
const bdld::Datum* name = record.theMap().find("name");
if (name) {
    assert(name->isString());
    assert("Alice" == name->theString());
    bsl::cout << "Name: " << name->theString() << bsl::endl;
}

const bdld::Datum* age = record.theMap().find("age");
if (age) {
    bsl::cout << "Age: " << age->theInteger() << bsl::endl;
}

// Destroy recursively frees the entire map structure
bdld::Datum::destroy(record, &ta);

Maps can be sorted by key for efficient O(log N) lookup, or unsorted for O(N) lookup. The DatumMapBuilder provides control over this behavior.

The bdld Package Ecosystem

The bdld package provides several supporting components that work with Datum:

  • bdld_manageddatum - Provides a smart-pointer-like wrapper that adds value-semantic operations and automatic memory management to Datum, making it easier to use in containers and other value-semantic contexts

  • bdld_datumarraybuilder - Utility for building Datum arrays in an exception-safe manner, particularly useful when the array size isn’t known in advance

  • bdld_datummapbuilder - Utility for building Datum maps (string keys)

  • bdld_datumintmapbuilder - Utility for building Datum int-maps (integer keys)

  • bdld_datummapowningkeysbuilder - Utility for building maps where the Datum owns copies of the string keys

  • bdld_datummaker - Provides a concise syntax for creating complex nested Datum structures, especially useful in tests

  • bdld_datumutil - Additional utility functions for working with Datum objects

  • bdld_datumerror - Type for representing error values within a Datum

  • bdld_datumudt - Type for representing user-defined types within a Datum

  • bdld_datumbinaryref - Type for representing binary data within a Datum

User-Defined Types

Datum can hold opaque user-defined types, making it extensible for custom data:

struct MyCustomType {
    int id;
    double value;
};

enum { MY_TYPE_ID = 1001 };  // Application-defined type identifier

MyCustomType custom = { 42, 3.14159 };

// Create a Datum holding the UDT
bdld::Datum udtDatum = bdld::Datum::createUdt(&custom, MY_TYPE_ID);

assert(udtDatum.isUdt());

// Retrieve the UDT
bdld::DatumUdt udt = udtDatum.theUdt();
assert(MY_TYPE_ID == udt.type());
assert(&custom == static_cast<MyCustomType*>(udt.data()));

bsl::cout << "UDT type: " << udt.type() << bsl::endl;
MyCustomType* retrieved = static_cast<MyCustomType*>(udt.data());
bsl::cout << "UDT id: " << retrieved->id
          << ", value: " << retrieved->value << bsl::endl;

// Note: UDTs are always external references
// Datum::destroy does nothing for the UDT itself

User-defined types are opaque to Datum - it only stores a pointer and type identifier. The application is responsible for managing the lifetime of UDT objects.

Deep Copying with clone

While copying a Datum performs a shallow copy, the clone method creates a deep copy with independent lifetime. Importantly, clone converts external references into owned data: a string reference becomes an owned string, and an array reference becomes an owned array with deep-copied elements. The only exception is user-defined types (UDTs), which remain as opaque pointers since Datum cannot know how to copy them:

bslma::TestAllocator ta;

// Create a long string that exceeds short-string optimization
const char* longText = "This is a long string that will definitely "
                       "exceed the short string optimization limit "
                       "on 64-bit platforms and require allocation";

bdld::Datum original = bdld::Datum::copyString(longText, &ta);
bsl::cout << "Original: " << original.theString() << bsl::endl;
bsl::cout << "Address:  "
          << static_cast<const void*>(original.theString().data())
          << bsl::endl;

// Shallow copy - both refer to the same string
bdld::Datum shallowCopy = original;
bsl::cout << "Shallow copy equal? " << (original == shallowCopy)
          << bsl::endl;
bsl::cout << "Same address? "
          << (original.theString().data() == shallowCopy.theString().data()
              ? "yes" : "no") << bsl::endl;

// Deep copy - creates independent copy of the string
bdld::Datum deepCopy = original.clone(&ta);
bsl::cout << "Deep copy: " << deepCopy.theString() << bsl::endl;
bsl::cout << "Address:   "
          << static_cast<const void*>(deepCopy.theString().data())
          << bsl::endl;
bsl::cout << "Deep copy equal? " << (original == deepCopy) << bsl::endl;
bsl::cout << "Different address? "
          << (original.theString().data() != deepCopy.theString().data()
              ? "yes" : "no") << bsl::endl;

// Now we can safely destroy the original
bdld::Datum::destroy(original, &ta);

// deepCopy is still valid and independent
assert(deepCopy.isString());
assert(longText == deepCopy.theString());
bsl::cout << "Deep copy still valid: " << deepCopy.theString()
          << bsl::endl;

// Clean up the deep copy
bdld::Datum::destroy(deepCopy, &ta);

// Note: shallowCopy now refers to freed memory - do not use!

The clone method is particularly useful when you need to transfer ownership or extend the lifetime of a Datum beyond its original context.

Efficient Memory Management with Managed Allocators

A common pattern when working with Datum is to perform computations that create many temporary Datum values, extract the result, and then discard all the temporaries. Using a managed allocator (e.g., bdlma::SequentialAllocator) for this pattern provides exceptional performance and safety benefits.

The key insight is that managed allocators like bdlma::SequentialAllocator allocate memory from a buffer and can deallocate all memory at once by simply resetting their state. This means we can create arbitrarily many temporary Datum objects during a calculation without worrying about:

  • Double-free errors from shallow copies

  • Circular dependencies in complex data structures

  • Individual deallocation overhead for each temporary

The pattern is simple: allocate all temporaries with a managed allocator, use clone to copy the final result to a longer-lived allocator, then “wink” all temporaries out of existence by releasing the managed allocator:

#include <bdld_datum.h>
#include <bdld_manageddatum.h>
#include <bdld_datumarraybuilder.h>
#include <bdlma_sequentialallocator.h>
#include <bdlma_testallocator.h>
#include <bsl_cassert.h>
#include <bsl_iostream.h>

using namespace BloombergLP;

// Approach 1: Return a ManagedDatum using the provided allocator
bdld::ManagedDatum computeResult(bslma::Allocator *resultAllocator)
{
    bdlma::SequentialAllocator tempAllocator;

    // Create many intermediate values during computation
    bdld::Datum step1 = bdld::Datum::copyString("intermediate_1", &tempAllocator);
    bdld::Datum step2 = bdld::Datum::copyString("intermediate_2", &tempAllocator);
    bdld::Datum step3 = bdld::Datum::copyString("intermediate_3", &tempAllocator);

    // Build an array with intermediate results
    bdld::DatumArrayBuilder builder(&tempAllocator);
    builder.pushBack(step1);
    builder.pushBack(step2);
    builder.pushBack(step3);
    builder.pushBack(bdld::Datum::createInteger(42));
    builder.pushBack(bdld::Datum::createDouble(3.14159));

    // The result is also in temporary storage
    bdld::Datum tempResult = builder.commit();

    // Note: All these Datum objects share the tempAllocator
    // We have multiple shallow copies referring to the same data
    // But we don't need to track which ones to destroy!

    // Clone the result to permanent storage
    bdld::ManagedDatum result(resultAllocator);
    result.clone(tempResult);

    // "Wink" all temporaries out of existence - no individual destroy calls!
    tempAllocator.release();
    // Note that this `release()` call is for demonstration purposes only, the
    // destructor of a managed allocator releases all memory allocated.

    return result;
}

// Approach 2: Use an out-parameter for the result
void computeResultOutParam(bdld::ManagedDatum *result)
{
    bdlma::SequentialAllocator tempAllocator;

    // Create temporary values (same as above)
    bdld::Datum step1 = bdld::Datum::copyString("computed", &tempAllocator);
    bdld::Datum step2 = bdld::Datum::createInteger(100);

    bdld::DatumArrayBuilder builder(&tempAllocator);
    builder.pushBack(step1);
    builder.pushBack(step2);

    bdld::Datum tempResult = builder.commit();

    // Clone into the out-parameter's allocator
    result->adopt(tempResult.clone(result->allocator()));

    // Wink out all temporaries
    tempAllocator.release();
}

int main()
{
    bslma::TestAllocator ta;

    // Using approach 1: function returns result
    bdld::ManagedDatum result1 = computeResult(&ta);

    assert(result1.datum().isArray());
    assert(5 == result1.datum().theArray().length());

    bsl::cout << "Result 1 array length: "
              << result1.datum().theArray().length() << bsl::endl;
    bsl::cout << "  Element 0: "
              << result1.datum().theArray()[0].theString() << bsl::endl;

    // Using approach 2: out-parameter
    bdld::ManagedDatum result2(&ta);
    computeResultOutParam(&result2);

    assert(result2.datum().isArray());
    assert(2 == result2.datum().theArray().length());

    bsl::cout << "Result 2 array length: "
              << result2.datum().theArray().length() << bsl::endl;

    // ManagedDatum handles cleanup automatically
}

This pattern provides several critical advantages:

Safety from Shallow Copies

Since all temporaries disappear at once via release(), we never risk double-freeing memory even if we made shallow copies during computation. The managed allocator doesn’t care how many Datum objects refer to the same string - when we release, everything goes away atomically.

No Circular Dependency Concerns

Complex calculations might create Datum arrays that reference other arrays, potentially forming cycles. With individual destroy calls, this would be problematic. With SequentialAllocator::release(), cycles don’t matter - everything is discarded together.

Exceptional Performance

Managed allocators like SequentialAllocator are extremely fast (just pointer bumping), and bulk deallocation via release() is instant. For computations creating thousands of temporary Datum values, this can be orders of magnitude faster than individual allocate/deallocate cycles.

Simplified Code

No need to carefully track which Datum objects to destroy or worry about the order of destruction. Just clone the result and release the rest.

This pattern is used extensively in systems like Bloomberg’s expression evaluators and scripting language implementations, where complex calculations create vast numbers of temporary values that only need to live for the duration of a single computation.

See Component of the Week #34: bdlma_sequentialallocator for more information on bdlma::SequentialAllocator.

Conclusion

bdld::Datum is a unique and powerful tool in the BDE arsenal. Its pointer-like behavior and small footprint make it invaluable for scenarios requiring large numbers of variant values, such as scripting language implementations and dynamic data interchange. While it requires more careful memory management than typical BDE types, the bdld package provides excellent supporting components (ManagedDatum, various builders) that simplify working with Datum objects.

The key to successfully using Datum is understanding its memory model: treat it like a smart raw pointer that requires explicit destroy calls, and use the builder components and ManagedDatum to handle the complexity in production code.

For more information, including comprehensive details on all supported types and memory management patterns, see the component documentation and the related components in the bdld package.