Component of the Week #19: `bdlde_utf8util`

Summary:

Provides basic utilities for working with UTF-8 encoded strings.

The bdlde_utf8util component defines the bdlde::Utf8Util utility class, which provides a suite of static methods for validating, counting, and manipulating UTF-8 encoded text according to RFC 3629.

The component provides functions in these main categories:

Validation: Check if a string contains valid UTF-8 encoding.
Code Point Counting: Count the number of Unicode code points in a UTF-8 string.
Iteration: Advance through UTF-8 strings by code points.
Size Determination: Calculate the byte size of a sequence of code points when encoded as UTF-8.
Encoding/Decoding: Encode code points to UTF-8 and extract code point values from UTF-8 sequences.

Each function is provided in two variants:

One that takes a string with an explicit length parameter
One that takes a null-terminated C-style string

Here’s an example that demonstrates encoding to UTF-8, validating the resulting UTF-8 string, and counting UTF-8 code points (recovering the number of code points originally encoded):

#include <bdlde_utf8util.h>
#include <bsl_iostream.h>
#include <bsl_string.h>

using namespace BloombergLP;

int main() {
    // Create a UTF-8 string with various Unicode characters
    bsl::string text;
    bdlde::Utf8Util::appendUtf8CodePoint(&text, 0x1F600);  // 😀
    bdlde::Utf8Util::appendUtf8CodePoint(&text, 0x20AC);   // €
    bdlde::Utf8Util::appendUtf8CodePoint(&text, 0x65E5);   // 日
    bdlde::Utf8Util::appendUtf8CodePoint(&text, 0x672C);   // 本
    bdlde::Utf8Util::appendUtf8CodePoint(&text, 0x8A9E);   // 語

    // Validate the UTF-8 string
    bool isValid = bdlde::Utf8Util::isValid(text.data(), text.length());

    // Count the code points in the string
    bsls::Types::IntPtr numCodePoints =
        bdlde::Utf8Util::numCodePointsRaw(text.data(), text.length());

    bsl::cout << "UTF-8 string is " << (isValid ? "valid" : "invalid")
              << bsl::endl
              << "Number of code points: " << numCodePoints << bsl::endl
              << "Number of bytes: " << text.length() << bsl::endl;

    // Output:
    // UTF-8 string is valid
    // Number of code points: 5
    // Number of bytes: 16

    return 0;
}

Some important features of bdlde::Utf8Util:

It enforces strict UTF-8 compliance per RFC 3629. In particular:
- It rejects overlong encodings (using more bytes than necessary).
- It rejects surrogate values (code points in range U+D800 to U+DFFF).
It considers an empty string to be valid UTF-8.
It properly handles embedded null bytes in strings with explicit length.

When working with UTF-8 text, be aware that code points are not the same thing as user-facing characters. Multiple code points may be combined to form what a would consider a single character. Manipulating user-facing characters is beyond the scope of bdlde::Utf8Util.

For more details and additional functions, see the documentation for bdlde_utf8util.

Component of the Week #19: bdlde_utf8util

Component of the Week #19: `bdlde_utf8util`