Provide basic utilities for UTF-8 encodings.
More...
Namespaces |
| namespace | bdlde |
Detailed Description
- Outline
-
-
- Purpose:
- Provide basic utilities for UTF-8 encodings.
-
- Classes:
-
-
- Description:
- This component provides, within the
bdlde::Utf8Util struct, a suite of static functions supporting UTF-8 encoded strings. Two interfaces are provided for each function, one where the length of the string (in bytes) is passed as a separate argument, and one where the string is passed as a null-terminated C-style string.
- A string is deemed to contain valid UTF-8 if it is compliant with RFC 3629, meaning that only 1-, 2-, 3-, and 4-byte sequences are allowed. Values above
U+10ffff are also not allowed.
- Six types of functions are provided:
-
isValid, which checks for validity, per RFC 3629, of a (candidate) UTF-8 string. "Overlong values", that is, values encoded in more bytes than necessary, are not tolerated; nor are "surrogate values", which are values in the range [U+d800 .. U+dfff].
-
advanceIfValid and advanceRaw, which advance some number of Unicode code points, each of which may be encoded in multiple bytes in a UTF-8 string. advanceRaw assumes the string is valid UTF-8, while advanceIfValid checks the input for validity and stops advancing if a sequence is encountered that is not valid UTF-8.
-
numCodePointsIfValid and numCodePointsRaw, which return the number of Unicode code points in a UTF-8 string. Note that numCodePointsIfValid both validates a (candidate) UTF-8 string and counts the number of Unicode code points that it contains.
-
numBytesIfValid, which returns the number of bytes a specified number of Unicode code points occupy in a UTF-8 string.
-
getByteSize, which returns the length of a single UTF-8 encoded character.
-
appendUtf8Character, which appends a single Unicode code point to a UTF-8 string.
- Embedded null bytes are allowed in strings that are accompanied by an explicit length argument. Naturally, null-terminated C-style strings cannot contain embedded null code points.
- The UTF-8 format is described in the RFC 3629 document at: and in Wikipedia at:
-
- Empty Input Strings:
- The utility functions provided by this component consider the empty string to be valid UTF-8. For those functions that take input as a
(pointer, length) pair, if 0 == pointer and 0 == length, then the input is interpreted as a valid, empty string. However, if 0 == pointer and 0 != length, the behavior is undefined. All such functions have a counterpart that takes a lone pointer to a null-terminated (C-style) string. The behavior is always undefined if 0 is supplied for that lone pointer.
-
- Usage:
- In this section we show intended use of this component.
-
- Example 1: Validating Strings and Counting Unicode Code Points:
- In this usage example, we will encode some Unicode code points in UTF-8 strings and demonstrate those that are valid and those that are not.
- First, we build an unquestionably valid UTF-8 string: Then, we check its validity and measure its length: Next, we encode a lone surrogate value,
0xd8ab, that we encode as the raw 3-byte sequence "\xed\xa2\xab" to avoid validation: Then, we cannot use numCodePointsRaw to count the code points in stringWithSurrogate, since the behavior of that method is undefined unless the string is valid. Instead, the numCodePointsIfValid method can be used on strings whose validity we are uncertain of: Now, we encode 0, which is allowed. However, note that we cannot use any interfaces that take a null-terminated string for this case: Finally, we encode 0x3a (:) as an overlong value using 2 bytes, which is not valid UTF-8 (since : can be "encoded" in 1 byte):
-
- Example 2: Advancing Over a Given Number of Code Points:
- In this example, we will use the various
advance functions to advance through a UTF-8 string.
- First, build the string using
appendUtf8CodePoint, keeping track of how many bytes are in each Unicode code point: Then, declare a few variables we'll need: bsls::Types::IntPtr rc;
int status;
const char *result;
const char *const start = string.c_str();
advanceRaw or advanceIfValid: rc = bdlde::Utf8Util::advanceRaw( &result, start, 2);
assert(2 == rc);
assert(3 + 2 == result - start);
rc = bdlde::Utf8Util::advanceIfValid(&status, &result, start, 2);
assert(0 == status);
assert(2 == rc);
assert(3 + 2 == result - start);
rc = bdlde::Utf8Util::advanceRaw( &result, start, 3);
assert(3 == rc);
assert(3 + 2 + 1 == result - start);
rc = bdlde::Utf8Util::advanceIfValid(&status, &result, start, 3);
assert(0 == status);
assert(3 == rc);
assert(3 + 2 + 1 == result - start);
rc = bdlde::Utf8Util::advanceRaw( &result, start, 4);
assert(4 == rc);
assert(3 + 2 + 1 + 4 == result - start);
rc = bdlde::Utf8Util::advanceIfValid(&status, &result, start, 4);
assert(0 == status);
assert(4 == rc);
assert(3 + 2 + 1 + 4 == result - start);
advanceIfValid, and wind up stopping when we encounter invalid input. The behavior of advanceRaw is undefined if it is used on invalid input, so we cannot use it here. Also note that we will stop at the beginning of the invalid Unicode code point, and not at the first incorrect byte, which is two bytes later: rc = bdlde::Utf8Util::advanceIfValid(&status, &result, start, INT_MAX);
assert(0 != status);
assert(5 == rc);
assert(3 + 2 + 1 + 4 + 4 == result - start);
assert(static_cast<int>(string.length()) > result - start);
string[3 + 2 + 1 + 4 + 4 + 2] = static_cast<char>(0x8a);
advanceIfValid does not return an error (non-zero) value to status when it encounters the end of the string: rc = bdlde::Utf8Util::advanceRaw( &result, start, INT_MAX);
assert(8 == rc);
assert(3 + 2 + 1 + 4 + 4 + 3 + 1 + 1 == result - start);
assert(static_cast<int>(string.length()) == result - start);
rc = bdlde::Utf8Util::advanceIfValid(&status, &result, start, INT_MAX);
assert(0 == status);
assert(8 == rc);
assert(3 + 2 + 1 + 4 + 4 + 3 + 1 + 1 == result - start);
assert(static_cast<int>(string.length()) == result - start);
-
- Example 3: Validating UTF-8 Read from a bsl::streambuf:
- In this usage example, we will demonstrate reading and validating UTF-8 from a stream.
- We write a function to read valid UTF-8 to a
bsl::string. We don't know how long the input will be, so we don't know how long to make the string before we start. We will grow the string in small, 32-byte increments. int utf8StreambufToString(bsl::string *output,
bsl::streambuf *sb)
{
enum { k_READ_LENGTH = 32 };
output->clear();
while (true) {
bsl::size_t len = output->length();
output->resize(len + k_READ_LENGTH);
int status;
IntPtr numBytes = bdlde::Utf8Util::readIfValid(&status,
&(*output)[len],
k_READ_LENGTH,
sb);
BSLS_ASSERT(0 <= numBytes);
BSLS_ASSERT(numBytes <= k_READ_LENGTH);
output->resize(len + numBytes);
if (0 < status) {
BSLS_ASSERT(k_READ_LENGTH - 4 < numBytes);
continue;
}
else if (0 == status) {
return 0;
}
else {
BSLS_LOG_ERROR("Invalid UTF-8 error %s at position %u.\n",
bdlde::Utf8Util::toAscii(status),
static_cast<unsigned>(output->length()));
return -1;
}
}
}
