Component of the Week #1: `bdlb_tokenizer`

Summary:

Provides a class for tokenizing text.

The task of extracting information from textual data - be it a simple comma-separated list of integers or extracting path components - comes up quite often during development. There is a BDE component that helps you to avoid writing such parsers by hand: bdlb_tokenizer.

The bdlb_tokenizer component provides a class bdlb::Tokenizer that makes handling such parsing tasks simple. All you need to do is give it the text you want to tokenize, the delimiter(s) that separate the tokens and you can use it as a range of tokens. For example, it’s easy to collect all the tokens in a vector or output them into a stream:

const char* text = "Mary had a little lamb";

bdlb::Tokenizer tokenizer(text, " ");
bsl::vector<bsl::string_view> tokens(tokenizer.begin(), tokenizer.end());

bsl::copy(tokenizer.begin(),
          tokenizer.end(),
          bsl::ostream_iterator<bsl::string_view>(bsl::cout, "\n"));

… or compute the total length of path segments without separators:

const char *text = "C:/some\\mixed delimiter\\windows/path";

bdlb::Tokenizer tokenizer(text, "\\/");
bsl::cout << bsl::accumulate(tokenizer.begin(),
                             tokenizer.end(),
                             0,
                             [](size_t                  accumulator,
                                const bsl::string_view& token) {
                                 return accumulator + token.length();
                             });

… or if you need a custom loop, you can simply use a range-based for:

const char *text = "/path/to/my/service/v1/resource";

bdlb::Tokenizer tokenizer(text, "/");
int tokenCountBeforeV1 = 0;
for (const bsl::string_view& token : tokenizer) {
    if ("v1" == token) break;
    ++tokenCountBeforeV1;
}

There’s more to bdlb::Tokenizer than the simple examples we’ve shown here - there’s a notion of hard and soft delimiters that treat consecutive delimiters differently, access to the actual delimiters separating the tokens, and more. Check out the documentation for bdlb_tokenizer for details.

Component of the Week #1: bdlb_tokenizer

Component of the Week #1: `bdlb_tokenizer`