bdlb::Tokenizer_Data Class Reference

#include <bdlb_tokenizer.h>

List of all members.

Public Member Functions
	Tokenizer_Data (const bsl::string_view &softDelimiters)
	Tokenizer_Data (const bsl::string_view &softDelimiters, const bsl::string_view &hardDelimiters)
int	inputType (char character) const

---

Detailed Description

This component-private class is used to hold delimiter information. Each Tokenizer object will have, as a private data member, an object of this class, and will pass the address of that member to the (private) constructor of each TokenizerIterator object it issues:

      +--------------------------------------+
      |   ,--------------.                   |
      |  ( Tokenizer_Data )                  |
      |   `--------------'\                  |
      |          |         \                 |
      |          |     ,----*------------.   |
      |          |    ( TokenizerIterator )  |
      |          |    /`-----------------'   |
      |          |   /                       |
      |     ,----*--o-.                      |
      |    ( Tokenizer )                     |
      |     `---------'                      |
      +--------------------------------------+
      bdlb_tokenizer

See Component bdlb_tokenizer

---

Constructor & Destructor Documentation

bdlb::Tokenizer_Data::Tokenizer_Data

(

const bsl::string_view &

softDelimiters

)

[explicit]

bdlb::Tokenizer_Data::Tokenizer_Data	(	const bsl::string_view &	softDelimiters,
		const bsl::string_view &	hardDelimiters
	)

Create a Tokenizer_Data object and load the d_charTypes data member such that it has the same value as if this (overly prescriptive) algorithm were used: (I) initialize each entry in d_charTypes array to a value indicating that the character having that index as its (e.g., ASCII) representation is a token character; (II) then, for each character in the specified softDelimiters sequence, overwrite the element at the corresponding index in d_charTypes with a value that indicates that the character is a soft delimiter character; (III) finally, for each character in the specified hardDelimiters sequence, overwrite the element at the corresponding index with a distinct value that indicates the character is a hard delimiter* character. Note that duplicate delimiter characters in the respective inputs are naturally ignored, and that a character that appears in both sets would naturally be considered hard. Also note that it is entirely reasonable to state, in any public interface, that the behavior is undefined unless the characters in the union of the two delimiter sequences are unique.

---

Member Function Documentation

int bdlb::Tokenizer_Data::inputType

(

char

character

)

const

Return the input type of the specified character: 0 for token, 1 for soft delimiter, 2 for hard delimiter.

---

The documentation for this class was generated from the following file:

• bdlb_tokenizer.h