baljsn_jsontokenizer

Detailed Description

Provide a tokenizer for viewing parts of a bdljsn::Json object.

Outline

• Purpose
• Classes
• Description
  • Tokens
    • JSON Object Members
    • JSON Array Elements
  • Usage
  • Example 1: Basic Usage

Purpose

Provide a tokenizer for viewing parts of a bdljsn::Json object.

Classes

• baljsn::JsonTokenizer: tokenizer for viewing parts of a bdljsn::Json

See also

baljsn_decoder, baljsn_tokenizer, baljsn_converter

Description

This component provides a class, baljsn::JsonTokenizer, that iterates through the structure of a bdljsn::Json object and provides Json type and (for some types) value data for each subordinate bdljsn::Json object. The top-level bdljsn::Json object must be of either bdljsn::JsonArray or bdldjn::JsonObject type. Both of those types can contain subordinate objects that are themselves either arrays or objects or any of the four scalar types. The structure is traversed in depth-first order.

The set of tokens returned by this tokenizer is isomorphic with the tokens returned by baljsn_tokenizer for use by baljsn_decoder . The only significant difference is that the latter tokenizer always returns values as bsl::string_views whereas this tokenizer returns bsl::string_view for element names (the name portion of a bdljsn::JsonObject member) but the value of an element value (the value portion of a bdljsn::JsonObject member or an element of a bdljsn::JsonArray) is returned as a const bdljsn::Json *`.

Tokens

The basic usage pattern for this tokenizer is to iteratively call the advanceToNextToken method. If the return value is 0 (success) the enumerated token value can be obtained by calling the tokenType method. advanceToNextToken fails when the tokenizer has been advanced past the last token.

The following table lists describes the set of enumerated values that can be returned by this tokenizer.

+-----------------+--------------------+-----------------------------+
|Enumerator       |Description         | Value                       |
+-----------------+--------------------+-----------------------------+
|e_BEGIN          |starting token      | none                        |
|e_ELEMENT_NAME   |element name        | value(bsl::string_view *)   |
|e_START_OBJECT   |start of an object  | none                        |
|e_END_OBJECT     |end of an object    | none                        |
|e_START_ARRAY    |start of an array   | none                        |
|e_END_ARRAY      |end of an array     | none                        |
|e_ELEMENT_VALUE  |scalar element value| value(const bdljsn::Json **)|
|e_ERROR          |error token         | none                        |
+-----------------+--------------------+-----------------------------+

Note that e_BEGIN is never returned by tokenType. That token type is the "prior" token that is discarded in the first call to advanceToNextToken after construction or reset.

Notice that two of the enumerated values – e_ELEMENT_NAME and e_ELEMENT_VALUE – have additional information that can be obtained via the appropriate value overload.

JSON Object Members

Recall that JSON Objects consist of sequences of name/value pairs called "members". e_ELEMENT_NAME is returned when the tokenizer is positioned to the name portion of a member and e_ELEMENT_VALUE is returned when the tokenizer is advanced to the associated scalar value. The appropriate value overload is used to get the actual member name (a bsl::string_view) and member value (a pointer to a bdljsn::Json object).

Also recall that the value portion of a member need not be scalar. Those can also consist of a subordinate JSON object or JSON array, in whice case, e_START_OBJECT or e_START_ARRAY, repsectively, are returned.

JSON Array Elements

e_ELEMENT_VALUE is also returned for each scalar element when the tokenizer is advancing through an array and value is invoked to get the value of that element. If the element is non-scalar – i.e., a subordinate object or array – the appropriate e_START_* token is returned.

Usage

This section illustrates intended use of this component.

Example 1: Basic Usage

The baljsn::JsonTokenizer allows the user to iteratively examine each of the piece-parts of a bdljsn::Json object having arbitrary complexity — containing subordinate JsonArrays/JsonObjects — as well as the scalar Json types. Suppose one must visualize a bdljsn::Json object having modest complexity:

{ "name"        : "Bob",
  "homeAddress" : { "street" : "Lexington Ave",
                    "city"   : "New York City",
                    "state"  : "New York"
                    },
  "age"         : 21
}

First, create JsonDoc, an ASCII string an representation of the above JSON document (having required escape sequences and quoting) – elided.

Then, create json, the programmatic representation of the JSON document.

 bdljsn::Json json;
 int          rc = bdljsn::JsonUtil::read(&json, JsonDoc);
 assert(0 == rc);

Next, define two macros and an array of indentation strings to simplify the formatting code.

 const char *const indents[] = { ""   // Four spaces per level.
                               , "    "
                               , "        "
                               , "                "
                               };
 
 #define PRINT(L, X) bsl::cout << indents[L] << (X) << bsl::endl;
 
 #define PRINTV(L, X, V) bsl::cout            \$
                         << indents[L]        \$
                         << (X) << ": "       \$
                         << (V) << bsl::endl;
 
 int level = 0; // indentation level

Then, create tokenizer, a baljsn::JsonTokenizer, that will generate a sequence of tokens (and sometimes values as well) for the previously initialized json object.

 baljsn::JsonTokenizer tokenizer(&json);
 
 while (0 == tokenizer.advanceToNextToken()) {

Now, for each token in the sequence, dispatch to the appropriate action. In each case, we output the value of the token at the current level of indentation.

For tokens that indicate the start of an JsonArray or JsonObject, we increase the level of indentation. That level is decreased when the end of the JsonArray or JsonObject is reported.

We expect two tokens to be generated for each member of a JsonObject: the element name and an associated value. In such cases we use the value method of extract the name (provided via bsl::string_view) and the value (itself a bdljsn::Json object).

Be aware that a e_ELEMENT_VALUE is also returned for each scalar element of a JsonArray – not featured in this example.

     baljsn::JsonTokenizer::TokenType tokenType = tokenizer.tokenType();
 
     switch(tokenType) {
         case baljsn::JsonTokenizer::e_START_ARRAY:   {
             PRINT(level, tokenType);
             ++level;
         } break;
         case baljsn::JsonTokenizer::e_END_ARRAY:     {
             --level;
             PRINT(level, tokenType);
         } break;
         case baljsn::JsonTokenizer::e_START_OBJECT:  {
             PRINT(level, tokenType);
             ++level;
         } break;
         case baljsn::JsonTokenizer::e_END_OBJECT:    {
             --level;
             PRINT(level, tokenType);
         } break;
         case baljsn::JsonTokenizer::e_ELEMENT_NAME:  {
             bsl::string_view sv;
             int              rc = tokenizer.value(&sv);
             assert(0 == rc);
 
             PRINTV(level, tokenType, sv);
         } break;
         case baljsn::JsonTokenizer::e_ELEMENT_VALUE: {
             const bdljsn::Json *jp;
             int                 rc = tokenizer.value(&jp);
             assert(0 == rc);
             assert( jp);
             assert(!jp->isArray());
             assert(!jp->isObject());
 
             PRINTV(level, tokenType, *jp);
         } break;
         case baljsn::JsonTokenizer::e_ERROR:         {
             PRINT(level, tokenType);
         } break;
         default: {
             assert(false && !"reachable");
         }
     }
 }

Notice, that the e_ELEMENT_VALUE is always associated with a scalar bdljsn::Json object. If the value portion of a member is itself an JsonArray (or JsonObject) then expect e_START_ARRAY (or e_START_OBJECT) at that point.

Finally, we can inspect the output for this bdljsn::Json object:

 START_OBJECT
     ELEMENT_NAME: homeAddress
     START_OBJECT
         ELEMENT_NAME: state
         ELEMENT_VALUE: "New York"
         ELEMENT_NAME: city
         ELEMENT_VALUE: "New York City"
         ELEMENT_NAME: street
         ELEMENT_VALUE: "Lexington Ave"
     END_OBJECT
     ELEMENT_NAME: age
     ELEMENT_VALUE: 21
     ELEMENT_NAME: name
     ELEMENT_VALUE: "Bob"
 END_OBJECT