ball_recordjsonformatter

Detailed Description

Provide a formatter for log records that renders output in JSON.

Outline

• Purpose
• Classes
• Description
  • Simplified Record Format Specification
  • JSON Record Format Specification
    • Field Format Specification
    • Verifying the Format Specification for setJsonFormat
      • The "timestamp" field format
      • The "pid" (process Id) field format
      • The "tid""ktid" (threadkernel thread Id) field format
      • The "file" field format
      • The "line" field format
      • The "category" field format
      • The "severity" field format
      • The "message" field format
      • The "attributes" format
      • A user-defined attribute format
  • The Record Separator
  • Usage
    • Example: Format log records as JSON and render them to stdout

Purpose

Provide a formatter for log records that renders output in JSON.

Classes

• ball::RecordJsonFormatter: formatter for rendering log records in JSON

See also

ball_record, ball_recordattributes

Description

This component provides a function object class, ball::RecordJsonFormatter, that formats a log record as JSON text elements according to a format specification (see {Record Format Specification}). ball::RecordJsonFormatter is designed to match the function signature expected by many concrete ball::Observer implementations that publish log records (for example, see ball::FileObserver2::setLogFileFunctor).

NOTE: ball::RecordJsonFormatter renders individual log records as JSON, but, for example, a resulting log file would contain a sequence of JSON strings, which is not itself valid JSON text.

Simplified Record Format Specification

RecordJsonFormatter supports a simplified format using printf-style (%-prefixed) conversion specifications via the setSimplifiedFormat method. This format provides a more concise way to specify common logging patterns without the verbosity of JSON arrays.

Note: The qjson:// scheme that uses this simplified format is registered by ball::RecordFormatterRegistryUtil, not by this component. Other schemes could be registered to use the same simplified format syntax.

The following table lists the %-prefixed conversion specifications that are recognized within a simplified format specification:

%d - timestamp in 'DDMonYYYY_HH:MM:SS.mmm' format (28AUG2020_14:43:50.375)
%i - timestamp in ISO 8601 format without fractional seconds
%I - timestamp in ISO 8601 format with millisecond precision
%T - thread Id in hexadecimal
%t - thread Id in decimal
%K - kernel thread Id in hexadecimal
%k - kernel thread Id in decimal
%p - process Id
%F - filename (basename of __FILE__ only)
%f - filename (full path from __FILE__)
%l - line number
%c - category name
%s - severity
%m - log message
%A - all user-defined attributes
%a[name] - specific user-defined attribute with the given name

Field specifications may be separated by whitespace (spaces, tabs, newlines) or commas, which are ignored by the parser. For example, these format specifications are equivalent:

"%d %T %s %c %m"
"%d, %T, %s, %c, %m"
"%d,%T,%s,%c,%m"

For example, the format specification:

"%d %T %s %c %m"

passed to setSimplifiedFormat would result in a log record like:

{ "timestamp": "28AUG2020_14:43:50.375",
  "tid": "0xA7654EFF3540",
  "severity": "INFO",
  "category": "MyCategory",
  "message": "Hello, world!"
}

Each %-prefixed field is rendered as a JSON key-value pair with a default field name (e.g., "timestamp", "tid", "severity"). Field names can be customized by prefixing a format specifier with <fieldName>:, for example: "myTime:%d" uses "myTime" as the field name instead of "timestamp". For more advanced formatting options, use the full JSON array format specification described below.

JSON Record Format Specification

A full featured format specification is, itself, a JSON array, supplied to a RecordJsonFormatter object by the setJsonFormat function. If no format is specified, the default format is used. Each array element specifies the format of a log record field or a user-defined attribute.

Note: The json:// scheme that uses this JSON array format is registered by ball::RecordFormatterRegistryUtil, not by this component. Other schemes could be registered to use the same JSON output format, like how qjson:// is an additional format syntax also resulting in JSON log records.

Here is a simple example:

[{"timestamp":{"format":"iso8601"}}, "pid", "tid", "severity", "message"]

would a result in a log record like:

{ "timestamp": "2020-08-28T14:43:50.375Z",
  "pid": 2313,
  "tid": 12349388604,
  "severity": "INFO",
  "message": "Hello, world!"
 }

The format specification is a JSON array, each element of which can be one of the following:

• A string containing the name of the fixed record field or the name of the user-defined attribute, in which case the field or attribute will be published in the default format. For example, [timestamp], would display the timestamp as: {"timestamp": "2020-08-28T14:43:50.375Z"}.
• A JSON object having the name of the fixed field or user-defined attribute and a set of key-values pairs used to customize the output format. For example, [{"timestamp": {"name": "My Time", "format": "bdePrint"}}], would display the timestamp as: {"My Time": "28AUG2020_14:43:50.375"}.

Field Format Specification

The following table lists the predefined string values for each fixed field and user-defined attributes in the log record:

   Tag                 Description                   Example
--------------    -------------------------        -------------
"timestamp"       creation date and time           ["timestamp"]
"pid"             process id of creator            ["pid"]
"tid"             thread id of creator             ["tid"]
"ktid"            kernel thread id of creator      ["ktid"]
"file"            file where created  (__FILE__)   ["file"]
"line"            line number in file (__LINE__)   ["line"]
"category"        category of logged record        ["category"]
"severity"        severity of logged record        ["severity"]
"message"         log message text                 ["message"]
"attributes"      all user-defined attributes      ["attributes"]
<attribute name>  specific user-defined attribute  ["bas.uuid"]

The output format of each field can be customized by replacing a string value in the JSON array with a JSON object having the same name and a set of key-value pairs (attributes).

Verifying the Format Specification for setJsonFormat

The sections that follow describe the set of fields that can be provided in the format specification supplied to setJsonFormat. RecordJsonFormatter::setJsonFormat will ignore fields in the provided format specification that are unknown, but will report an error if a known field contains a property that is not supported. For example: a format specification '["pid", { "timestamp" : {"unknown field!": "value"} }] will be accepted, but ["pid", {"timestamp": {"format": "unknown format" }}] will produce an error.

Each key-value pair of a JSON object that specifies a format of an output of a fixed record field or a user-defined attribute has the following constrains:

• The key is a string of known value listed in the column "Key" in the tables below. Any string that does not match the listed values is ignored.
• The value is a string of known value (except for the "name" key) in the column "Value Constraint" in the tables below. If the value does not match the string values specified in the tables, the format specification is considered to be inconsistent with the expected schema, and is rejected by the RecordJsonFormatter::setJsonFormat method.

The "timestamp" field format

The format attributes of the "timestamp" object are given in the following table:

                                            Value             Default
           Key              Description     Constraint        Value
------------------------  ----------------  -----------     ------------
"name"                    name by which     JSON string     "timestamp"
                          "timestamp" will
                          be published
 
"format"                  datetime format   "iso8601",     "iso8601"
                                            "bdePrint"
                                            (*Note*)
 
"fractionalSecPrecision"  second precision  "none",         "milliseconds"
                                            "milliseconds",
                                            "microseconds"
 
"timeZone"                time zone         "utc",          "utc"
                                            "local"

Note: The default "bdePrint" format denotes the following datetime format:

 DDMonYYYY_HH:MM:SS.mmm

For example, the following record format specification:

 [ { "timestamp": { "name": "Time",
                    "fractionalSecPrecision": "microseconds",
                    "timeZone": "local" } }
 ]

would a result in a log record like:

 { "Time": "28AUG2020_17:43:50.375345" }

The "pid" (process Id) field format

The format attributes of the process Id field are given in the following table:

                                                 Value        Default
  Key                 Description                Constraint   Value
------    -------------------------------------  -----------  -------
"name"    name by which "pid" will be published  JSON string  "pid"

For example, the following record format specification:

 [ { "pid": { "name": "Process Id" } } ]

would a result in a log record like:

 { "Process Id": 2313 }

The "tid""ktid" (threadkernel thread Id) field format

The format attributes of the thread Id field are given in the following table:

                                                   Value        Default
  Key                   Description                Constraint   Value
--------  -------------------------------------  -----------  ------------
"name"    name by which "tid"/"ktid" will be     JSON string  "tid"/"ktid"
          published
 
"format"  output format                          "decimal",   "decimal"
                                                 "hex"

For example, the following record format specification:

 [ { "tid": { "name": "Thread Id",
              "format": "hex" } }
 ]

would a result in a log record like:

 { "Thread Id": 0xA7654EFF3540 }

The "file" field format

The format attributes of the "file" field are given in the following table:

                                                              Default
  Key          Description             Value Constraint       Value
------    --------------------  ----------------------------- -------
"name"    name by which "file"  JSON string                   "file"
          will be published
 
"path"    file path             "full" (__FILE__),            "full"
                                "file" (basename of __FILE__)

For example, the following record format specification:

 [ { "file": { "name": "File",
               "path": "file" } }
 ]

would a result in a log record like:

 { "File": "test.cpp" }

The "line" field format

The format attributes of the "line" field are given in the following table:

                                                  Value        Default
  Key                Description                  Constraint   Value
------   ---------------------------------------  -----------  -------
"name"   name by which "line" will be published   JSON string  "line"

For example, the following record format specification:

 [ { "line": { "name": "Line" } } ]

would a result in a log record like:

 { "Line": 512 }

The "category" field format

The format attributes of the "category" field are given in the following table:

                                                    Value         Default
  Key                 Description                   Constraint    Value
------  ------------------------------------------  -----------  ----------
"name"  name by which "category" will be published  JSON string  "category"

For example, the following record format specification:

 [ { "category": { "name": "Category" } } ]

would a result in a log record like:

 { "category": "Server" }

The "severity" field format

The format attributes of the "severity" field are given in the following table:

                                                    Value         Default
  Key                  Description                  Constraint    Value
------  ------------------------------------------  -----------  ----------
"name"  name by which "severity" will be published  JSON string  "severity"

For example, the following record format specification:

 [ { "severity": { "name": "severity" } } ]

would a result in a log record like:

 { "Severity": "ERROR" }

The "message" field format

A message is a JSON string which is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes: (", \, \/, , \f,
, \r, \t, \u{4 hex digits}).

The format attributes of the "message" field are given in the following table:

                                                   Value         Default
  Key                 Description                  Constraint    Value
------  -----------------------------------------  -----------  ---------
"name"  name by which "message" will be published  JSON string  "message"

For example, the following record format specification:

 [ { "message": { "name": "msg" } } ]

would a result in a log record like:

 { "msg": "Log message" }

The "attributes" format

The "attributes" JSON object has no attributes. For example, the following record format specification:

 [ "attributes" ]

would (assuming there are two attributes "bas.requestid" and "mylib.security") result in a log record like:

 { "bas.requestid": 12345, "mylib.security": "My Security" }

A user-defined attribute format

Each user-defined attribute has a single "name" attribute that can be used to rename the user-defined attribute:

                                       Value        Default
  Key         Description              Constraint   Value
------   ----------------------------  -----------  -------
"name"   name by which a user-defined  JSON string   none
         attribute will be published

For example, the following record format specification:

 [ { "bas.uuid": { "name": "BAS.UUID" } } ]

would a result in a log record like:

 { "BAS.UUID": 3593 }

The Record Separator

The record separator is a string that is printed after each formatted record. The default value of the record separator is a single newline, but it can be set to any string of the user's choice using the RecordJsonFormatter::setRecordSeparator function.

Usage

This section illustrates intended use of this component.

Example: Format log records as JSON and render them to stdout

Suppose an application needs to format log records as JSON and output them to stdout.

First we instantiate a JSON record formatter:

 ball::RecordJsonFormatter formatter;

Next we set a format specification to the newly created formatter:

const int rc = formatter.setJsonFormat("[\"tid\",\"message\"]");
assert(0 == rc);  (void)rc;

The chosen format specification indicates that, when a record is formatted using formatter, the thread Id attribute of the record will be output followed by the message attribute of the record.

Then we create a default ball::Record and set the thread Id and message attributes of the record to dummy values:

ball::Record record;
 
record.fixedFields().setThreadID(6);
record.fixedFields().setMessage("Hello, World!");

Next, invocation of the formatter function object to format record to bsl::cout:

formatter(bsl::cout, record);

yields this output, which is terminated by a single newline:

{"tid":6,"message":"Hello, World!"}

Finally, we change the record separator and format the same record again:

formatter.setRecordSeparator("\n\n");
formatter(bsl::cout, record);

The record is printed in the same format, but now terminated by two newlines:

{"tid":6,"message":"Hello, World!"}