March 2, 2026

BALL Record Formatter Plugins: Flexible Log Output Formatting

Introduction

BDE 4.36.0 introduces a new plugin-based architecture for BALL record formatting that allows applications to specify log output formats using URI-style scheme identifiers in configuration files. This enhancement enables developers to easily switch between different output formats — text, or JSON — and different configuration formats for JSON - the original JSON-based (json://), or the simplified JSON (qjson://) - without code changes, making it particularly valuable for log aggregation systems, monitoring tools, and cloud-native deployments.

The appropriate formatter is selected based on the URI scheme prefix in the format specification string. Applications can use the following format schemes: text://, json://, and qjson://. The latter two both result in JSON formatted log output, only the syntax of the format specification differs.

Quick Start: Using Format Schemes

Prior to this release, BALL logging configuration only supported text-based format strings using printf-style specifiers. Now, format specifications can include a scheme identifier that selects the formatter type:

// Traditional text format (backward compatible - no scheme required)
ball::FileObserver2 observer;
observer.setFormat("%d %p:%t %s %f:%l %c %m\n");

// Text format with explicit scheme (equivalent to above)
observer.setFormat("text://%d %p:%t %s %f:%l %c %m\n");

// Simplified JSON format (qjson) with printf-style specifiers
observer.setFormat("qjson://%i %s %m");

// JSON format with array notation
observer.setFormat("json://[\"timestamp\",\"severity\",\"message\"]");

#ifdef BSLS_COMPILERFEATURES_SUPPORT_RAW_STRINGS
// JSON format with more readable raw string literal
observer.setFormat(R"(json://["timestamp","severity","message"])");
#endif

Backward Compatibility: When no scheme is provided in the format specification, the text formatter is used by default. All existing format strings without a scheme prefix continue to work exactly as before, ensuring seamless migration.

Configuration files can now specify these formats directly:

<LoggingConfig>
    ...
    <LogFileFormat>qjson://%i %s %c %m</LogFileFormat>
</LoggingConfig>

When an unknown or invalid scheme is encountered, BALL automatically falls back to a default text format to ensure logging continues uninterrupted.

Supported Format Schemes

Text Format (`text://`)

The traditional BALL text formatter using printf-style format specifiers. This is the default format when no scheme is specified.

Syntax: text://FORMAT_STRING

Common Format Specifiers:

Specifier	Description
`%d`	Timestamp (DDMonYYYY_HH:MM:SS.mmm)
`%D`	Timestamp with microseconds (DDMonYYYY_HH:MM:SS.mmmuuu)
`%dtz`	Timestamp with timezone (DDMonYYYY_HH:MM:SS.mmm+0000)
`%Dtz`	Timestamp with microseconds and timezone
`%i`	ISO 8601 timestamp (no fractional seconds)
`%I`	ISO 8601 timestamp (with milliseconds)
`%O`	ISO 8601 timestamp (with microseconds)
`%p`	Process ID
`%t`	Thread ID (decimal)
`%T`	Thread ID (hexadecimal)
`%k`	Kernel thread ID (decimal)
`%K`	Kernel thread ID (hexadecimal)
`%s`	Severity level (INFO, WARN, ERROR, etc.)
`%f`	Source file name (full path from `__FILE__`)
`%F`	Source file name (basename of `__FILE__` only)
`%l`	Line number
`%c`	Category name
`%m`	Log message
`%x`	Log message (non-printable characters in hex)
`%X`	Log message (entirely in hex)
`%a`	Unlogged attributes (see below)
`%A`	All attributes (see ball_scoped_attributes)
`%a[name]`	Specific attribute by name
`%av[name]`	Specific attribute value only
`%u`	User fields
`%%`	Literal `%` character

Note on Attributes: %a logs only attributes not already printed by %a[name] or %av[name] specifiers elsewhere in the format. %A logs all attributes regardless. This prevents duplicate attribute output when specific attributes are already being logged individually.

Examples:

// Compact single-line format
observer.setFormat("text://%d %s %c %m\n");

// Detailed format with file location
observer.setFormat("text://%i [%s] %c (%f:%l) %m\n");

// Format including attributes
observer.setFormat("text://%d %p:%t %s %f:%l %c %a %m\n");

Sample Output (from the compact single-line format example):

23NOV2025_14:30:45.123 WARN TRADING.RISK Limit exceeded for account A12345

JSON Format (`json://`)

Structured JSON output using array notation for field specifications. Each log record is formatted as a complete JSON object with the specified fields.

Syntax: json://ARRAY_SPECIFICATION

Array Notation: Fields are specified using JSON array syntax:

["field"] - Single field
["field1","field2","field3"] - Multiple fields
[{"field":{"option":"value"}}] - Fields with formatting options

Available Fields:

timestamp - Date and time (DDMonYYYY_HH:MM:SS.mmm format by default)
pid - Process ID
tid - Thread ID
ktid - Kernel thread ID
severity - Severity level
file - Source file name
line - Line number
category - Category name
message - Log message
attributes - All attributes as nested JSON object
<attribute name> - Specific user-defined attribute by name

Examples:

// Essential fields only
observer.setFormat("json://[\"timestamp\",\"severity\",\"message\"]");

// Comprehensive logging with metadata
observer.setFormat("json://[\"timestamp\",\"pid\",\"tid\",\"severity\","
                   "\"category\",\"file\",\"line\",\"message\"]");

// With attributes for structured logging
observer.setFormat("json://[\"timestamp\",\"severity\",\"category\","
                   "\"attributes\",\"message\"]");

// Specific attribute by name
observer.setFormat("json://[\"timestamp\",\"severity\",\"bas.uuid\",\"message\"]");

Sample Output (from the third example with attributes):

{
  "timestamp": "23NOV2025_14:30:45.123",
  "severity": "WARN",
  "category": "TRADING.RISK",
  "attributes": {},
  "message": "Limit exceeded for account A12345"
}

Simplified JSON Format (`qjson://`)

A convenience format that combines the structured nature of JSON with the familiar printf-style syntax. This format is particularly useful for quick prototyping and configuration by users familiar with traditional text formats.

Syntax: qjson://FORMAT_SPECIFIERS

Format Specifiers (subset of text format specifiers that map to JSON fields):

Specifier	JSON Field / Description
`%d`	`timestamp` (DDMonYYYY format)
`%i`	`timestamp` (ISO 8601 format without fractional seconds)
`%I`	`timestamp` (ISO 8601 format with milliseconds)
`%p`	`pid`
`%t`	`tid` (decimal)
`%T`	`tid` (hexadecimal)
`%k`	`ktid` (decimal)
`%K`	`ktid` (hexadecimal)
`%s`	`severity`
`%f`	`file` (full path)
`%F`	`file` (basename only)
`%l`	`line`
`%c`	`category`
`%m`	`message`
`%A`	`attributes` (all user-defined attributes)
`%a[name]`	specific attribute by name

Examples:

// Simple format
observer.setFormat("qjson://%i %s %m");

// Detailed format
observer.setFormat("qjson://%i %p:%t %s %c %f:%l %m");

// Custom field names using colon syntax
observer.setFormat("qjson://time:%i level:%s cat:%c msg:%m");

Sample Output (from the simple format):

{
  "timestamp": "2025-11-23T14:30:45.123Z",
  "severity": "WARN",
  "message": "Limit exceeded for account A12345"
}

Sample Output with Custom Field Names (from the custom names example):

{
  "time": "2025-11-23T14:30:45.123Z",
  "level": "WARN",
  "cat": "TRADING.RISK",
  "msg": "Limit exceeded for account A12345"
}

qjson Format Specification Syntax

The qjson:// format specification is stricter than the text format: it only allows %-specifiers (from the table above), whitespace, commas, and label-override prefixes. Arbitrary literal text is not permitted. In particular, a trailing \n is unnecessary (the record separator — a newline by default — is appended automatically and is independently configurable via setRecordSeparator) and will be rejected by the parser.

The full syntax rules are:

Whitespace (spaces, tabs, newlines) between specifiers is silently skipped and has no effect on the output.
Commas may optionally separate specifiers for readability (e.g., qjson://%i, %s, %m). Leading commas, trailing commas, and consecutive commas are errors.
Label overrides allow renaming the JSON key produced by a specifier. Place the desired key name followed by a colon (:) immediately before the %-specifier:
```
// "ts" instead of "timestamp", "lvl" instead of "severity"
observer.setFormat("qjson://ts:%i lvl:%s %c %m");
```
Optional whitespace is allowed between the colon and the %. Label overrides are not supported for %A (all attributes).
Everything else — including literal characters such as [, ], \n, or any punctuation that is not a comma — causes the parser to reject the format specification.

Choosing the Right Format

Use Text Format when:

Human readability is the primary concern
Logs are viewed directly in console or log files
Minimal parsing is required
Backward compatibility with existing log processors is needed

Use JSON Format when:

Logs are consumed by structured log aggregation systems (Splunk, Elasticsearch, etc.)
You need precise control over JSON field names and structure
Integrating with JSON-based monitoring and alerting systems
Working with cloud-native logging infrastructure

Use qjson Format when:

You want JSON output but prefer the simplicity of printf-style specifiers
Rapidly prototyping log formats
Transitioning from text to JSON formatting
Configuration simplicity is more important than explicit JSON structure control

Integration with File Observers

All BALL observers now support format scheme specification:

ball::FileObserver2 - via setFormat() method
ball::StreamObserver - via setFormat() method
ball::CstdioObserver - via setFormat() method
ball::FileObserver - via setFileLogFormat() and setStdoutLogFormat() methods
ball::AsyncFileObserver - via setFileLogFormat() and setStdoutLogFormat() methods

Note: FileObserver and AsyncFileObserver have separate methods for configuring file and stdout formats independently, while the newer observers use a single setFormat() method.

Configuration Example

#include <ball_fileobserver2.h>
#include <ball_loggermanager.h>
#include <ball_severityutil.h>

int main() {
    ball::LoggerManagerConfiguration lmc;
    ball::LoggerManagerScopedGuard lmGuard(configuration);

    bsl::shared_ptr<ball::FileObserver2> observer =
                                    bsl::make_shared<ball::FileObserver2>();

    // Set JSON format for structured logging
    observer->setFormat("json://[\"timestamp\",\"severity\",\"category\","
                        "\"file\",\"line\",\"message\"]");

    observer->enableFileLogging("application.log");

    ball::LoggerManager::singleton().registerObserver(observer, "default");

    // All log records will now be formatted as JSON
    BALL_LOG_SET_CATEGORY("APPLICATION.STARTUP");
    BALL_LOG_INFO << "Application initialized successfully";

    return 0;
}

Sample Output (written to application.log):

{
  "timestamp": "23NOV2025_14:35:12.456",
  "severity": "INFO",
  "category": "APPLICATION.STARTUP",
  "file": "main.cpp",
  "line": 19,
  "message": "Application initialized successfully"
}

Runtime Format Changes

Format specifications can be changed at runtime without restarting the application:

ball::FileObserver2 observer;

// Start with text format
observer.setFormat("text://%d %s %m\n");

// ... application runs ...

// Switch to JSON format dynamically
// (e.g., in response to configuration change or admin command)
observer.setFormat("json://[\"timestamp\",\"severity\",\"message\"]");

Fallback Behavior

When an unrecognized scheme is specified or format parsing fails, BALL’s behavior depends on whether the observer already has a formatter configured:

No existing formatter: Falls back to a default text format
Existing formatter: Keeps the current formatter unchanged

The setFormat()/setFileLogFormat()/setStdoutLogFormat() methods return a non-zero value to indicate an error.

Default Fallback Format: text://\n%d %p:%t %s %f:%l %c %a %m\n

This ensures that misconfiguration doesn’t result in lost log messages.

ball::FileObserver2 observer;

// First setFormat call - no existing formatter
int rc = observer.setFormat("xml://some-unknown-format");
assert(0 != rc);
// Falls back to default text format
// Logging will use: "\n%d %p:%t %s %f:%l %c %a %m\n"

// Second setFormat call - existing formatter is present
rc = observer.setFormat("another-bad://format");
assert(0 != rc);
// Keeps the current formatter (default text format from above)
// Does NOT replace with fallback again

Migration Guide

Existing Applications

Existing BALL applications require no changes. Format strings without a scheme prefix are automatically treated as text format:

// These are equivalent:
observer.setFormat("%d %s %m\n");
observer.setFormat("text://%d %s %m\n");

Adopting JSON Format

To migrate to JSON format:

Identify critical log messages that should be structured
Choose JSON fields that match your log analysis needs
Update format specification using qjson:// or json:// scheme
Verify output in your log aggregation system
Update log parsing rules if necessary

Example migration:

// Before: text format
observer.setFormat("%d [%s] %c - %m\n");

// After: equivalent qjson format (simpler syntax)
observer.setFormat("qjson://%d %s %c %m");

Best Practices

Use JSON for production logging to structured log aggregation systems
Include timestamps in all formats for temporal analysis
Add category information to enable log filtering and routing
Consider file and line numbers for debugging but balance against log volume
Include attributes when using contextual logging (see ball_scoped_attributes)
Test format changes in development before deploying to production
Document format specifications in configuration management systems
Use qjson format for easier configuration by operations teams

Performance Considerations

Text format is the most efficient for high-throughput logging
JSON formats add parsing and formatting overhead but remain suitable for most applications
Field selection impacts performance — include only necessary fields
Pre-production testing recommended for performance-critical applications

BALL Record Formatter Plugins: Flexible Log Output Formatting

Introduction

Quick Start: Using Format Schemes

Supported Format Schemes

Text Format (text://)

JSON Format (json://)

Simplified JSON Format (qjson://)

qjson Format Specification Syntax

Choosing the Right Format

Integration with File Observers

Configuration Example

Runtime Format Changes

Fallback Behavior

Migration Guide

Existing Applications

Adopting JSON Format

Best Practices

Performance Considerations

See Also

Text Format (`text://`)

JSON Format (`json://`)

Simplified JSON Format (`qjson://`)