bdldfp_decimalconvertutil

Detailed Description

Outline

• Purpose
• Classes
• Description
  • Encoding Formats
    • IEEE Decimal Interchange Format
    • Multi-Width Encoding Format
    • Variable-Width Encoding Formats
  • Conversion between Binary to Decimal
  • Usage
    • Example 1: Sending Decimals As Octets Using Network Format
    • Example 2: StoringSending Decimals In Binary Floating-Point

Purpose

Provide decimal floating-point conversion functions.

Classes

• bdldfp::DecimalConvertUtil: Namespace for decimal FP conversion functions

See also

bdldfp_decimal, bdldfp_decimalplatform

Description

This component provides namespace, bdldfp::DecimalConvertUtil, containing functions that are able to convert between the native decimal types of the platform and various other possible representations, such as binary floating-point, network encoding formats.

Encoding Formats

This utility contains functions to encode decimal values to and from three different encoding formats:

• the IEEE decimal interchange format using decimal encoding for the significant (also known as the Densely Packed Decimal format, see IEEE
  1. - 2008, section 3.5.2, for more details)
• the multi-width encoding format, which is a custom format that can encode subsets of decimal values using a smaller number of bytes
• the variable-width encoding format, which is a custom format that is similar to the multi-width encoding format with the main difference being that it self describes its own width

64-bit decimal values encoded by the IEEE decimal interchange format always uses 8 bytes, which can be inefficient. The two custom encoding formats provided by this to enable more space efficient encoding of values commonly encountered by financial applications.

In the full IEEE encoding, 50 bits are used for the trailing bits of the mantissa, 13 bit is used for the combination field (exponent + special states to indicate NaN and Inf values + leading bits of the mantissa), and 1 bit is used for the significant. The basic idea for the custom encoding formats is that the mantissa and exponent of many values (in typical financial applications) can fit into fewer bits than those provided by the full encoding. We can define a set of narrow formats to encode these smaller values without loss of precision. For example, a ticker values less than 100 dollars with a 2 decimal places of precision can be encoded using a 2 bytes encoding, using no sign bit, 3 bits for the exponent, and 13 bits for the mantissa.

IEEE Decimal Interchange Format

The IEEE decimal interchange format is defined by the IEEE standard. 64 bit decimal values encoded by this format always uses 8 bytes. The decimalFromNetwork and decimalToNetwork functions can be used encode to and decode from this format.

Multi-Width Encoding Format

The multi-width encoding format uses a set of narrow encoding formats having sizes smaller than that used by the for IEEE format. Each of the narrower encoding format is used to encode a subset of values that can be represented by the full format. The following configuration is used to encode 64-bit decimal values:

|------|----------|----------|-----|----------|----------------|
| size | S (bits) | E (bits) |   B | T (bits) | max signficand |
|------|----------|----------|-----|----------|----------------|
|   1* |        0 |        1 |  -2 |        7 |            127 |
|   2  |        0 |        2 |  -3 |       14 |          16383 |
|   3  |        0 |        3 |  -6 |       21 |        2097151 |
|   4  |        1 |        5 | -16 |       26 |       67108863 |
|   5  |        1 |        5 | -16 |       34 |    17179869183 |
|------|-------------------------------------------------------|
|    8 |            FULL IEEE INTERCHANGE FORMAT**             |
|------|-------------------------------------------------------|
 
S = sign, E = exponent, B = bias, T = significant
 
* 1 byte encoding will be supported by the decoder but not the encoder. This
  is done due to the relatively large performance impact of adding the 1
  byte encoding to the encoder (10%). Preserving the encoding size in the
  decoder allows us to easily enable this encoding size at a later time.
 
** If the value to be encoded can not fit in the 5-byte encoding or is -Inf,
   +Inf, or Nan, then the full 8-byte IEEE format will be used.

Since the multi-width encoding format consists of subformats having varying widths, the size of the subformat used must be supplied long with the encoding to the decode function. This is not required for either the IEEE format or the variable-width encoding format.

The decimal64ToMultiWidthEncoding and decimal64FromMultiWidthEncoding can be used to encode to and decode from this format. Currently, only 64-bit decimal values are supported by this encoding format.

Variable-Width Encoding Formats

The variable-width encoding format can encode decimal values using a variable number of bytes, similar to the multi-width encoding format. The difference is that the variable-width encoding format can self-describe its own size using special state (typically, predicate bits), so the decode function does not require the size of the encoding to work. The following configuration is used to encode 64-bit decimal values:

|------|------------|---|---|-----|----|-----------------|
| size |          P | S | E |   B |  T | max significant |
|------|------------|---|---|-----|----|-----------------|
|    2 |        0b0 | 0 | 2 |  -2 | 13 |            8191 |
|    3 |       0b10 | 0 | 3 |  -4 | 19 |          524287 |
|    4 |       0b11 | 1 | 5 | -16 | 24 |        16777215 |
|------|------------|------------------------------------|
|    9 | 0b11111111 |        FULL IEEE FORMAT*           |
|------|------------|------------------------------------|
 
P = predicate (bit values)
S = sign (bits), E = exponent (bits), B = bias
T = significant (bits)
 
* If the value to be encoded can not fit in the 4-byte encoding or is -Inf,
  +Inf, or Nan, then the full 8-byte IEEE format will be used prefixed by a
  1 byte predicate having the value of 0xFF.

The decimal64ToVariableWidthEncoding and decimal64FromVariableWidthEncoding can be used to encode to and decode from this format. Currently, only 64-bit decimal values are supported by this encoding format.

Conversion between Binary to Decimal

The desire to convert numbers from binary to decimal format is fraught with misunderstanding, and is often accompanied by ill-conceived attempts to "correct rounding errors" and otherwise coerce results into aesthetically pleasing forms. In the Bloomberg environment, there is the additional complication of IBM/Perkin-Elmer/Interdata floating point. This is a floating-point format that uses a base-16 rather than a base-2 underlying representation, and the Bloomberg environment contains numbers that began as decimals, were converted to IBM format, and then were converted from IBM format to IEEE format.

Generically, when a decimal is converted to floating-point (using, for example, scanf from text, or DecimalConvertUtil::decimalToDouble), the result is the representable binary number nearest in value to that decimal. If there are two such, the one whose least significant bit is 0 is generally chosen. (This is also true for conversion of decimals to 32-bit Perkin-Elmer format, but when the Perkin-Elmer float is then converted to IEEE float, the resulting value is not the same as converting the decimal to IEEE float directly, because the Perkin-Elmer format can lose up to three bits of representation compared to the IEEE format.) Unless the decimal value is exactly a multiple of a power of two (e.g., 3.4375 = 55 * 1/16), the converted binary value cannot equal the decimal value, it can only be close. This utility provides decimal{,32,64,128}To{Float,Double} functions to convert decimal floating-point numbers to their closest binary floating-point values.

When converting from decimal to binary, it is hoped that two different decimal values (in the representable range of the target binary type) convert to two different binary values. There is a maximum number of significant digits for which this will be true. For example, all decimals with 6 significant digits convert to distinct float values, but 8589973000 and 8589974000, with 7 significant digits, convert to the same float value. Similarly, all decimals with 15 significant digits convert to unique double values but 900719925474.0992 and 900719925474.0993, with 16 significant digits, convert to the same double value. Over restricted ranges, the maximum may be higher - for example, every decimal value with 7 or fewer significant digits between 1e-3 and 8.5e9 converts to a unique float value.

Because binary floating-point values are generally not equal to their decimal progenitors, "converting from binary to decimal" does not have a single meaning, and programmers who seek such an operation therefore need to know and specify the conversion they want. Common examples of conversions a programmer might seek are listed below:

1. Express the value as its nearest decimal value.
   • For this conversion, use the conversion constructors:
   • Decimal{32,64,128}(value)
2. Express the value rounded to a given number of significant digits. (The significant digits of a decimal number are the digits with all leading and trailing 0s removed; e.g., 0.00103, 10.3 and 10300 each have 3 significant digits.) This conversion is the one that leads programmers to complain about "rounding error" (for example, .1f rounded to 9 digits is .100000001) but is the appropriate one to use when the programmer knows that the binary value was originally converted from a decimal value with that many significant digits.
   • For this conversion, use:
   • Decimal{32,64,128}From{Float,Double}(value, digits)
3. Express the value using the minimum number of significant digits for the type of the binary such that converting the decimal value back to binary will yield the same value. (Note that 17 digits are needed for double and 9 for float, so not all decimal types can hold such a result.)
   • For this conversion, use:
   • Decimal{64,128}FromFloat(value, 9) or
   • Decimal128FromDouble(value, 17)
4. Express the value using a number of decimal places that restores the original decimal value from which the binary value was converted, assuming that the original decimal value had sufficiently few significant digits so that no two values with that number of digits would convert to the same binary value. (That number is 15 for double and 6 for float in general but 7 over a limited range that spans [1e-3 .. 8.5e9]).
   • For this conversion, use:
   • Decimal{32,64,128}From{Float,Double}(value)
5. Express the value as the shortest decimal number that converts back exactly to the binary value. For example. given the binary value 0x3DCCCCCD above, that corresponding shortest decimal value is (unsurprisingly) .1, while the next lower value 0x3DCCCCCC has the shortest decimal .099999994 and the next higher value 0x3DCCCCCE has the shortest decimal .010000001. This is the most visually appealing result, but can be expensive and slow to compute.
   • For this conversion, use:
   • Decimal{32,64,128}From{Float,Double}(value, -1)
6. Express the value using a number of decimal places that restores the original decimal value assuming that it is a float which originated as an IBM/Perkin-Elmer/Interdata float value itself originally converted from a decimal value.
   • For this conversion, use:
   • Decimal{32,64,128}FromFloat(value, 6)
7. Express the value exactly as a decimal. For example, the decimal value .1 converts to the 32-bit IEEE float value 0x3DCCCCCD, which has the exact value .100000001490116119384765625. This conversion is seldom useful, except perhaps for debugging, since the exact value may have over
   1. digits, and as well cannot be represented as a decimal floating-point type since those types do not have enough digits.
   • For this conversion, use sprintf into a large-enough buffer:
   • char buf[2000]; double value; sprintf(buf, "%.1100f", value);
   • The result will have trailing 0s, which may be trimmed.
8. Express the value rounded to a given number of decimal places. (The decimal places of a decimal number are the number of digits after the decimal point, with trailing 0s removed; .01, 10.01, and 1000.01 each have two decimal places.) This conversion can be problematic when the integer portion of the value is large, as there may not be enough precision remaining to deliver a meaningful number of decimal places. As seen above, for example, for numbers near one trillion, there is not enough precision in a double for 4 decimal places.
   • For this conversion, use sprintf into a large-enough buffer:
   • char buf[2000]; double value; sprintf(buf, "%.*f", places, value);

Usage

This section shows the intended use of this component.

Example 1: Sending Decimals As Octets Using Network Format

Suppose you have two communicating entities (programs) that talk to each other using a binary (as opposed to text) protocol. In such protocol it is important to establish a so-called network format, and convert to and from that format in the protocol layer. The sender (suppose that it is an IBM server that has just finished an expensive calculation involving millions of numbers and needs to send the result to its client) will need to convert the data to network format before sending:

unsigned char  msgbuffer[256];
unsigned char *next = msgbuffer;
 
BDEC::Decimal64 number(BDLDFP_DECIMAL_DD(1.234567890123456e-42));
unsigned char   expected[] = {
                          0x25, 0x55, 0x34, 0xb9, 0xc1, 0xe2, 0x8e, 0x56 };
 
next = bdldfp::DecimalConvertUtil::decimalToNetwork(next, number);
 
assert(memcmp(msgbuffer, expected, sizeof(number)) == 0);

The receiver/client shall then restore the number from network format:

unsigned char  msgbuffer[] ={
                          0x25, 0x55, 0x34, 0xb9, 0xc1, 0xe2, 0x8e, 0x56 };
unsigned char *next = msgbuffer;
 
BDEC::Decimal64 number;
BDEC::Decimal64 expected(BDLDFP_DECIMAL_DD(1.234567890123456e-42));
 
next = bdldfp::DecimalConvertUtil::decimalFromNetwork(number, next);
 
assert(number == expected);

Example 2: StoringSending Decimals In Binary Floating-Point

Suppose you have two communicating entities (programs) that talk to each other using a legacy protocol that employs binary floating-point formats to send/receive numbers. So your application layer will have to store the decimal into a binary FP variable, ensure that it can be restored (in other words that it has "fit" into the binary type) when sending, and restore the decimal number (from the binary type) when receiving:

const BDEC::Decimal64 number(BDLDFP_DECIMAL_DD(1.23456789012345e-42));
 
typedef bdldfp::DecimalConvertUtil Util;
double dbl = Util::decimalToDouble(number);
 
if (Util::decimal64FromDouble(dbl) != number) {
    // Do what is appropriate for the application
}

Note that the above assert would probably be a lot more complicated if statement in production code. It may actually be acceptable to put the decimal onto the wire with certain amount of imprecision.

The receiver would then restore the number using the appropriate decimal64FromDouble function:

BDEC::Decimal64 restored = Util::decimal64FromDouble(dbl);
 
assert(number == restored);