Topics: |

Values are rounded before storage or before display, depending on the format. Integer fields (format I) and packed decimal fields (format P) are rounded or truncated before being stored. Floating-point fields (formats F and D) and decimal floating-point fields (formats M and X) are stored as entered and rounded for display.

In general, when a final decimal digit is less than 5, the data value rounds down. A data value with a final digit of 5 or greater rounds up. The following rounding algorithm is used:

- The incoming value is multiplied by 10.
- This multiplication
repeats the same number of times as the number of decimal places
in the target format. For example, if 123.78 is input to a packed
decimal field with one decimal place, it is multiplied by 10 once:
1237.8

- Next, 0.5 is
added if the incoming value is positive or subtracted if the incoming value
is negative:
1237.8 + 0.5 = 1238.3

or, if the input value was -123.78,

-1237.8 - 0.5 = -1238.3

- The value is
truncated, and the decimal is shifted to the left.
123.8

or, if the original value was negative,

-123.8

The following chart illustrates the rounding differences between WebFOCUS numeric field formats. Subsequent topics discuss these differences in detail.

Format |
Type |
Format |
Input |
Stored |
Display |
---|---|---|---|---|---|

I |
Integer |
I3 |
123.78 |
0124 |
124 |

F |
Floating-Point Single-Precision |
F5.1 F3 |
123.78 123.78 |
123.7800 123.7800 |
124 123.8 |

D |
Floating-Point Double-Precision |
D3 D5.1 |
123.78 123.78 |
123.780000000000 123.780000000000 |
124 123.8 |

M |
Decimal Precision Floating-Point |
M5.1 M3 |
123.78 123.78 |
123.7800 123.7800 |
124 123.8 |

X |
Extended Decimal Precision Floating-Point |
X3 X5.1 |
123.78 123.78 |
123.780000000000 123.780000000000 |
124 123.8 |

P |
Packed |
P3 P5.1 |
123.78 123.78 |
0000124 00001238 |
124 123.8 |

Note: For floating-point fields (format D or F), the stored values of decimal numbers are in hexadecimal and may convert to a value very slightly less than the actual decimal number. When the final digit is 5, these numbers may round down instead of up. Using the SET FLOATMAPPING command to treat double precision numbers as decimal precision numbers can help alleviate this problem.

An integer value entered with no decimal places is stored as entered.

When a value with decimal places is entered into an integer field using a transaction, that value is rounded, and the result is stored. If the fractional portion of the value is less than 0.5, the value is rounded down; if the fractional portion of the value is greater than or equal to 0.5, the value is rounded up.

However, if an integer field is computed, the decimal portion of the resulting value is truncated, and the integer portion of the answer is stored (or printed). For example, if the result of a calculation is 123.78, the value stored is 123.

Format type F describes single-precision floating-point numbers stored internally in 4 bytes. Format F is comparable to COBOL COMP-1. Format type D describes double-precision floating-point numbers stored internally in 8 bytes. Format D is comparable to COBOL COMP-2.

Formats F and D store as many decimal places as are input, up to the limit of the storage allocated to the field. Format D is more accurate than format F for larger numbers, since D fields can store up to 15 significant digits, and format F fields are not accurate beyond a maximum of 8 digits. Floating-point fields are stored in a logarithmic format. The first byte stores the exponent. The remaining 3 or 7 bytes store the mantissa, or value.

When the number of decimal places input is greater than the number of decimal places specified in the format, F and D field values are stored as they are input, up to the limit of precision. These values are rounded for display according to the field format. For example, if 123.78 is entered in a floating-point field with one decimal place, 123.78 is stored, and 123.8 is displayed.

Format type M describes decimal precision floating-point numbers stored internally in 8 bytes. Format type X describes extended decimal precision floating-point numbers stored internally in 16 bytes.

Formats M and X store as many decimal places as are input, up to the limit of the precision. Format X is more accurate than format M for larger numbers, since X fields can store up to 37 significant digits, and M fields are not accurate beyond a maximum of 15 digits. Like formats F and D, these fields are stored in a logarithmic format. The first byte stores the exponent. The remaining bytes store the mantissa, or value, in a binary format. The primary difference between formats M and X and formats F and D is the base to which the exponent is applied. Formats M and X use base 10, eliminating the rounding issues seen in formats F and D, which use base 16.

When the number of decimal places input is greater than the number of decimal places stored in the format, M and X field values are stored as they are input, up to the limit of precision. These values are rounded for display according to the field format. For example, if 123.78 is entered in a format M or X field with one decimal place, 123.78 is stored, and 123.8 is displayed.

The command SET FLOATMAPPING = {__D__|M|X} is available to automatically determine how a floating-point number will be used and stored in subsequent HOLD files.
The default format is D.

In packed-decimal format (format type P), each byte contains two digits, except the last byte, which contains a digit and the sign (D for negative numbers, C for positive). Packed fields are comparable to COBOL COMP-3.

Packed field values are rounded to the number of digits specified in the field format before they are stored. When the number of decimal places input is greater than the number that can be stored, P field values are rounded first, then stored or displayed.

Packed fields are precisely accurate when sufficient decimal places are available to store values. Otherwise, since values are rounded before being stored, accuracy cannot be improved by increasing the number of digits displayed. For example, if 123.78 is input to a packed field with 1 decimal place, 123.8 is stored. If the field format is then changed to P6.2 using a COMPUTE or DEFINE, 123.80 will be displayed. If the format is changed to P6.2 in the Master File, 12.38 is displayed.

Note: Continuous improvement to our expression handler, providing more efficient and more accurate results, may expose some rounding differences between releases when using packed fields. Enhancements have improved the accuracy of the calculations when working with packed numbers. Rounding of a packed field is done at the time of storage, changing the actual number. This is different from precision-based fields, which round when they are displayed, ensuring that the original number is retained.

For floating-point fields (format F or D), the stored values of decimal numbers are in hexadecimal and may convert to a value very slightly less than the actual decimal number. When the final digit is 5, these numbers may round down instead of up.

The following example shows an input value with two decimal places, which is stored as a packed field with two decimal places, a packed field with one decimal place, a D field with one decimal place, an F field with one decimal place, an M field with one decimal place, and an X field with one decimal place:

Master File

FILE=FIVE, SUFFIX=FOC SEGNAME=ONLY, SEGTYPE=S1,$ FIELD=PACK2,,P5.2,$ FIELD=PACK1,,P5.1,$ FIELD=DOUBLE1,,D5.1,$ FIELD=FLOAT1,,F5.1,$ FIELD=MATH1,,M5.1,$ FIELD=XMATH1,,X5.1,$

Program to Load Data

This MODIFY creates a file with six fields: a P field with two decimal places, a P field with one decimal place, a D field with one decimal place, an F field with one decimal place, an M field with one decimal place, and an X field with one decimal place. The same data values are then loaded into each field.

CREATE FILE FIVE MODIFY FILE FIVE FIXFORM PACK2/5 PACK1/5 DOUBLE1/5 FLOAT1/5 MATH1/5 XMATH1/5 MATCH PACK2 ON MATCH REJECT ON NOMATCH INCLUDE DATA 1.05 1.05 1.05 1.05 1.05 1.05 1.15 1.15 1.15 1.15 1.15 1.15 1.25 1.25 1.25 1.25 1.25 1.25 1.35 1.35 1.35 1.35 1.35 1.35 1.45 1.45 1.45 1.45 1.45 1.45 1.55 1.55 1.55 1.55 1.55 1.55 1.65 1.65 1.65 1.65 1.65 1.65 1.75 1.75 1.75 1.75 1.75 1.75 1.85 1.85 1.85 1.85 1.85 1.85 1.95 1.95 1.95 1.95 1.95 1.95 END

TABLE Request

This TABLE request prints the values and a total for all six fields.

TABLE FILE FIVE PRINT PACK2 PACK1 DOUBLE1 FLOAT1 MATH1 XMATH1 ON TABLE SUMMARIZE ON TABLE SET PAGE NOLEAD END

The following report results:

PACK2 PACK1 DOUBLE1 FLOAT1 MATH1 XMATH1 ----- ----- ------- ------ ----- ------ 1.05 1.1 1.1 1.1 1.1 1.1 1.15 1.2 1.1 1.1 1.2 1.2 1.25 1.3 1.3 1.3 1.3 1.3 1.35 1.4 1.4 1.4 1.4 1.4 1.45 1.5 1.4 1.4 1.5 1.5 1.55 1.6 1.6 1.6 1.6 1.6 1.65 1.7 1.6 1.6 1.7 1.7 1.75 1.8 1.8 1.8 1.8 1.8 1.85 1.9 1.9 1.9 1.9 1.9 1.95 2.0 1.9 1.9 2.0 2.0 TOTAL 15.00 15.5 15.0 15.0 15.0 15.0

Note that for the PACK2 value 1.15, the single and double precision floating-point fields round to 1.1 because the value 1.15 could not be converted exactly to binary, so they were stored as slightly less than 1.15 (for example, 1.1499999) and rounded down instead of up. All of the single and double precision values, except 1.25 and 1.75, are stored as repeating decimals in hexadecimal.

The PACK2 values are not rounded. They are stored and displayed as they were entered.

Since the PACK1 values are rounded up before they are stored, the PACK1 total is 0.5 higher than the PACK2 total.

The D field total is the same as the PACK2 total because the D field values are stored as input, and then rounded for display.

WebFOCUS | |

Feedback |