Re: Division rounding error

On 7/26/2018 11:06 AM, Rettig, Roger wrote:

In our old but currently being replace ERP system, some of the files have a
quantity field and a separate field for the number of decimal places in the
quantity field. So you could have a quantity of 12345 with the decimal
field of 2 would actually be 123.45.

In my attempt to not have a large Select statement, I code the calculation
like this:

QtyPer = PZQTY /(10**PSDCC);

So if PZQTY = 12345 and PSDCC = 2 it would calculate to:

QtyPer = 12345/(10**2) = 12345/100 = 123.45

However I am getting a result of 123.499. Has anyone seen this before? Am
I missing something?

In classic, RPG III fixed format style, calculations are done one at a time:

* EXPR is a pretend opcode, used as shorthand
10 EXPR PSDCC RESULT1
PZQTY DIV RESULT1 QTYPER

The length and precision of RESULT1 and QTYPER were set by the programmer.

When using an expression, like QtyPer = PZQTY /(10**PSDCC); the length
and precision of the intermediate result (RESULT1) are set by the
compiler. The compiler has rules for how it creates these intermediate
values.

I start with Expressions:
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzasd/expr9.htm#expr9

Then, Presicion Rules for Numeric Operations:
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzasd/prcsnr.htm

There are sections in there that help me drive the intermediate values
the way I want them to go. Generally speaking, the compiler wants to
preserve as many significant digits as possible.