Re: Division rounding error

No, you're not missing anything, Roger. My guess is that the compiler
went to floating point on you, so you're getting precision artifacts.
You might try this:

QtyPer = PZQTY / %int(10**PSDCC)

That will force the denominator to be an integer and may help you avoid
the floating point artifacts.



On 7/26/2018 10:06 AM, Rettig, Roger wrote:

In our old but currently being replace ERP system, some of the files

have a

quantity field and a separate field for the number of decimal places in

the

quantity field. So you could have a quantity of 12345 with the decimal
field of 2 would actually be 123.45.

In my attempt to not have a large Select statement, I code the

calculation

like this:

QtyPer = PZQTY /(10**PSDCC);

So if PZQTY = 12345 and PSDCC = 2 it would calculate to:

QtyPer = 12345/(10**2) = 12345/100 = 123.45

However I am getting a result of 123.499. Has anyone seen this before?

Am

I missing something?

Thanks.

--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L)
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our affiliate
link: http://amzn.to/2dEadiD