× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.


  1.  Home
  2. RPG400-L
  3. July 2018

Re: Division rounding error

No, you're not missing anything, Roger.  My guess is that the compiler went to floating point on you, so you're getting precision artifacts.  You might try this:

QtyPer = PZQTY / %int(10**PSDCC)

That will force the denominator to be an integer and may help you avoid the floating point artifacts.



On 7/26/2018 10:06 AM, Rettig, Roger wrote:
In our old but currently being replace ERP system, some of the files have a
quantity field and a separate field for the number of decimal places in the
quantity field. So you could have a quantity of 12345 with the decimal
field of 2 would actually be 123.45.

In my attempt to not have a large Select statement, I code the calculation
like this:

QtyPer = PZQTY /(10**PSDCC);

So if PZQTY = 12345 and PSDCC = 2 it would calculate to:

QtyPer = 12345/(10**2) = 12345/100 = 123.45

However I am getting a result of 123.499. Has anyone seen this before? Am
I missing something?

Thanks.




As an Amazon Associate we earn from qualifying purchases.

This thread ...
Follow-Ups:
Replies:
Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.