× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.


  1.  Home
  2. RPG400-L
  3. July 2018

Re: Division rounding error

On 2018-07-26 12:33 PM, Joe Pluta wrote:
No, you're not missing anything, Roger.  My guess is that the compiler went to floating point on you, so you're getting precision artifacts. You might try this:

QtyPer = PZQTY / %int(10**PSDCC)

That will force the denominator to be an integer and may help you avoid the floating point artifacts.

Better to use %intH, to force it to round up if the floating point result of 10**PSDCC is a little bit below the decimal value of 10**PSDCC.

%INT(99.999999) = 99. %INTH(99.999999) = 100.

QtyPer = PZQTY / %inth(10**PSDCC)



As an Amazon Associate we earn from qualifying purchases.

This thread ...
Replies:
Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.