Re: Division rounding error -- RPG400-L

Good News Everybody!

The new search engine is LIVE!

Please report any problems to david (at) midrange.com.

Subject: Re: Division rounding error
From: Barbara Morris <bmorris@xxxxxxxxxx>
Date: Thu, 26 Jul 2018 16:25:12 -0400
List-archive: <https://archive.midrange.com/rpg400-l/>
List-help: <mailto:rpg400-l-request@midrange.com?subject=help>
List-id: "RPG programming on the IBM i \(AS/400 and iSeries\)" <rpg400-l.midrange.com>
List-post: <mailto:rpg400-l@midrange.com>
List-subscribe: <https://lists.midrange.com/mailman/listinfo/rpg400-l>, <mailto:rpg400-l-request@midrange.com?subject=subscribe>
List-unsubscribe: <https://lists.midrange.com/mailman/options/rpg400-l>, <mailto:rpg400-l-request@midrange.com?subject=unsubscribe>

On 2018-07-26 12:33 PM, Joe Pluta wrote:

No, you're not missing anything, Roger. My guess is that the compiler went to floating point on you, so you're getting precision artifacts. You might try this:

QtyPer = PZQTY / %int(10**PSDCC)

That will force the denominator to be an integer and may help you avoid the floating point artifacts.

Better to use %intH, to force it to round up if the floating point result of 10**PSDCC is a little bit below the decimal value of 10**PSDCC.

%INT(99.999999) = 99. %INTH(99.999999) = 100.

QtyPer = PZQTY / %inth(10**PSDCC)

This mailing list archive is Copyright 1997-2026 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.