Re: Strange Result -- RPG400-L

× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.

John,
The ** (exponentiation) operator outputs data type 8F (8 byte aka 64-bit aka "double precision" floating point), so unfortunately it will lose precision just as floating point numbers do. This is explained here in the ILE RPG reference manual:
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzasd/prectdf.htm

Sorry to be the bearer of bad news :-( I was quite unhappy when I first learned of this years ago.

-SK

On 10/21/2020 9:58 AM, john erps wrote:

Hi,

I have the following program.

ctl-opt actgrp(*new);

dcl-s bignum1 packed(30);
dcl-s bignum2 packed(30:10);

*inLR = *on;

bignum1 = 777777777777777777777777777777;
bignum2 = bignum1 / 10 ** 10;

dsply (%char(bignum2));

Return;

Could anybody explain the following result?

DSPLY 77777777777777786880.0000000000

Apparently due to intermediate precision some digits are lost.
But i thought that the default behavior would be keeping as much digits as
possible.

When i change the division to bignum1 / 10000000000 i get the correct
result.

We're on 7.3.

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.