Re: Strange Result

[Scott Klement <rpg400-l@xxxxxxxxxxxxxxxx>]

John,
The ** (exponentiation) operator outputs data type 8F (8 byte aka 64-bit 
aka "double precision" floating point), so unfortunately it will lose 
precision just as floating point numbers do.  This is explained here in 
the ILE RPG reference manual:
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzasd/prectdf.htm

Sorry to be the bearer of bad news :-(   I was quite unhappy when I 
first learned of this years ago.

-SK


On 10/21/2020 9:58 AM, john erps wrote:
> Hi,
>
> I have the following program.
>
> ctl-opt actgrp(*new);
>
> dcl-s bignum1 packed(30);
> dcl-s bignum2 packed(30:10);
>
> *inLR = *on;
>
> bignum1 = 777777777777777777777777777777;
> bignum2 = bignum1 / 10 ** 10;
>
> dsply (%char(bignum2));
>
> Return;
>
> Could anybody explain the following result?
>
> DSPLY  77777777777777786880.0000000000
>
> Apparently due to intermediate precision some digits are lost.
> But i thought that the default behavior would be keeping as much digits as
> possible.
>
> When i change the division to bignum1 / 10000000000 i get the correct
> result.
>
> We're on 7.3.