Re: Calculation gives wrong result

4/7 can't be represented exactly as a decimal fraction.  It creates a repeating series of numbers that goes on indefinitely....

Since computers need to set a limit to the size of a number, RPG will only allow a maximum of 63 digits, so the result won't be accurate.  It is similar to how 1/3 results in .333333 (repeating forever) which is always slightly smaller than 1/3.  Since it is slightly smaller than the actual 4/7, when you multiply it out, you end up with 3.9999, and since the result has no decimal positions, the .99999 is dropped

Doing the multiplication first solves the problem because 28/7 doesn't have the same problem.


On 8/29/22 3:57 PM, Justin Taylor wrote:

Both of these should give 4:
7 * (4 / 7)
7 * 4 / 7

RPG gives 3 and 4 as the result. I assume it has something to do with
intermediate result variables, but I can't see how. Can someone explain?

RPG sample:
Dcl-s result packed(7) ;

Dcl-s x packed(7) inz(7) ;
Dcl-s y uns(10) inz(4) ;
Dcl-s z packed(7) inz(7) ;

result = x * (y / z) ; // 3 but should be 4
result = x * y /z ; // 4

Return;


TIA