Re: Unexpected division quotient

hi Kurt,

That depends on the possible values of ds_Inp.RevDec. When you use the ** (exponential) operator, it will do it's math using double-precision floating point. (As Chuck said earlier...) Trouble is, double-precision floating point only stores about 14-15 digits of precision... (I don't remember the exact number.)

As soon as you exceed 14-15 digits, the output of 10 ** ds_Inp.RevDec will be wrong... and therefore, it'll still be wrong after converting it to integer.

-SK


On 7/2/2012 4:20 PM, Anderson, Kurt wrote:

Thanks Chuck, I wouldn't have thought of checking the Visual Age RPG documentation (j/k).

Thanks for pointing that out, though.

For my own sanity, based on that, can I trust that both of the following will always work?
C = a * (10 ** (ds_Inp.RevDec * -1));
C = a / %int(10 ** ds_Inp.RevDec);

ds_Inp.RevDec is 4,0 zoned (which is why I used %int instead of %dec).

-Kurt

So is it simply luck that using the exponential operator on a negative value is giving me the value I want?

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of CRPence
Sent: Monday, July 02, 2012 4:07 PM
To: rpg400-l@xxxxxxxxxxxx
Subject: Re: Unexpected division quotient

Searching the v5r4 InfoCenter on the following tokens, yields the quoted text that follows:
exponentiation rpg

"VisualAge RPG Language Reference

Precision of Intermediate Results

Table 52 describes the default precision rules in more detail.

Table 52. Precision of Intermediate Results
Operation Result Precision

Note:
The following operations produce a numeric result. Ln is the length of the operand in digits where n is either r for result or a numeric representing the operand. Dn is the number of digits to the right of the decimal point where n is either r for result or a numeric representing the operand. T is the temporary value.

Note that if any operand has a floating point representation (for example, it is *the result of the exponentiation operator* ), the result also is a floating point value and the precision rules no longer apply.
A floating point value has the precision available from double precision floating point representation.

...
"

Regards, Chuck

On 02 Jul 2012 15:45, Anderson, Kurt wrote:

IBM i 7.1

There is a situation where we have an integer sent to us carrying an
amount value. Another field contains the number of implied decimals in
that integer.

Here is what I'm seeing:

dividend = 418
divisor = 10 to the power of 4 (implied decimal value) = 10000.
quotient = .041799 (but I'm expecting .041800)

C = a / (10 ** ds_Inp.RevDec);
// C = .041799


I have tried with and without the H-spec ExPrOpts( *ResDecPos ).

When I break down the code into two lines, it does work. And 2 lines
is fine, b/c this is still better than the 19 lines of code we used to
have, but I was would like to understand why this is happening.

Example of how to get .041800.
B = (10 ** ds_Inp.RevDec);
C = a / b;
// C = .041800

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at http://archive.midrange.com/rpg400-l.