RE: Unexpected division quotient

Scott,

I should be safe because the greatest value we receive is 4 currently (and that's been true for 10 years), and at most we currently store up to 6 decimals, although to be safe (since RevDec is 2,0) I could check for a value of 13 and error it. Or I could change it to a loop, which would still be better than having to check for each possible value of RevDec (which is what it currently does today).

Divisor = 1;
For i = 1 to ds_Inp.RevDec;
Divisor *= 10;
EndFor;

C = a / divisor;

I wanted to see if it was 14 or 15, though it looks like the 7.1 IBM documentation isn't going to give an exact number.
"Double-precision floating-point is generally accurate to 15 digits of precision."
http://publib.boulder.ibm.com/infocenter/iseries/v7r1m0/index.jsp?topic=%2Fdb2%2Frbafzch2num.htm

Thanks for the feedback,
Kurt

-----Original Message-----
From: Scott Klement [mailto:rpg400-l@xxxxxxxxxxxxxxxx]
Sent: Monday, July 02, 2012 4:31 PM
To: RPG programming on the IBM i / System i
Cc: Anderson, Kurt
Subject: Re: Unexpected division quotient

hi Kurt,

That depends on the possible values of ds_Inp.RevDec. When you use the
** (exponential) operator, it will do it's math using double-precision
floating point. (As Chuck said earlier...) Trouble is,
double-precision floating point only stores about 14-15 digits of precision... (I don't remember the exact number.)

As soon as you exceed 14-15 digits, the output of 10 ** ds_Inp.RevDec
will be wrong... and therefore, it'll still be wrong after converting
it to integer.

-SK


On 7/2/2012 4:20 PM, Anderson, Kurt wrote:

Thanks Chuck, I wouldn't have thought of checking the Visual Age RPG documentation (j/k).

Thanks for pointing that out, though.

For my own sanity, based on that, can I trust that both of the following will always work?
C = a * (10 ** (ds_Inp.RevDec * -1));
C = a / %int(10 ** ds_Inp.RevDec);

ds_Inp.RevDec is 4,0 zoned (which is why I used %int instead of %dec).

-Kurt

So is it simply luck that using the exponential operator on a negative value is giving me the value I want?

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx
[mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of CRPence
Sent: Monday, July 02, 2012 4:07 PM
To: rpg400-l@xxxxxxxxxxxx
Subject: Re: Unexpected division quotient

Searching the v5r4 InfoCenter on the following tokens, yields the quoted text that follows:
exponentiation rpg

"VisualAge RPG Language Reference

Precision of Intermediate Results

Table 52 describes the default precision rules in more detail.

Table 52. Precision of Intermediate Results
Operation Result Precision

Note:
The following operations produce a numeric result. Ln is the length of the operand in digits where n is either r for result or a numeric representing the operand. Dn is the number of digits to the right of the decimal point where n is either r for result or a numeric representing the operand. T is the temporary value.

Note that if any operand has a floating point representation (for example, it is *the result of the exponentiation operator* ), the result also is a floating point value and the precision rules no longer apply.
A floating point value has the precision available from double precision floating point representation.

...
"

Regards, Chuck

On 02 Jul 2012 15:45, Anderson, Kurt wrote:

IBM i 7.1

There is a situation where we have an integer sent to us carrying an
amount value. Another field contains the number of implied decimals
in that integer.

Here is what I'm seeing:

dividend = 418
divisor = 10 to the power of 4 (implied decimal value) = 10000.
quotient = .041799 (but I'm expecting .041800)

C = a / (10 ** ds_Inp.RevDec);
// C = .041799


I have tried with and without the H-spec ExPrOpts( *ResDecPos ).

When I break down the code into two lines, it does work. And 2 lines
is fine, b/c this is still better than the 19 lines of code we used
to have, but I was would like to understand why this is happening.

Example of how to get .041800.
B = (10 ** ds_Inp.RevDec);
C = a / b;
// C = .041800

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing
list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at http://archive.midrange.com/rpg400-l.