RE: Unexpected division quotient

Also, keep in mind that the max value for an integer (10I 0) is
2147483647
and an unsigned int (10U 0) is 4294967295

So using an integer (10i 0) will only allow you up to 10**9 . Even an
unsigned int will only get up to 10**9.

Having RevDec as 2,0 will overflow an integer if the value is ever over
9.

If you use an integer intermediate field you may want to set a 20i 0 (8
byte integer) which has a max value of 9223372036854775807. Which will
get you up to 10**18.

The %INT biff should automatically use an appropriate sized intermediate
field when used in an eval statement.
The size of the integer is only important if you are using a work field
that you are defining as 10i 0 for the purpose of containing the
"Divisor".


Chris Hiebert
-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx
[mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Anderson, Kurt
Sent: Monday, July 02, 2012 3:44 PM
To: RPG programming on the IBM i / System i
Subject: RE: Unexpected division quotient

Scott,

I should be safe because the greatest value we receive is 4 currently
(and that's been true for 10 years), and at most we currently store up
to 6 decimals, although to be safe (since RevDec is 2,0) I could check
for a value of 13 and error it. Or I could change it to a loop, which
would still be better than having to check for each possible value of
RevDec (which is what it currently does today).

Divisor = 1;
For i = 1 to ds_Inp.RevDec;
Divisor *= 10;
EndFor;

C = a / divisor;

I wanted to see if it was 14 or 15, though it looks like the 7.1 IBM
documentation isn't going to give an exact number.
"Double-precision floating-point is generally accurate to 15 digits of
precision."
http://publib.boulder.ibm.com/infocenter/iseries/v7r1m0/index.jsp?topic=
%2Fdb2%2Frbafzch2num.htm

Thanks for the feedback,
Kurt

-----Original Message-----
From: Scott Klement [mailto:rpg400-l@xxxxxxxxxxxxxxxx]
Sent: Monday, July 02, 2012 4:31 PM
To: RPG programming on the IBM i / System i
Cc: Anderson, Kurt
Subject: Re: Unexpected division quotient

hi Kurt,

That depends on the possible values of ds_Inp.RevDec. When you use the
** (exponential) operator, it will do it's math using double-precision
floating point. (As Chuck said earlier...) Trouble is,
double-precision floating point only stores about 14-15 digits of
precision... (I don't remember the exact number.)

As soon as you exceed 14-15 digits, the output of 10 ** ds_Inp.RevDec
will be wrong... and therefore, it'll still be wrong after converting
it to integer.

-SK


On 7/2/2012 4:20 PM, Anderson, Kurt wrote:

Thanks Chuck, I wouldn't have thought of checking the Visual Age RPG

documentation (j/k).

Thanks for pointing that out, though.

For my own sanity, based on that, can I trust that both of the

following will always work?

C = a * (10 ** (ds_Inp.RevDec * -1));
C = a / %int(10 ** ds_Inp.RevDec);

ds_Inp.RevDec is 4,0 zoned (which is why I used %int instead of %dec).

-Kurt

So is it simply luck that using the exponential operator on a negative

value is giving me the value I want?

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx
[mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of CRPence
Sent: Monday, July 02, 2012 4:07 PM
To: rpg400-l@xxxxxxxxxxxx
Subject: Re: Unexpected division quotient

Searching the v5r4 InfoCenter on the following tokens, yields the

quoted text that follows:

exponentiation rpg

"VisualAge RPG Language Reference

Precision of Intermediate Results

Table 52 describes the default precision rules in more detail.

Table 52. Precision of Intermediate Results
Operation Result Precision

Note:
The following operations produce a numeric result. Ln is the

length of the operand in digits where n is either r for result or a
numeric representing the operand. Dn is the number of digits to the
right of the decimal point where n is either r for result or a numeric
representing the operand. T is the temporary value.

Note that if any operand has a floating point representation

(for example, it is *the result of the exponentiation operator* ), the
result also is a floating point value and the precision rules no longer
apply.

A floating point value has the precision available from double

precision floating point representation.

...
"

Regards, Chuck

On 02 Jul 2012 15:45, Anderson, Kurt wrote:

IBM i 7.1

There is a situation where we have an integer sent to us carrying an
amount value. Another field contains the number of implied decimals
in that integer.

Here is what I'm seeing:

dividend = 418
divisor = 10 to the power of 4 (implied decimal value) = 10000.
quotient = .041799 (but I'm expecting .041800)

C = a / (10 ** ds_Inp.RevDec);
// C = .041799


I have tried with and without the H-spec ExPrOpts( *ResDecPos ).

When I break down the code into two lines, it does work. And 2 lines
is fine, b/c this is still better than the 19 lines of code we used
to have, but I was would like to understand why this is happening.

Example of how to get .041800.
B = (10 ** ds_Inp.RevDec);
C = a / b;
// C = .041800

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing

list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at http://archive.midrange.com/rpg400-l.

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing
list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at
http://archive.midrange.com/rpg400-l.