RE: Unexpected division quotient

Booth,

That works in one line. It seems like an unnecessary extra step, but it works:
C = a * (10 ** (ds_Inp.RevDec * -1));

Is there a better way than multiplying by -1 on the 'to the power of' portion?

-Kurt

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Booth Martin
Sent: Monday, July 02, 2012 3:58 PM
To: RPG programming on the IBM i / System i
Subject: Re: Unexpected division quotient

FWIW:
I was taught to use multiplication wherever possible instead of division.

What happens if you multiply instead of divide?

418 X .00010 = .0418

Is that a possible solution for the problem?



On 7/2/2012 3:45 PM, Anderson, Kurt wrote:

IBM i 7.1

There is a situation where we have an integer sent to us carrying an amount value. Another field contains the number of implied decimals in that integer.

Here is what I'm seeing:

dividend = 418
divisor = 10 to the power of 4 (implied decimal value) = 10000.
quotient = .041799 (but I'm expecting .041800)

C = a / (10 ** ds_Inp.RevDec);
// C = .041799


I have tried with and without the H-spec ExPrOpts( *ResDecPos ).

When I break down the code into two lines, it does work. And 2 lines is fine, b/c this is still better than the 19 lines of code we used to have, but I was would like to understand why this is happening.

Example of how to get .041800.
B = (10 ** ds_Inp.RevDec);
C = a / b;
// C = .041800

Thanks,
Kurt Anderson
Sr. Programmer/Analyst
CustomCall Data Systems, a division of Enghouse Ltd.

--
Booth Martin
802-461-5349
http://www.martinvt.com

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at http://archive.midrange.com/rpg400-l.