Re: Unexpected division quotient

[Alan Campin <alan0307d@xxxxxxxxx>]

You didn't say how you defined B.

On Mon, Jul 2, 2012 at 2:45 PM, Anderson, Kurt <KAnderson@xxxxxxxxxxxx> wrote:
> IBM i 7.1
>
> There is a situation where we have an integer sent to us carrying an amount value.  Another field contains the number of implied decimals in that integer.
>
> Here is what I'm seeing:
>
> dividend = 418
> divisor = 10 to the power of 4 (implied decimal value) = 10000.
> quotient = .041799  (but I'm expecting .041800)
>
> C = a / (10 ** ds_Inp.RevDec);
> // C = .041799
>
>
> I have tried with and without the H-spec ExPrOpts( *ResDecPos ).
>
> When I break down the code into two lines, it does work.  And 2 lines is fine, b/c this is still better than the 19 lines of code we used to have, but I was would like to understand why this is happening.
>
> Example of how to get .041800.
> B = (10 ** ds_Inp.RevDec);
> C = a / b;
> // C = .041800
>
> Thanks,
> Kurt Anderson
> Sr. Programmer/Analyst
> CustomCall Data Systems, a division of Enghouse Ltd.
> --
> This is the RPG programming on the IBM i / System i (RPG400-L) mailing list
> To post a message email: RPG400-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
> or email: RPG400-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/rpg400-l.