Re: The target for a numeric operation is too small to hold the resul t (C G D F)

[Ted J Barry <tjbarry@xxxxxxxxxxxxx>]

Hi All,

I have a similar situation where I have two different database fields. The 
result field is 10, 4 and the other is 11,4. Obviously too small, but what if 
you don't want to go around changing all of the database fields? Is there a 
routine anyone knows of to handle getting the result you want without having to 
change the db?

Thanks,

Ted

>>> boldt@xxxxxxxxxx 07/20/04 05:39AM >>>

Boer, Sacha wrote:

> Hi Tim, I'm not quite sure who posted the original question or whether it's 
> been answered yet, but please bear in mind that up to 5.1 you numeric limit 
> is 30,15 (from the top of my head) and as of 5.2 I believe they stretched it 
> to 60, something.

> Anyway, your division of 17,6 by 17,6 would need a theoretical resultfield of 
> 34,12. 

> OS400 will create a "workfield" for you when you code the entire calculation 
> into one statement. 34.12 exceeds the limits so here you'll get the error. 

> 

Sacha: You need to read Chapter 21 of the RPG IV Reference. 

Specifically, the section called "Precision of Intermediate Results". 

Actually, the precision differs a bit between releases, since the 

maximum size of a decimal value has changed from 30 (up to v5r1) to 31 

(in v5r2) and again to 63 (in v5r3). So the calculation is slightly 

different in each release.

Consider "(g_g_txamount-linetax)/linetax" where each variable is a 

(17,6) decimal. First, the subtraction yields an intermediate result 

with decimal precision (17,6). Next, applying the precision rules for 

division gives an intermediate with the following precisions:

<=V5R1 -> (30,12)

V5R2 -> (31,13)

V5R3 -> (63,45)

In each case, you get 18 digits to the left of the decimal point. If the 

result of the division has more than 18 digits left of the decimal 

point, you get an overflow exception. More likely, if you try to assign 

the value to a target variable with fewer than 19 digits to the left of 

the decimal (such as a 17,6 decimal), you get an overflow exception.

As someone else suggested, Tim needs to fix up the logic error in his 

program. Clearly, a computation is yielding a value that's too big for 

the intended target variable.

Cheers! Hans

--

This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing list

To post a message email: RPG400-L@xxxxxxxxxxxx 

To subscribe, unsubscribe, or change list options,

visit: http://lists.midrange.com/mailman/listinfo/rpg400-l 

or email: RPG400-L-request@xxxxxxxxxxxx 

Before posting, please take a moment to review the archives

at http://archive.midrange.com/rpg400-l.