× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.



So what you're saying is that truncation is mandatory, so DB2 is doing high order truncation instead of low order.

I went back to where I'm originally calculating A, and forced it to use a shorter length. That worked.


Thanks




-----Original Message-----
From: Birgitta Hauser [mailto:Hauser@xxxxxxxxxxxxxxx]
Sent: Friday, February 12, 2016 12:09 AM
To: 'Midrange Systems Technical Discussion' <midrange-l@xxxxxxxxxxxx>
Subject: AW: SQL precision problem

The problem occurs because the calculation is performed under the cover as float.
One rule when working with float is, if the (calculated) maximum length is reached, decimal positions are cut at the end.

I assume the length for your numeric values in your job is 31 (can be changed up to 63 in STRSQL and IBM i Navigator's or ACS's RUN AN SQL SCRIPT)

If you change the cast function to a precicion of 24 or , it calculates correctly.
Select x.*,
(A - B) / B * 100.00 as RESULT
from (Values(cast(1.5267 as decimal(24, 4)), cast(1,6 as decimal(12, 4)))) x (a, b);;

If you change the maximum precision in your job/program from 31 to 63 it also return the expected result.

Mit freundlichen Grüßen / Best regards

Birgitta Hauser

"Shoot for the moon, even if you miss, you'll land among the stars." (Les
Brown)
"If you think education is expensive, try ignorance." (Derek Bok) "What is worse than training your staff and losing them? Not training them and keeping them!"

-----Ursprüngliche Nachricht-----
Von: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] Im Auftrag von Justin Taylor
Gesendet: Thursday, 11.2 2016 22:50
An: MIDRANGE-L (midrange-l@xxxxxxxxxxxx)
Betreff: SQL precision problem

I'm trying to figure out how SQL arithmetic involving decimals is giving an integer result. I've put together a sample query that replicates my question.

with Values as (
select cast(1.5267 as decimal(31,4)) as A, cast(1.6 as decimal(12,4)) as B
from SYSIBM.SYSDUMMY1
)
select Values.*, (A - B) / B * 100.00 as RESULT
from Values;

This gives: 1.5267 1.6 0


The RESULT column has a type of decimal(31,2), but the value is zero. With a precision of 2, RESULT should actually be -4.58.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.



As an Amazon Associate we earn from qualifying purchases.

This thread ...

Follow-Ups:
Replies:

Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.