× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.


  1.  Home
  2. MIDRANGE-L
  3. February 2016

Re: SQL precision problem

On Fri, Feb 12, 2016 at 11:13 AM, John Yeung <gallium.arsenide@xxxxxxxxx>
wrote:

(p,s)
(A-B) --> (27,4)
(A-B) / B --> (31,0)

Hm. Where did the (27, 4) come from? Was that meant to be (31, 4)?


​From the SQL Manual...assuming I did the math right...at it looks like I
didn't :)
The scale of the result of addition and subtraction is max (s,s'). The
precision is
min(mp,max(p-s,p'-s') +max(s,s')+1).

scale max(4,4) = 4
precision ​min(mp,max(p-s,p'-s') +max(s,s')+1)
== min(31, max(31-4,12-4) + max(4,4) + 1)
== min(31, max(27,8) + 4 + 1
== min(31, 27 + 5)
== min(31, 32)
== 31

Missed the +5 in my OP.

Charles

As an Amazon Associate we earn from qualifying purchases.

This thread ...
Replies:
Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.