RE: [xsl] Comparing numbers with different precisions

[Clint Redwood <clint@xxxxxxxxxxxxxxx>]

Hi David,

I'm comparing data loaded into a system with the values extracted from that
system. Not all the numbers are the same precision, but it would seem that the
system is rounding slightly less than XSL treats as double precision.

I'm already casting the two values to xs:double to carry out the comparison,
but that is still regarding the values as different. The comparision I'd like
to make ideally is "Is the value A the same as B if A is rounded to the same
precision that B is stated.". I've considered comparing as float, but I think
that may not really have sufficient precision.

I also note that the xpath function "round" doesn't have any precision
associated with it, so I can't just count the decimal places of the shorter
value and round the longer to that precision.

Yours,

Clint.

-----Original message-----
To:	xsl-list@xxxxxxxxxxxxxxxxxxxxxx;
CC:	Clint Redwood <clint@xxxxxxxxxxxxxxx>;
From:	David Carlisle <davidc@xxxxxxxxx>
Sent:	Thu 09-02-2012 09:40
Subject:	Re: [xsl] Comparing numbers with different precisions
> On 09/02/2012 09:24, Clint Redwood wrote:
> > Hi!
> >
> > I've a slightly awkward comparison, where I have two numbers of
> > differing precisions, and I want to regard them as the same if the
> > less preicse is the more precise rounded down to the same precision.
> >
> > Examples are: 17.166666666666668 ~= 17.1666666666667 and
> > 8.333333333333334 ~= 8.33333333333333
> >
> > Unfortunately, the precision is not always the same for all the
> > numbers I'm trying to compare.
> >
> > I just wondered if anyone on this list had done anything similar, and
> > had a good way of doing it, while I try and figure out a suitable way
> > to do it.
> >
> > Yours,
> >
> > Clint Redwood.
>
> How big are your precisions? in particular do you want xpath to treat
> then as doubles (which will mean that anything after 16 or so places
> will be lost anyway) and you can just use double precision numeric
> functions such as round to deal with the rounding, or do you need
> arbitrary precision decimals or (equivalently srings0 and expect to do
> teh rounding "by hand".
>
> David
>
>
>
> ________________________________________________________________________
> The Numerical Algorithms Group Ltd is a company registered in England
> and Wales with company number 1249803. The registered office is:
> Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.
>
> This e-mail has been scanned for all viruses by Star. The service is
> powered by MessageLabs.
> ________________________________________________________________________