Subject: RE: [xsl] Floating point precision error From: "Michael Kay" <mike@xxxxxxxxxxxx> Date: Thu, 26 Jan 2006 08:49:23 -0000 |
I agree, this result is way out. It looks to me suspiciously as if it's carried out using single-precision rather than double-precision floating point. Saxon (both 6.5.5 and 8.6.1) give <?xml version="1.0" encoding="utf-8"?>6.4_ 3000_ 19200_ dom4j doesn't include an XSLT processor of its own, as far as I know. Use xsl:system-property('xsl:vendor') to find out which processor you are using. I changed the code to explicitly use single-precision float and the result was <?xml version="1.0" encoding="UTF-8"?>6.4_ 3000_ 19200.000286102295_ which is still closer than the result you are seeing. Michael Kay http://www.saxonica.com/ > -----Original Message----- > From: Thang Le [mailto:tle@xxxxxxxxxxx] > Sent: 26 January 2006 01:16 > To: xsl-list@xxxxxxxxxxxxxxxxxxxxxx > Subject: [xsl] Floating point precision error > > This is something I don't understand. Given this XSL snippet: > > <xsl:value-of select="$nodes[1]/unitamount" />_ > <xsl:value-of select="$nodes[1]/quantity" />_ > <xsl:value-of select="($nodes[1]/quantity * $nodes[1]/unitamount)" />_ > > I'm getting the printout as: > 6.4_ > 3000_ > 19199.908447265625_ > > The literal values in XML is 6.4 and 3000. If I round the result to 2 > decimal places, I still get 19199.91, a far cry from what it > should be > (19200). > > I'm using dom4j 1.6, so I guess that's XSLT 1.0? Also this > only happens > on SunOS and not on Windows. Still, the error is huge and obvious. > Isn't there a way to force the calculation to be more exact? > > Thanks, > > Thang Le > > -- > NeoMail - http://neomail.sourceforge.net
Current Thread |
---|
|
<- Previous | Index | Next -> |
---|---|---|
[xsl] Floating point precision erro, Thang Le | Thread | RE: [xsl] Floating point precision , Thang X. Le |
[xsl] The f:sqrt() and stdDev() fun, Dimitre Novatchev | Date | [xsl] Ken Clark is out of the offic, Clark |
Month |