RE: [xsl] Floating point precision error

["Michael Kay" <mike@xxxxxxxxxxxx>]

I agree, this result is way out. It looks to me suspiciously as if it's
carried out using single-precision rather than double-precision floating
point.

Saxon (both 6.5.5 and 8.6.1) give

<?xml version="1.0" encoding="utf-8"?>6.4_
  3000_
  19200_  

dom4j doesn't include an XSLT processor of its own, as far as I know. Use
xsl:system-property('xsl:vendor') to find out which processor you are using.

I changed the code to explicitly use single-precision float and the result
was

<?xml version="1.0" encoding="UTF-8"?>6.4_
  3000_
  19200.000286102295_  

which is still closer than the result you are seeing.

Michael Kay
http://www.saxonica.com/

 

> -----Original Message-----
> From: Thang Le [mailto:tle@xxxxxxxxxxx] 
> Sent: 26 January 2006 01:16
> To: xsl-list@xxxxxxxxxxxxxxxxxxxxxx
> Subject: [xsl] Floating point precision error
> 
> This is something I don't understand. Given this XSL snippet:
> 
> <xsl:value-of select="$nodes[1]/unitamount" />_
> <xsl:value-of select="$nodes[1]/quantity" />_
> <xsl:value-of select="($nodes[1]/quantity * $nodes[1]/unitamount)" />_
> 
> I'm getting the printout as:
> 6.4_
> 3000_
> 19199.908447265625_ 
> 
> The literal values in XML is 6.4 and 3000. If I round the result to 2 
> decimal places, I still get 19199.91, a far cry from what it 
> should be 
> (19200).
> 
> I'm using dom4j 1.6, so I guess that's XSLT 1.0? Also this 
> only happens 
> on SunOS and not on Windows. Still, the error is huge and obvious. 
> Isn't there a way to force the calculation to be more exact?
> 
> Thanks,
> 
> Thang Le
> 
> -- 
> NeoMail - http://neomail.sourceforge.net