Subject: Re: [xsl] use XSLT or XQuery in Saxon? From: Niclas Hedhman <niclas@xxxxxxxxxxx> Date: Thu, 6 Jan 2005 18:21:10 +0800 |
On Thursday 06 January 2005 06:53, alan m wrote: > I have extremely large (over 300 MB) XML file and tens > of thousands of small xml files generated after > applying various XSLT on the one big XML file. I don't know whether Mr Kay have tested Saxon with 100+MB files or not, but we did (6.5.?), and could not get a simple transform to complete within hours (I think we gave up after ~4hours on a 80-100MB file), on a machine with 1GB of RAM. I wrote a custom transformer in Java doing exactly what we needed using; * SAX events * Only keeping one branch/leaf of the XML tree in memory at any time. * Aggregation of content into small mutable value objects, which were output and discarded when completed. 1500 files, varying from 360MB to ~10MB of a total of ~10GB could be processed in a linear speed of ~2MB per second, or close to the disk drive speed, on a dual CPU workstation. I suspect that you will end up in 'custom transformer' territory, but perhaps Saxon has improved and can deal with the transforms you give it. I suggest that you make some simple tests first, which somewhat ressemble what you need to do later. Cheers Niclas -- --------------- If at first you don't succeed, destroy all evidence that you tried. - Steven Wright +---------//-------------------+ | http://www.dpml.net | | http://niclas.hedhman.org | +------//----------------------+
Current Thread |
---|
|
<- Previous | Index | Next -> |
---|---|---|
RE: [xsl] use XSLT or XQuery in Sax, Michael Kay | Thread | Re: [xsl] use XSLT or XQuery in Sax, Niclas Hedhman |
RE: [xsl] XML/XSLT formatting probl, Chris Hicks | Date | Re: [xsl] use XSLT or XQuery in Sax, Niclas Hedhman |
Month |