Re: [xsl] slow xsltproc XInclude processing w/complex document?

Subject: Re: [xsl] slow xsltproc XInclude processing w/complex document?
From: Mike Trotman <mike.trotman@xxxxxxxxxxxxx>
Date: Tue, 06 Jul 2004 23:42:37 +0100

I've had what look like memory problems using xsltproc with the document function.

I've tried many variations (e.g. storing the document in a variable etc.) but all the symptoms seem to point to xsltproc keeping all loaded docuements in memory and not releasing memory when they go out of scope.

I'm not sure from Jeni's comment whether XSLT processors are supposed to hold all files in memory - or whether she is just emphasizing that they aren't on disc,

The details below are from a query I made on the Perl XML lists.
I was expecting the memory consumed by the document stored in variable 'tdoc' to be released after each 'item' had been processed
but I could watch the memory on my machine climb towards its limit of 512mB - and then totally lock up!

I had a simple XSLT script:

1 <xsl:stylesheet version="1.0" xmlns:xsl="";>
14 <xsl:output method="xml" encoding="UTF-8" omit-xml-declaration='yes' indent='yes'/>
16 <!-- ============================================================ -->
17 <xsl:template match='/LIST'>
18 <LIST>
19 <xsl:message>Start of List</xsl:message>
20 <xsl:apply-templates select='item'/>
21 </LIST>
22 </xsl:template>
23 <!-- ============================================================ -->
24 <xsl:template match='item'>
27 <xsl:variable name='tdoc' select='document(@href)'/>
28 <!--
29 -->
30 <ok>item <xsl:value-of select='position()'/></ok>
33 </xsl:template>
34 <!-- ============================================================ -->
35 </xsl:stylesheet>
which reads a simple XML file:
1 <LIST>
2 <item href="out/tm_1283901457.xml"/>
3 <item href="out/tm_2822501119.xml"/>
4 <item href="out/tm_2822601119.xml"/>
5 <item href="out/tm_2822701119.xml"/>
6 <item href="out/tm_2822801119.xml"/>
7 <item href="out/tm_2822901119.xml"/>

 5854 <item href="out/tm_3407901268.xml"/>
 5855 <item href="out/tm_3408001268.xml"/>
 5856 </LIST>

Each 'out/tm_...' document is about 10K in size.

If I run this stylesheet against the XML document :
In xsltproc
(and possibly xalan - haven't run many tests and have been trimming down the XSLT to try and find the minimum to produce the error)
it crashes my server - or just seems to hang forever.
In saxon, sablotron, msxsl it works as expected.
(Same behaviour under Windows XP and RedHat Linux 7.3.)

It works fine in all processors if the 'tdoc' variable definition is commented out.

Windows versions:

U:\jobs\ISIS\DataTranslator>xsltproc -V
Using libxml 20610, libxslt 10107 and libexslt 805
xsltproc was compiled against libxml 20610, libxslt
libxslt 10107 was compiled against libxml 20610
libexslt 805 was compiled against libxml 20610

ppm> query XML
Querying target 1 (ActivePerl
could not find ParserDetails.ini in D:/Perl/site/lib/XML/SAX
1. XML-LibXML [1.58] Interface to Gnome libxml2 xml parsing and DOM library
2. XML-LibXML-Common [0.13] Routines and Constants common for XML::LibXML and XML::GDOME
3. XML-LibXSLT [1.57] Interface to Gnome libxslt library
4. XML-NamespaceSupport [1.08] a simple generic namespace support class
5. XML-Parser [2.34] A Perl module for parsing XML documents
6. XML-SAX [0.12] Simple API for XML
7. XML-Simple [2.09] Easy API to read/write XML (esp config files)

Linux Versions
[miket@dl02 DataTranslator]$ xsltproc -V
Using libxml 20507, libxslt 10024 and libexslt 715
xsltproc was compiled against libxml 20501, libxslt 10024 and libexslt 715
libxslt 10024 was compiled against libxml 20501
libexslt 715 was compiled against libxml 20501
[miket@dl02 DataTranslator]$

cpan> m XML::LibXML Module id = XML::LibXML DESCRIPTION Interface to the libxml library CPAN_USERID PHISH (Christian Glahn <christian.glahn@xxxxxxxxxx>) CPAN_VERSION 1.58 CPAN_FILE P/PH/PHISH/XML-LibXML-1.58.tar.gz DSLI_STATUS RmhO (released,mailing-list,hybrid,object-oriented) MANPAGE XML::LibXML - Perl Binding for libxml2 INST_FILE /usr/lib/perl5/site_perl/5.6.1/i386-linux/XML/ INST_VERSION 1.58

cpan> m XML::LibXSLT Module id = XML::LibXSLT CPAN_USERID MSERGEANT (Matt Sergeant <matt@xxxxxxxxxxxx>) CPAN_VERSION 1.57 CPAN_FILE M/MS/MSERGEANT/XML-LibXSLT-1.57.tar.gz MANPAGE XML::LibXSLT - Interface to the gnome libxslt library INST_FILE /usr/lib/perl5/site_perl/5.6.1/i386-linux/XML/ INST_VERSION 1.57

Jeni Tennison wrote:

Hi Paul,

I've been running some tests on a document that includes nested
Xinclude directives. The document is complex: upwards of 1500 files,
nested to a depth of up to 4 levels. Total size of content is about

As you probably know, XSLT holds the files that it loads in memory. You haven't said how much memory your machines have, but it might simply be that holding all those files (and the flattened result document) in memory is causing problems.

Another thing is that you're traversing every single node in those
documents. Every node visit takes time, because the processor has to
work out what to do with the node, so cutting down the node visits
would be good. You could try, for example, changing the identity
template that you're using at the moment for:

<xsl:template match="node()">
 <xsl:copy-of select="." />

<xsl:template match="*[.//xi:include]">
   <xsl:copy-of select="@*" />
   <xsl:apply-templates />

Overall, though, if speed is an issue, you would be much better off
using a SAX Filter to do the transformation: that way you wouldn't be
storing the documents in memory, each node would have to be visited
only once, and the output can stream out (if that helps). Mind you,
you say that the XInclude resolution is only a test, so perhaps you
can't do your real transformation using SAX...



Jeni Tennison

XSL-List info and archive:
To unsubscribe, go to:
or e-mail: <mailto:xsl-list-unsubscribe@xxxxxxxxxxxxxxxxxxxxxx>

-- Datalucid Limited 8 Eileen Road South Norwood London SE25 5EJ United Kingdom

tel :0208-239-6810
mob: 0794-725-9760
email: mike.trotman@xxxxxxxxxxxxx

UK Co. Reg:   4383635
VAT Reg.:   798 7531 60


Current Thread