Re: [xsl] Notes on Comparison of XSL FO Renderers

Subject: Re: [xsl] Notes on Comparison of XSL FO Renderers
From: "W. Eliot Kimber" <eliot@xxxxxxxxxx>
Date: Sat, 18 Oct 2003 09:38:50 -0500
Innodata Isogen has an inherent interest in knowing what the performance characteristics of the various FO processors are. I already make a point of tracking the feature and extension differences among the various FO implementations, so this would be a natural extension of that activity.

Given this interest, I would be happy to act as an informal performance testing service for FO implementations.

If FO implementors will provide me with the items listed below, I will perform what I think should be fair and representative performance tests and publish the results. If I can get this information in the next week, I should be able to publish results in time for the XML 2003 conference (my fall is going to be very busy with client work, but I can carve out some time for this testing).

What I need for a given FO implementation is:

1. A copy of the latest generally-available FO implementation (I already have XEP and XSL Formatter). If the evaluation license does not allow processing of arbitrarily-sized documents, I will need a full license as well (I promise I won't use any such licenses for production work--Innodata Isogen already owns licenses for FO implementations that we use for our own business purposes).

2. Specification of how to configure the software and operating environment for maximum performance: memory settings, OS options, hardware settings, etc.

3. An explanation of any startup time costs that should be factored out or noted in the test results, e.g. JVM load, first-time startup, etc.

4. Specification of the operating framework within which the implementation should be tested: from Ant, integrated with an Apache server, a Tomcat server, IIS, or from the command line). If possible, please provide any driving wrappers, ant tasks, etc., so I don't have to write them myself.

5. Specification of the XSLT processor to use and how to use it for maximum performance. This is based on the presumption that some FO implementations may include optimized integrations with specific XSLT engines.

For my purposes, my customers would probably never use a system that did not include XSLT extensions, especially custom collators. Currently only Saxon provides facilities for integrating custom collators, so for my purposes, the only useful comparison would be one in which saxon 6.5.3 is the XSLT processor in all cases.

Thus, to be fair, I think that any FO implementations that can use an embedded XSLT engine that is not Saxon should be tested both with and without.

6. How to generate PDF:
   - Via Acrobat Distiller
   - Via direct PDF generation

For direct generation, whether or not it is possible to produce vector graphics w/in the resulting PDF (i.e., by including vector EPS graphics or via SVG).

This is very important because PDF intended to be used for commercial printing of documents should use vector graphics in the PDF in order to ensure optimal print quality for line drawings. PDF for Web delivery may also need vector graphics where the graphics are quite commplex, such as complex parts diagrams, circuit diagrams, and so on. Rasterized line drawings are usually not acceptable in these cases.

By "vector EPS" graphics, I mean EPS graphics that are PostScript drawing commands, as opposed to EPS graphics that are bitmaps. EPS graphics may include a "preview" image--some FO processors can only include the preview bitmap when generating PDF directly, for the simple reason that their tool does not include a PostScript interpreter (which would be a silly thing to do given how inexpensive Acrobat Distiller is--for most people it's free because it's included with other products or their enterprise has a site license for full Acrobat). However, it is possible to generate vector PDF graphics from SVG using tools like the FOP SVG "rasterizer" [it's not actually a rasterizer since its output is not a raster graphic, but I quibble] and the latest versions of both XEP and XSL Formatter provide this feature. But if your graphics are in EPS format, your only real choice for generating PDFs with vector graphics is to print to PostScript and then distill it--this can incur a significant time cost over the direct PDF generation process.

7. A set of test cases (XML documents, graphics [including vector EPS or SVG graphics, as relevant], XSLT transforms) that your FO implementation can process. These should be realistic and include both small and some very large documents so I can graph representative performance trends based on document length and complexity.

My testing method would be basically:

- Use an isolated Win2K SP4 machine on which all unnecessary processes and services have been turned off. The machine I have available for this work is a dual-proc 1Ghz PIII with 1Gig of RAM. This machine is obviously not state of the art but is probably representative of what most people actually have. In any case, we're interested in relative numbers, not absolute values.

- Use a simple Jython/Java-based framework to run the tests and capture the timings (I have a little cobbled-together Python performance testing framework lying about that I can use for this).

- For each test case document (that is, the union of all the documents provided by all the FO implementations being tested), process it using the specified environment 10 times, discarding the numbers from the first run, if the first run incurs startup overhead. I think 10 iterations is sufficient to get good numbers. I can always increase this if I need to.

- If relevant, for each test case document, process it using a Saxon-based process, if the base XSLT process is not Saxon.

I think this testing methodology would be both fair and informative if the test cases are realistic. Unfortunately, I would not have time to develop my own test cases--all the real docs I have require the use of Saxon extensions and I wouldn't have time to modify the XSLTs involved to remove that dependency.

If people want to provide test cases for this purpose, I would be happy to accept them.

For test results, I will publish the raw data and trend graphs showing the performance trends as the size and nature of test cases change.

The above approach should provide a reasonable basis on which to compare the performance of different FO implementations. However, in a real production process, there are additional variables that can't really be controlled for in this type of head to head test. These include:

- The nature of the final rendition. Not everyone produces PDF. For example, IBM's publishing system provides facilities for driving AFP printers from FO for doing things like printing utility bills.

For producers of PDF, the PDF to be generated may require many different characteristics: is it intended for Web delivery? Is it input to a commercial printing process? Does it require a CMYK or process color model? Should it include color separated pages?

For many production processes, the PDF generated by the rendition engine must still be passed through a "pre-flight" process to do things like convert RGB colors to CMYK or process colors, rationalize rule weights, add or adjust page and trim boxes, etc. The degree to which you both require these characteristics in the PDF and the degree to which your FO implementation can produce them may determine the overall processing speed. For example, if you need CMYK colors and your FO implementation can only produce RGB colors and your printing house requires CMYK PDF as input (many printing house will do the preflighting for you), then you'll have to add a preflight step to your production processs. The time cost of this preflight step should be charged to the FO implementation since it could have, in theory, produced CMYK colors directly. You get the idea.

- The FO features implemented. If an FO implementation is fast but doesn't do what you need, it's performance is not really relevant.

This is all to caution, as David Tolpin did, that just comparing raw performance numbers is not enough--you have to consider all the features of the FO implementation within the context of the total system it will be a part of.

For example, most technical documentation support systems don't have performance as a top requirement because renditions are created relatively infrequently and many technical documents are relatively small (200 pages or fewer). But a few technical document use cases require top performance (e.g., aircraft maintenance manuals for commercial aircraft). By contrast, a Web-delivered, on-demand-printing application almost certainly requires top performance, possibly at the cost of FO features, as would something like an invoice printing application.

The other thing to keep in mind is that as long as Moore's law continues to hold (and we seem to have at least another good ten years left in that curve), many performance problems can be solved simply by throwing hardware at the problem. For example, I improved the performance of XSL Formatter 2.5 by at least a factor of three simply by replacing my old laptop with a new one. So while software performance is always important and should not be neglected by implementors, it is not always the most important consideration when building a complete processing system--it is often cheaper in terms of implementation and maintenance and lost features to use a hardware solution instead of choosing the fastest product. It is often the case that the fastest software also comes at a cost premium, so you have to ask the question: is it cheaper overall to spend a bit more on hardware and less on license charges? Do I get any other advantage or disadvantage by doing so? When you can build a state-of-the-art speed-optimized server for 2-5 thousand Euros (depending mostly on the size of the disk drive and number of processors), it's hard to justify a 10x greater license charge over a slower, but equally functional product, just to get a software performance speed up.

You also have to ask: what are my user's expectations or requirements for performance? You may be optimizing performance where such optimization is not required or where other parts of the system are so slow that they swamp other things, including FO processing.


W. Eliot Kimber
Innodata Isogen

XSL-List info and archive:

Current Thread