Innodata Isogen has an inherent interest in knowing what the performance
characteristics of the various FO processors are. I already make a point
of tracking the feature and extension differences among the various FO
implementations, so this would be a natural extension of that activity.
Given this interest, I would be happy to act as an informal performance
testing service for FO implementations.
If FO implementors will provide me with the items listed below, I will
perform what I think should be fair and representative performance tests
and publish the results. If I can get this information in the next week,
I should be able to publish results in time for the XML 2003 conference
(my fall is going to be very busy with client work, but I can carve out
some time for this testing).
What I need for a given FO implementation is:
1. A copy of the latest generally-available FO implementation (I already
have XEP and XSL Formatter). If the evaluation license does not allow
processing of arbitrarily-sized documents, I will need a full license as
well (I promise I won't use any such licenses for production
work--Innodata Isogen already owns licenses for FO implementations that
we use for our own business purposes).
2. Specification of how to configure the software and operating
environment for maximum performance: memory settings, OS options,
hardware settings, etc.
3. An explanation of any startup time costs that should be factored out
or noted in the test results, e.g. JVM load, first-time startup, etc.
4. Specification of the operating framework within which the
implementation should be tested: from Ant, integrated with an Apache
server, a Tomcat server, IIS, or from the command line). If possible,
please provide any driving wrappers, ant tasks, etc., so I don't have to
write them myself.
5. Specification of the XSLT processor to use and how to use it for
maximum performance. This is based on the presumption that some FO
implementations may include optimized integrations with specific XSLT
engines.
For my purposes, my customers would probably never use a system that did
not include XSLT extensions, especially custom collators. Currently only
Saxon provides facilities for integrating custom collators, so for my
purposes, the only useful comparison would be one in which saxon 6.5.3
is the XSLT processor in all cases.
Thus, to be fair, I think that any FO implementations that can use an
embedded XSLT engine that is not Saxon should be tested both with and
without.
6. How to generate PDF:
- Via Acrobat Distiller
- Via direct PDF generation
For direct generation, whether or not it is possible to produce
vector graphics w/in the resulting PDF (i.e., by including vector EPS
graphics or via SVG).
This is very important because PDF intended to be used for
commercial printing of documents should use vector graphics in the PDF
in order to ensure optimal print quality for line drawings. PDF for Web
delivery may also need vector graphics where the graphics are quite
commplex, such as complex parts diagrams, circuit diagrams, and so on.
Rasterized line drawings are usually not acceptable in these cases.
By "vector EPS" graphics, I mean EPS graphics that are PostScript
drawing commands, as opposed to EPS graphics that are bitmaps. EPS
graphics may include a "preview" image--some FO processors can only
include the preview bitmap when generating PDF directly, for the simple
reason that their tool does not include a PostScript interpreter (which
would be a silly thing to do given how inexpensive Acrobat Distiller
is--for most people it's free because it's included with other products
or their enterprise has a site license for full Acrobat). However, it is
possible to generate vector PDF graphics from SVG using tools like the
FOP SVG "rasterizer" [it's not actually a rasterizer since its output is
not a raster graphic, but I quibble] and the latest versions of both XEP
and XSL Formatter provide this feature. But if your graphics are in EPS
format, your only real choice for generating PDFs with vector graphics
is to print to PostScript and then distill it--this can incur a
significant time cost over the direct PDF generation process.
7. A set of test cases (XML documents, graphics [including vector EPS or
SVG graphics, as relevant], XSLT transforms) that your FO implementation
can process. These should be realistic and include both small and some
very large documents so I can graph representative performance trends
based on document length and complexity.
My testing method would be basically:
- Use an isolated Win2K SP4 machine on which all unnecessary processes
and services have been turned off. The machine I have available for this
work is a dual-proc 1Ghz PIII with 1Gig of RAM. This machine is
obviously not state of the art but is probably representative of what
most people actually have. In any case, we're interested in relative
numbers, not absolute values.
- Use a simple Jython/Java-based framework to run the tests and capture
the timings (I have a little cobbled-together Python performance testing
framework lying about that I can use for this).
- For each test case document (that is, the union of all the documents
provided by all the FO implementations being tested), process it using
the specified environment 10 times, discarding the numbers from the
first run, if the first run incurs startup overhead. I think 10
iterations is sufficient to get good numbers. I can always increase this
if I need to.
- If relevant, for each test case document, process it using a
Saxon-based process, if the base XSLT process is not Saxon.
I think this testing methodology would be both fair and informative if
the test cases are realistic. Unfortunately, I would not have time to
develop my own test cases--all the real docs I have require the use of
Saxon extensions and I wouldn't have time to modify the XSLTs involved
to remove that dependency.
If people want to provide test cases for this purpose, I would be happy
to accept them.
For test results, I will publish the raw data and trend graphs showing
the performance trends as the size and nature of test cases change.
The above approach should provide a reasonable basis on which to compare
the performance of different FO implementations. However, in a real
production process, there are additional variables that can't really be
controlled for in this type of head to head test. These include:
- The nature of the final rendition. Not everyone produces PDF. For
example, IBM's publishing system provides facilities for driving AFP
printers from FO for doing things like printing utility bills.
For producers of PDF, the PDF to be generated may require many different
characteristics: is it intended for Web delivery? Is it input to a
commercial printing process? Does it require a CMYK or process color
model? Should it include color separated pages?
For many production processes, the PDF generated by the rendition engine
must still be passed through a "pre-flight" process to do things like
convert RGB colors to CMYK or process colors, rationalize rule weights,
add or adjust page and trim boxes, etc. The degree to which you both
require these characteristics in the PDF and the degree to which your FO
implementation can produce them may determine the overall processing
speed. For example, if you need CMYK colors and your FO implementation
can only produce RGB colors and your printing house requires CMYK PDF as
input (many printing house will do the preflighting for you), then
you'll have to add a preflight step to your production processs. The
time cost of this preflight step should be charged to the FO
implementation since it could have, in theory, produced CMYK colors
directly. You get the idea.
- The FO features implemented. If an FO implementation is fast but
doesn't do what you need, it's performance is not really relevant.
This is all to caution, as David Tolpin did, that just comparing raw
performance numbers is not enough--you have to consider all the features
of the FO implementation within the context of the total system it will
be a part of.
For example, most technical documentation support systems don't have
performance as a top requirement because renditions are created
relatively infrequently and many technical documents are relatively
small (200 pages or fewer). But a few technical document use cases
require top performance (e.g., aircraft maintenance manuals for
commercial aircraft). By contrast, a Web-delivered, on-demand-printing
application almost certainly requires top performance, possibly at the
cost of FO features, as would something like an invoice printing
application.
The other thing to keep in mind is that as long as Moore's law continues
to hold (and we seem to have at least another good ten years left in
that curve), many performance problems can be solved simply by throwing
hardware at the problem. For example, I improved the performance of XSL
Formatter 2.5 by at least a factor of three simply by replacing my old
laptop with a new one. So while software performance is always important
and should not be neglected by implementors, it is not always the most
important consideration when building a complete processing system--it
is often cheaper in terms of implementation and maintenance and lost
features to use a hardware solution instead of choosing the fastest
product. It is often the case that the fastest software also comes at a
cost premium, so you have to ask the question: is it cheaper overall to
spend a bit more on hardware and less on license charges? Do I get any
other advantage or disadvantage by doing so? When you can build a
state-of-the-art speed-optimized server for 2-5 thousand Euros
(depending mostly on the size of the disk drive and number of
processors), it's hard to justify a 10x greater license charge over a
slower, but equally functional product, just to get a software
performance speed up.
You also have to ask: what are my user's expectations or requirements
for performance? You may be optimizing performance where such
optimization is not required or where other parts of the system are so
slow that they swamp other things, including FO processing.
Cheers,
Eliot
--
W. Eliot Kimber
Innodata Isogen
eliot@xxxxxxxxxx
www.isogen.com
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list