Re: [xsl] Latest XSLTMark benchmark

[Daniel Veillard <Daniel.Veillard@xxxxxxx>]

On Mon, Apr 02, 2001 at 09:16:19AM -0400, Eugene Kuznetsov wrote:
> Everyone: the XSLTMark 2.0 results page has been updated to explain the
> error, and we have also contacted xml.com.
> 
> Forward to what should be done:
> 
> > I do think that the benchmark should be measuring parsing plus
> > transformation plus serialization, because that is the most typical usage
> > scenario, and because if you don't measure that, a processor that
> > optimizes
> > parsing or serialization based on knowledge of the stylesheet
> > gets no credit
> > for it.
> 
> These are persuasive points, especially when it comes to some of the
> next-generation optimizations. Do any other XSLT implementors have an
> opinion on "parse+transform" vs. "transform only"?

  Sounds to me that both are useful metrics. When I run xsltproc
(the front-end for libxslt), the --timing options shows up:
   1/ stylesheet parsing time
   2/ data parsing time
   3/ transformation duration
   4/ saving time

 All 4 are potentially interesting, an implementor of course
has in mind trying to initiate as much as possible in the first
phase to get step 2 and 3 be as fast as possible.

> Kevin also correctly points out the usefulness of measuring memory
> consumption. This is even trickier to do right (although we have done some
> internal tests), but would be a very useful metric. Presumably one would
> express it as "kilobyte per kilobyte", where the first kilobyte is input
> size and the second one is heap size delta -- measuring "memory efficiency"
> of a processor.

  In this case please measure the total memory footprint too, so that
people see what is the *actual* amount of memory that a given transform
will require, not only the heap, but also the code ! Assuming a lot
of the XSLT is expected to happen on the client side, if you run 
a Java virtual machine which requires 15Megs to just warm up, then
this should be accounted too, at this level the actual size used by
the processor to represent your document input tree may not even
be significant :-)
  We already got feedback on this thread that a Java based processor
may requires a significant run warmup to show up the best level of
performances (it helps giving the system time to swap everything else
out too ;-), maybe it's time to accept the fact that different processors
have different trade-off and a benchmark is only a benchmark, i.e. 
only one axis of evaluation of a given program or code base (when
available).

Daniel

-- 
Daniel Veillard      | Red Hat Network http://redhat.com/products/network/
veillard@xxxxxxxxxx  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

 XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list