RE: EZ/X Speed:

[Kay Michael <Michael.Kay@xxxxxxx>]

It's well known that any competent developer can drive his own product
faster than he can drive his competitor's, so these figures are not really
very interesting. And once developers start using the language of
second-hand car dealers in making their claims, you get quite suspicious.

It's important in any performance assessment of XSLT to take a wide variety
of stylesheets and documents of different levels of complexity, because at
the current level of maturity every product will have some things it does
badly and others it does well.

It's also important to use a variety of metrics: elapsed time for processing
a single stylesheet may not be the most important measure. In my view for
many scenarios the important metric is throughput, how many source documents
per minute can you process through the same stylesheet.

One point that's worth watching for is that the last few ounces of XSLT
conformance can be quite expensive in processing speed. For example, some of
the code in SAXON that handles namespace nodes on the output tree is
painful, given that it's dealing with situations that are very unlikely to
arise in practice. Performance comparisons should only be made between
products that fully conform to the spec.

EZ/X documentation gives no information about exactly what is implemented,
except to say that it is "highly conformant" and "supports the 8 October
proposed recommendation". I found as counter-evidence that it treats
<xsl:message> as a literal result element.

I haven't got many of my stylesheets to run successfully under EZ/X, most of
them are failing with an array bounds exception, but for what it's worth, I
did run the 800k dictionary sample, and it took 25 seconds with EZ/X against
14 seconds with SAXON and about 4 seconds with xt. This was a very crude
measurement running in each case from the command line. I've they're going
to use speed as their unique selling point they've got their work cut out.

Mike Kay

> -----Original Message-----
> From: Michael Sick [mailto:mike@xxxxxxxxxxxxx]
> Sent: 07 December 1999 18:50
> To: xsl-list@xxxxxxxxxxxxxxxx
> Cc: Steve Muench
> Subject: EZ/X Speed: Rebunking Oracle's Debunk: WAS (Getting 
> Some Facts
> Straight on Performance Claims)
> 
> 
> REBUNKING ORACLE'S DEBUNK
> Last week, Oracle's Steve Muench suggested that XSL transforms with
> Activated EZ/X are not, in fact, usually 2-3X faster than his 
> company's most
> recent XML/XSL offering. Steve is dead wrong, and though he 
> curtly suggested
> we do our homework better, we'll need to help him with his 
> this time...
> 
> WHO'S FASTEST?
> First, however, before any major flame erupts about Java XSL speed,
> Activated Intelligence wants to acknowledge that the current 
> speed leader is
> James Clark's XT - a fine package and worthy of respect for 
> many reasons. We
> do not claim EZ/XSL to be faster than XT yet <grin>, though our EZ/XML
> parser speeds past his XP parser package... If Activated also 
> manages to
> beat James' XSL package it will be a good day for all Java 
> XSL developers,
> because a new performance standard will have been set. We 
> look forward to
> that day, hopefully not too far off.
> 
> A REVIEW
> Steve didn't offer any source code, made some questionable 
> assumptions, and
> then posted his own benchmark numbers. These may have 
> satisfied him, but
> since he didn't post any code we couldn't help him identify 
> what he did
> wrong.
> 
> THANKS FOR HELPING, STEVE!
> Steve's comments helped us identify an EZ/X filesize quirk we had
> overlooked. We're grateful for his critical insights, 
> Activated is always
> glad to get help from our competitors. In fact, we have now 
> addressed an
> issue that caused EZ/X to produce output files that were legal but
> unnecessarily large. Happily, fixing this quirk made EZ/X 
> even faster -
> thanks, Steve!
> 
> ACTUAL FILES VS. NULL STREAMS
> Steve suggested that writing to our NullOutputStream class was somehow
> causing EZ/X to report unfairly fast results. Steve's 
> suggestion is simply
> wrong, and our goal was to measure the transform time and not 
> Java's i/o
> speed. We've now included an additional benchmark that shows 
> EZ/X speed
> compared to Oracle when writing to a BufferedOutputStream. 
> Writing to actual
> files rather than to our null stream really doesn't change anything.
> 
> THE TEST RESULTS
> As you can see, EZ/X is just about 2-3x faster than Oracle's 
> latest version,
> affirming our prior claim. We'll be delighted for Steve to 
> produce code that
> backs up his claims, but you can try ours out right now - 
> just download EZ/X
> and give it a shot - http://www.activated.com/download/ezx.zip
> 
> K6-450 Processor running Windows NT 4.0 SP5
> -------------------------------------------
> Benchmark   EZ/X     Oracle    Times Faster With EZ/X
> --------- --------  -------- -----------
>   1           32ms      81ms     2.5
>   2           42ms      96ms     2.3
>   3          735ms    1819ms     2.5
>   4         9110ms   24256ms     2.7
> 
> Dual Pentium II 500 Processors running Windows 2000 Build 2183 (RC3)
> -------------------------------------------------------------------
> Benchmark   EZ/X     Oracle    Times Faster With EZ/X
> --------- --------  -------- -----------
>   1           26ms      58ms     2.2
>   2           39ms      78ms     2.0
>   3         7485ms   21423ms     2.9
>   4          541ms    1549ms     2.9
> 
> Dual Pentium Pro 200 Processors running Windows NT 4.0 SP3
> ----------------------------------------------------------
> Benchmark   EZ/X     Oracle    Times Faster With EZ/X
> --------- --------  -------- -----------
>   1           67ms     137ms     2.0
>   2           95ms     183ms     1.9
>   3        18749ms   48899ms     2.6
>   4         1367ms    3680ms     2.7
> 
> 
> JVM
> ---
> All benchmarks were run using Sun JDK version "1.2.2", build
> JDK-1.2.2-001 with native threads and symcjit enabled. In all the
> tests the VM is set to use the default heap settings, except in one
> case when running Oracle to process "all_well_10x.xml" with
> "formatPlay.xsl". We found it necessary to increase the Java heap
> significantly in order to keep Oracle's package from throwing an
> OutOfMemoryException.
> 
> 
> Benchmark Description (from Steve's mail)
> --------- ------------------------------
>    1      Ken Holman's showtree-19991008.xsl
>           on a 667 byte source document
> 
>    2      Mike Brown's Fancy_XML_Tree_Viewer_34.xsl
>           on the same 667 byte source document
> 
>    3      formatPlay.xsl
>           on a 2,096,966 byte source document
>           produced by taking Jon Bosak's
>           209,710 byte all_well.xml (Shakespeare's
>           "All's Well that Ends Well") and repeating
>           it ten times inside the file.
> 
>    4      dict2.xsl
>           on a 197,164 byte source document
>           that is file 1 of 4 of Webster's
>           Revised Unabridged Dictionary
> 
> 
> Best Regards,
> 
> Michael Sick
> Activated's EZ/X Team
> 


 XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list