RE: [xsl] text() word lists

["Michael Kay" <mhk@xxxxxxxxx>]

XSLT 2.0 solution:

Firstly, taking the string value of the element gets rid of all the
element markup, which doesn't seem to play any role in this problem.

Then you can tokenize using the tokenize() function, being as clever as
you care about how to recognize word boundaries and inter-word space.

Then you can convert everything to lower case using the lower-case()
function.

Then you can group using for-each-group.

So it looks something like this:

Sorted by order of first appearance:

<xsl:for-each-group select="
   for $w in tokenize(string(foo), "[\s.?!]*") return lower-case($w)">
  <xsl:value-of select="current-grouping-key(), '  -  ',
count(current-group())"/>
</xsl:for-each>

Sorted by descending frequency:

<xsl:for-each-group select="
   for $w in tokenize(string(foo), "[\s.?!]*") return lower-case($w)">
  <xsl:sort select="count(current-group())" order="descending"/>
  <xsl:value-of select="current-grouping-key(), '  -  ',
count(current-group())"/>
</xsl:for-each>

Michael Kay

(If Dimitre's solution is "almost trivial", then I'm not sure how to
describe this one!)


> -----Original Message-----
> From: owner-xsl-list@xxxxxxxxxxxxxxxxxxxxxx 
> [mailto:owner-xsl-list@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of 
> Dimitre Novatchev
> Sent: 06 February 2004 16:38
> To: xsl-list@xxxxxxxxxxxxxxxxxxxxxx
> Subject: Re: [xsl] text() word lists 
> 
> 
> > Hi there,
> > 
> > I'm sure this is a faq, and I've checked the faq and 
> archive. I swear 
> > I remember someone asking about it, but I couldn't find it, so here 
> > goes.
> > 
> > I want to take an XML file of unknown elements and create
> > a word frequency list / word list.  Now, an entry on sorting in the 
> > xslt faq says this is just what xslt is bad at.  (And I'm 
> sure there 
> > are some that would say 'just go use perl', but let's say I 
> want to do 
> > it in xslt(1 or 2).
> > 
> > XSLT2 makes the tokenization of strings much easier, so 
> assuming I'm 
> > using that, if I have:
> > 
> > <foo>
> > <blort>  This is a <wibble>Test</wibble>, only a test!</blort>
> > <blort>  This really is a <wibble>great big test</wibble>, 
> only a test!
> > </blort>
> > </foo> 
> > 
> > I don't know that foo|wibble|blort  will be the element names.
> > 
> > But I want to produce both:
> > 
> > a  -- 4
> > test  -- 4
> > only -- 2
> > is  -- 2
> > this  -- 2
> > big -- 1
> > great -- 1
> > really -- 1
> > 
> > Which (unless I've missed something) should be
> > a case-insensitive list grouped by frequency
> > sorted alphabetically within this, and ignoring
> > punctuation.
> > 
> > But also:
> > 
> > a  -- 4
> > big -- 1
> > great -- 1
> > is  -- 2
> > only -- 2
> > test  -- 4
> > this  -- 2
> > really -- 1
> > 
> > Which is the same list by not grouped
> > by frequency.
> > 
> > Suggestions? Solutions?
> > 
> > Many thanks for any help,
> > -James
> > ---
> > Dr James Cummings, Oxford Text Archive, University of Oxford 
> > James.Cummings at ota.ahds.ac.uk http://users.ox.ac.uk/~jamesc/
> 
> Using FXSL and Saxon 7 (This was intended to be essentially 
> an XSLT 1.0 solution, until I realized that there cannot be 
> references to variables in xsl:key -- I need to change this a 
> little bit to work in XSLT 1.0) one would write:
> 
> <xsl:stylesheet version="1.0"  
> xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
>  xmlns:ext="http://exslt.org/common";
> >
> 
>    <xsl:import href="strSplit-to-Words.xsl"/>
>    
>    <xsl:key name="kWordByVal" match="word" 
>    use="translate(., $vUpper, $vLower)"/>
>    
>    <xsl:variable name="vUpper" select="'ABCDEFGHIJKLMNOPQRSTUVWXYZ'"/>
>    <xsl:variable name="vLower" select="'abcdefghijklmnopqrstuvwxyz'"/>
> 
>    <xsl:output indent="yes" omit-xml-declaration="yes"/>
>    
>     <xsl:template match="/">
>       <xsl:variable name="vwordNodes">
>         <xsl:call-template name="str-split-to-words">
>           <xsl:with-param name="pStr" select="/"/>
>           <xsl:with-param name="pDelimiters" 
>                           select="', &#9;&#10;&#13;!'"/>
>         </xsl:call-template>
>       </xsl:variable>  
>       
>       <xsl:for-each 
>        select="ext:node-set($vwordNodes)/*[normalize-space()]
>                     [generate-id()
>                     =
>                      generate-id(key('kWordByVal',
>                                      translate(., $vUpper, $vLower)
>                                     )[1])
>                     ]">
>           <xsl:sort select="count(key('kWordByVal',
>                                      translate(., $vUpper, $vLower)
>                                    )
>                                )"
>                             data-type="number"
>                             order="descending" />
>           
>           <xsl:value-of 
>           select="concat('&#xA;', 
>                          translate(., $vUpper, $vLower),
>                          ' - ',
>                          count(key('kWordByVal',
>                                      translate(., $vUpper, $vLower)
>                                    )
>                                )
>                          )"/>
>         
>         </xsl:for-each>
>       
>     </xsl:template>
> </xsl:stylesheet>
> 
> When this transformation is applied on your source.xml:
> 
> <foo> 
>  <blort>  This is a <wibble>Test</wibble>, only a 
> test!</blort>  <blort>  This really is a <wibble>great big 
> test</wibble>, only a test! 
>  </blort>
> </foo> 
> 
> The wanted result is produced:
> 
> a - 4
> test - 4
> this - 2
> is - 2
> only - 2
> really - 1
> great - 1
> big - 1
> 
> 
> For the other output you just have to change the "select" 
> attribute of xsl:sort.
> 
> Solving this kind of tasks is almost trivial using FXSL.
> 
> 
> Cheers,
> 
> Dimitre Novatchev
> FXSL developer, 
> 
http://fxsl.sourceforge.net/ -- the home of FXSL
Resume: http://fxsl.sf.net/DNovatchev/Resume/Res.html


__________________________________
Do you Yahoo!?
Yahoo! Finance: Get your refund fast by filing online.
http://taxes.yahoo.com/filing.html

 XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list


 XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list