Re: [xsl] Using XSLT to build an index

["Mark" <mark@xxxxxxxxxxxx>]

Hi Michael,
Thanks for the note. I took a peek and would call the small section I looked 
at a class-index or interactive table of contents - I did not go very deep. 
I'll keep it in mind as I try rolling my own. For me, the two languages pose 
some interesting problems since the Czech collator must place the diagraph 
'ch' between 'h' and 'i' for Czech words, and for English collator, put it 
in the 'c' section. My thanks to the design committee for the XSLT feature 
that permits me to select the collators I want when I want them!

I've simplified my problem tremendously by abstracting out of the main file 
the subset of phrases of interest with the HTML page title and href:

<Text ref="1984-2628.htm" title="Czechoslovak Stamp 2628" eng="Council for 
Mutual Economic Assistance 1949-1984" czech="Rada vzC!jemnC) hospodC!EskC) 
pomoci 1949-1984"/>

So, I simply have to run through a few thousand <Text> elements rather than 
raw, unformatted text.

Thanks,
Mark

-----Original Message----- 
From: Michael Kay
Sent: Sunday, October 30, 2011 4:07 PM
To: xsl-list@xxxxxxxxxxxxxxxxxxxxxx
Subject: Re: [xsl] Using XSLT to build an index

On 30/10/2011 22:07, G. Ken Holman wrote:
> At 2011-10-30 14:47 -0700, Mark wrote:
> > The list archives did not seem to contain an XSLT stylesheet that could 
> > index an XML file, but I may have missed it. Is it practical to write my 
> > own XSLT 2 indexing stylesheet? If so, I have a bilingual XML file that I 
> > want to index.
>
> Where you simply want all words, except your stop words, collected to 
> automate the index generation, I've never been successful with automated 
> indexing myself.  For my books I've authored the components of the index, 
> and then pointed to those components from within the code.
Certainly, an automatically-generated index will never come close to one
produced by hand. However, the index to the Saxon documentation at
http://www.saxonica.com/documentation/index-entries/intro.xml is
automatically-generated, and we thought it better than having nothing at
all.

This index is constructed by taking all the title, subtitle, and index
elements (the index tag allows phrases to be marked for manual
indexing), and extracts tokens using

<xsl:for-each
select="tokenize(replace(.,'\.(\s|$)','$1'),'[\W\p{S}-[\-\.'']]+')[not(.=$stop-words)]">
<token section="{$section}" page="{$page}" subpage="{$subpage}"
title="{$title}" value="{.}"/>
</xsl:for-each>

It then sorts and groups the tokens to produce the index.

I wouldn't claim this regular expression is perfect, but it worked for
the intended purpose. It could easily be improved if we had time to work
on it.

The replace() preprocesses the input by removing any full stop that is
followed by a space or by the end of the string.

The tokenize() then splits the string on any separator that is a
non-word character or a Unicode separator character, other than hyphen
or full-stop. This means that for the Saxon index, hyphens and embedded
periods are considered word characters, resulting in index entries for
terms like "xsl:strip-space" and "1.1".