Subject: RE: [xsl] Regular expression /s whitespace : Which whitespace? From: "Michael Kay" <mike@xxxxxxxxxxxx> Date: Fri, 21 Apr 2006 11:14:45 +0100 |
> I am using > count(tokenize(lower-case(.),'(\s|[,.!:;])+')[string(.)]) -a > technique I retrieved from the list for counting words. I > have been questioned about the regular expression that is > being used to find white spaces. The content can contain many > kinds of whitespaces and i am being asked to defend using > this expression to find words. Does the saxon 8b > interpretation of this regular expression covers as whitespaces Splitting text into words is potentially much more complicated than this, depending on what you are trying to achieve and what natural languages you are trying to process. (For example, in English, how should hypenated words be handled? What if the hyphen appears just before a newline?) Only you can judge whether this crude approach is good enough to meet your particular requirements. The definition of \s is found in http://www.w3.org/TR/xmlschema-2/#regexs (which XPath 2.0 references normatively): it matches any one of the four characters x09, x0A, x0D, x20. You could consider replacing the second argument of tokenize with '\W+' which matches any sequence of characters classified in Unicode as separators, punctuation, or "other" - loosely, non-word characters. Michael Kay http://www.saxonica.com/
Current Thread |
---|
|
<- Previous | Index | Next -> |
---|---|---|
Re: [xsl] Regular expression /s whi, Wendell Piez | Thread | [xsl] Phonetic Sorting for Japanese, Subhojit Chatterjee |
Re: [xsl] Regular expression /s whi, Wendell Piez | Date | RE: [xsl] Character 150 withs Windo, Michael Kay |
Month |