unsure about the original reason to restrict regex, as it seems to
just confuse people when a regex they lovingly crafted elsewhere
doesn't work (not that the various java, Perl, etc schisms help).
I don't know the history in full, but I think there were several reasons XSD adopted a "minimal" regex subset:
(a) they wanted to be sure it could be widely implemented using existing regex engines (i.e. a highest common factor approach)
(b) they wanted to exclude anything that didn't make sense in an international Unicode context (so things like word boundaries were immediately suspect)
(c) they wanted to make sure that what they included was well specified. Finding solid specifications of regex constructs is remarkably difficult; there's a culture of very informal specification. Many times when adding constructs to the XPath spec, we've had to do empirical tests on existing regex engines such as PCRE to see how they actually handle edge cases, and very often we find differences between different engines that couldn't be guessed from the documentation. For example, there's a sorry history of patches to the spec regarding the handling of a newline character appearing as the last thing in the input. It's a shame when a feature gets left out because we can't decide what it should do in edge cases, but the standards process tends to lead to people asking such questions and expecting answers.
Michael Kay
Saxonica