Hi Fanyin Wang,
I'm not sure it is really a grouping problem, though it is related. But I can tell you where your 
script fails. I suspect you use the following line to determine the distinctness of some item/@id, 
but it obviously fails:
       <xsl:if test="not(preceding-sibling::item/@id = $pass_id)">
At least one problem is that the preceding-sibling does not go up the section element an down a 
preceding section. Use the preceding axis for this. But I tried and this also doesn't seem to work 
and I haven't figured out why exactly.
But I did come up with a solution (see below) that builds an index on what you are searching and 
uses that index to determine whether an item with a specific @id value is the first in document 
order or not. The xsl:key defines the index, which contains a rather complex match expression, but 
this expression merely combines your own if and foreach in the item template. The tricky part comes 
in passage2 template, where I use generate-id to determine whether the first of all OE-items with 
the same id is actually the current item. Only the first in document order is printed.
Grtz,
Geert
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xml:space="default">
 <xsl:key name="OE-related-items-by-id" match="item[@title != 
''][following-sibling::item[itemmetadata/qtimetadata/qtimetadatafield[fieldlabel = 
'item_type_code'][fieldentry ='OE']]]" use="@id" />
 <!--xsl:template match="/">
   <xsl:apply-templates select="//item" mode="passage"/>
 </xsl:template-->
 <xsl:template match="/">
   <xsl:apply-templates select="//item" mode="passage2"/>
 </xsl:template>
 <xsl:template match="item" mode="passage">
   <xsl:if 
test="descendant::qtimetadatafield/fieldentry[preceding-sibling::node()[text()='item_type_code']]/text()='OE'">
     <xsl:for-each select="preceding-sibling::item[@title !='']">
       <xsl:variable name="pass_id" select="@id"/>
       <xsl:if test="not(preceding-sibling::item/@id = $pass_id)">
         pa_id= <xsl:value-of select="@id"/><br></br>
       </xsl:if>
     </xsl:for-each>
   </xsl:if>
 </xsl:template>
 <xsl:template match="item" mode="passage2">
   <xsl:if test="generate-id(.) = generate-id(key('OE-related-items-by-id', @id)[1])">
     pa_id= <xsl:value-of select="@id"/><br></br>
  </xsl:if>
</xsl:template>
</xsl:stylesheet>