Styjun

I'm trying to make synonyms work right and for that I'm trying to understand better graphs in a token stream.

For that purpose I've built this code:

 Builder builder = CustomAnalyzer.builder();
 builder.withTokenizer(StandardTokenizerFactory.class);
 MySynonymGraphFilterFactory.registerSynonyms(Arrays.asList(
 Arrays.asList("go to", "navigate", "open")
 ));
 builder.addTokenFilter(MySynonymGraphFilterFactory.class, "synonyms", "unused");

(MySynonymGraphFilterFactory is just a hack to pass a list of lists for synonyms. It expands everything mapping everything to everything.)

 builder.addTokenFilter(FlattenGraphFilterFactory.class); // nothing changes with this!
 Analyzer analyzer = builder.build();
 TokenStream ts = analyzer.tokenStream("*", new StringReader("go to the webpage!"));

Then I call a tokenStreamToString() function that just dumps terms, position increments and position lengths (the code for that function is included at the bottom of this question):

 System.out.println(tokenStreamToString(ts));

What I don't understand is this. I get the same output whether I include FlattenGraphFilter or not. This is the output:

navigate<2> (0)open<2> (0)go to the webpage

(angle brackets show position lengths of the preceding term; parenthesis show position increments of the following term)

There's something I'm not understanding here. I'd thought that flattening the stream meant that no token will have position length > 1... was I wrong? I would greatly appreciate any help with understanding this.

PS: The implementation of my debugging function tokenStreamToString():

 static String tokenStreamToString(TokenStream stream) throws IOException
 
 CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);
 PositionIncrementAttribute posAtt = stream.addAttribute(PositionIncrementAttribute.class);
 PositionLengthAttribute posLenAtt = stream.addAttribute(PositionLengthAttribute.class);
 StringBuilder sb = new StringBuilder();
 stream.reset();
 while(stream.incrementToken())
 
 int inc = posAtt.getPositionIncrement();
 if(inc != 1)
 sb.append('(').append(inc).append(')');

 sb.append(termAtt.toString());

 int posLen = posLenAtt.getPositionLength();
 if(posLen != 1)
 sb.append('<').append(posLen).append('>');
 sb.append(' ');
 
 return sb.toString();
 

Thanks.