WordSeparatorAnalyzer

com.google.appengine.api.search.dev

Class WordSeparatorAnalyzer

  • java.lang.Object
    • Analyzer
      • com.google.appengine.api.search.dev.WordSeparatorAnalyzer

  • public class WordSeparatorAnalyzer
extends Analyzer
    A custom analyzer to tokenize text like the Search API backend. It detects when provided text is in a CJK language and uses CJKTokenizer to tokenize it if it is. CJKTokenizer tokenizes based on bigrams, so a string like "ABCD" will be tokenized to ["A", "AB", "BC", "CD", "D"]. If the string is not CJK, we assume that it uses standard latin word separators. For latin text, this uses a slightly-customized LetterTokenizer and passes tokens through StandardFilter and LowerCaseFilter. The LetterTokenizer is customized to use the same word separators as ST-BTI.

    • Constructor Summary

      Constructors 
      Constructor and Description
      WordSeparatorAnalyzer()
      Create a new WordSeparatorAnalyzer that always tries to detect CJK.
      WordSeparatorAnalyzer(boolean detectCjk)
      Create a new WordSeparatorAnalyzer.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method and Description
      static java.lang.String normalize(java.lang.String tokenizeString)
      Transforms to lowercase and replaces all word separators with spaces.
      static java.lang.String removeDiacriticals(java.lang.String input)
      Removes all diacritical marks from the input.
      static java.util.List<java.lang.String> tokenList(java.lang.String tokenizeString)
      Returns a list of tokens for a string.
      TokenStream tokenStream(java.lang.String fieldName, java.io.Reader reader)
      Constructs a tokenizer that can tokenize CJK or latin text.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • WordSeparatorAnalyzer

        public WordSeparatorAnalyzer(boolean detectCjk)
        Create a new WordSeparatorAnalyzer.
        Parameters:
        detectCjk - If true, will attempt to detect and segment CJK. If false, assumes all text can be segmented using word separators.

      • WordSeparatorAnalyzer

        public WordSeparatorAnalyzer()
        Create a new WordSeparatorAnalyzer that always tries to detect CJK.

    • Method Detail

      • tokenStream

        public TokenStream tokenStream(java.lang.String fieldName,
                               java.io.Reader reader)
        Constructs a tokenizer that can tokenize CJK or latin text.
        Parameters:
        fieldName - Ignored.
        reader - A stream to tokenize. mark() and reset() support is not needed.
        Returns:
        A TokenStream that represents the tokenization of the data in reader.

      • tokenList

        public static java.util.List<java.lang.String> tokenList(java.lang.String tokenizeString)
        Returns a list of tokens for a string.

      • normalize

        public static java.lang.String normalize(java.lang.String tokenizeString)
        Transforms to lowercase and replaces all word separators with spaces.

      • removeDiacriticals

        public static java.lang.String removeDiacriticals(java.lang.String input)
        Removes all diacritical marks from the input. This has the effect of transforming marked glyphs into their "equivalent" non-marked form. For example, "éøç" becomes "eoc".