Package org.apache.solr.analysis

Class LowerCaseTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.solr.analysis.LowerCaseTokenizer
All Implemented Interfaces:
Closeable, AutoCloseable
@Deprecated public final class LowerCaseTokenizer extends org.apache.lucene.analysis.Tokenizer
Deprecated.
Use LetterTokenizer and LowerCaseFilter
LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

    org.apache.lucene.util.AttributeSource.State

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    DEFAULT_MAX_WORD_LEN
    Deprecated.
     

    Fields inherited from class org.apache.lucene.analysis.Tokenizer

    input

    Fields inherited from class org.apache.lucene.analysis.TokenStream

    DEFAULT_TOKEN_ATTRIBUTE_FACTORY

  • Constructor Summary

    Constructors
    Constructor
    Description
    LowerCaseTokenizer()
    Deprecated.
    Construct a new LowerCaseTokenizer.
    LowerCaseTokenizer(org.apache.lucene.util.AttributeFactory factory)
    Deprecated.
    Construct a new LowerCaseTokenizer using a given AttributeFactory.
    LowerCaseTokenizer(org.apache.lucene.util.AttributeFactory factory, int maxTokenLen)
    Deprecated.
    Construct a new LowerCaseTokenizer using a given AttributeFactory.

  • Method Summary

    Modifier and Type
    Method
    Description
    final void
    end()
    Deprecated.
     
    final boolean
    incrementToken()
    Deprecated.
     
    void
    reset()
    Deprecated.
     

    Methods inherited from class org.apache.lucene.analysis.Tokenizer

    close, correctOffset, setReader, setReaderTestPoint

    Methods inherited from class org.apache.lucene.util.AttributeSource

    addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Field Details

    • DEFAULT_MAX_WORD_LEN

      public static final int DEFAULT_MAX_WORD_LEN
      Deprecated.
      See Also:

  • Constructor Details

    • LowerCaseTokenizer

      public LowerCaseTokenizer()
      Deprecated.
      Construct a new LowerCaseTokenizer.

    • LowerCaseTokenizer

      public LowerCaseTokenizer(org.apache.lucene.util.AttributeFactory factory)
      Deprecated.
      Construct a new LowerCaseTokenizer using a given AttributeFactory.
      Parameters:
      factory - the attribute factory to use for this Tokenizer

    • LowerCaseTokenizer

      public LowerCaseTokenizer(org.apache.lucene.util.AttributeFactory factory, int maxTokenLen)
      Deprecated.
      Construct a new LowerCaseTokenizer using a given AttributeFactory.
      Parameters:
      factory - the attribute factory to use for this Tokenizer
      maxTokenLen - maximum token length the tokenizer will emit. Must be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024)
      Throws:
      IllegalArgumentException - if maxTokenLen is invalid.

  • Method Details

    • incrementToken

      public final boolean incrementToken() throws IOException
      Deprecated.
      Specified by:
      incrementToken in class org.apache.lucene.analysis.TokenStream
      Throws:
      IOException

    • end

      public final void end() throws IOException
      Deprecated.
      Overrides:
      end in class org.apache.lucene.analysis.TokenStream
      Throws:
      IOException

    • reset

      public void reset() throws IOException
      Deprecated.
      Overrides:
      reset in class org.apache.lucene.analysis.Tokenizer
      Throws:
      IOException