LetterTokenizer
and LowerCaseFilter
@Deprecated public final class LowerCaseTokenizer extends Tokenizer
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
AttributeSource.State
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_WORD_LEN
Deprecated.
|
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
LowerCaseTokenizer()
Deprecated.
Construct a new LowerCaseTokenizer.
|
LowerCaseTokenizer(AttributeFactory factory)
Deprecated.
Construct a new LowerCaseTokenizer using a given
AttributeFactory . |
LowerCaseTokenizer(AttributeFactory factory,
int maxTokenLen)
Deprecated.
Construct a new LowerCaseTokenizer using a given
AttributeFactory . |
Modifier and Type | Method and Description |
---|---|
void |
end()
Deprecated.
|
boolean |
incrementToken()
Deprecated.
|
void |
reset()
Deprecated.
|
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public static final int DEFAULT_MAX_WORD_LEN
public LowerCaseTokenizer()
public LowerCaseTokenizer(AttributeFactory factory)
AttributeFactory
.factory
- the attribute factory to use for this Tokenizer
public LowerCaseTokenizer(AttributeFactory factory, int maxTokenLen)
AttributeFactory
.factory
- the attribute factory to use for this Tokenizer
maxTokenLen
- maximum token length the tokenizer will emit.
Must be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024)IllegalArgumentException
- if maxTokenLen is invalid.public final boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
public final void end() throws IOException
end
in class TokenStream
IOException
public void reset() throws IOException
reset
in class Tokenizer
IOException
Copyright © 2000-2020 Apache Software Foundation. All Rights Reserved.