@Deprecated public class Token extends PackedTokenAttributeImpl implements FlagsAttribute, PayloadAttribute
The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.
The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".
A Token can optionally have metadata (a.k.a. payload) in the form of a variable
length byte array. Use PostingsEnum.getPayload()
to retrieve the
payloads from the index.
A few things to note:
TokenStreams
can be chained, one cannot assume that the Token's
current type is correct.builder
DEFAULT_TYPE
Constructor and Description |
---|
Token()
Deprecated.
Constructs a Token will null text.
|
Token(CharSequence text,
int start,
int end)
Deprecated.
Constructs a Token with the given term text, start
and end offsets.
|
Modifier and Type | Method and Description |
---|---|
void |
clear()
Deprecated.
Resets the term text, payload, flags, positionIncrement, positionLength,
startOffset, endOffset and token type to default.
|
Token |
clone()
Deprecated.
|
void |
copyTo(AttributeImpl target)
Deprecated.
|
boolean |
equals(Object obj)
Deprecated.
|
int |
getFlags()
Deprecated.
|
BytesRef |
getPayload()
Deprecated.
|
int |
hashCode()
Deprecated.
|
void |
reflectWith(AttributeReflector reflector)
Deprecated.
|
void |
setFlags(int flags)
Deprecated.
|
void |
setPayload(BytesRef payload)
Deprecated.
|
end, endOffset, getPositionIncrement, getPositionLength, getTermFrequency, setOffset, setPositionIncrement, setPositionLength, setTermFrequency, setType, startOffset, type
append, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toString
reflectAsString
finalize, getClass, notify, notifyAll, wait, wait, wait
chars, codePoints
public Token()
public Token(CharSequence text, int start, int end)
text
- term textstart
- start offset in the source textend
- end offset in the source textpublic int getFlags()
getFlags
in interface FlagsAttribute
FlagsAttribute
public void setFlags(int flags)
setFlags
in interface FlagsAttribute
FlagsAttribute
public BytesRef getPayload()
getPayload
in interface PayloadAttribute
PayloadAttribute
public void setPayload(BytesRef payload)
setPayload
in interface PayloadAttribute
PayloadAttribute
public void clear()
clear
in class PackedTokenAttributeImpl
public boolean equals(Object obj)
equals
in class PackedTokenAttributeImpl
public int hashCode()
hashCode
in class PackedTokenAttributeImpl
public Token clone()
clone
in class PackedTokenAttributeImpl
public void copyTo(AttributeImpl target)
copyTo
in class PackedTokenAttributeImpl
public void reflectWith(AttributeReflector reflector)
reflectWith
in class PackedTokenAttributeImpl
Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.