Package org.apache.solr.update
Class UpdateLog
- java.lang.Object
-
- org.apache.solr.update.UpdateLog
-
- All Implemented Interfaces:
AutoCloseable
,SolrMetricProducer
,PluginInfoInitialized
public class UpdateLog extends Object implements PluginInfoInitialized, SolrMetricProducer
This holds references to the transaction logs. It also keeps a map of unique key to location in log (along with the update's version). This map is only cleared on soft or hard commit- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
UpdateLog.DBQ
Holds the query and the version for a DeleteByQuery commandprotected static class
UpdateLog.DeleteUpdate
static class
UpdateLog.LogPtr
class
UpdateLog.RecentUpdates
static class
UpdateLog.RecoveryInfo
static class
UpdateLog.State
static class
UpdateLog.SyncLevel
protected static class
UpdateLog.Update
-
Field Summary
Fields Modifier and Type Field Description static int
ADD
protected com.codahale.metrics.Meter
applyingBufferedOpsMeter
static String
BUFFER_TLOG_NAME
protected com.codahale.metrics.Gauge<Integer>
bufferedOpsGauge
protected TransactionLog
bufferTlog
protected boolean
cancelApplyBufferUpdate
static int
COMMIT
protected com.codahale.metrics.Meter
copyOverOldUpdatesMeter
protected String
dataDir
protected UpdateLog.SyncLevel
defaultSyncLevel
static int
DELETE
static int
DELETE_BY_QUERY
protected LinkedList<UpdateLog.DBQ>
deleteByQueries
protected boolean
existOldBufferLog
static int
FLAGS_IDX
The index of the flags value in an entry from the transaction log.protected Collection<String>
globalStrings
protected long
id
protected String
lastDataDir
static String
LOG_FILENAME_PATTERN
protected Deque<TransactionLog>
logs
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>
map
protected int
maxNumLogsToKeep
protected Long
maxVersionFromIndex
protected LinkedList<TransactionLog>
newestLogsOnStartup
protected int
numDeletesByQueryToKeep
protected int
numDeletesToKeep
protected int
numOldRecords
protected int
numRecordsToKeep
protected int
numVersionBuckets
protected LinkedHashMap<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>
oldDeletes
static int
OPERATION_MASK
static int
PREV_POINTER_IDX
The index of the previous pointer in an entry from the transaction log.static int
PREV_VERSION_IDX
The index of the previous version in an entry from the transaction log.protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>
prevMap
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>
prevMap2
protected TransactionLog
prevMapLog
protected TransactionLog
prevMapLog2
protected TransactionLog
prevTlog
protected TransactionLog
prevTlogOnPrecommit
protected UpdateLog.RecoveryInfo
recoveryInfo
protected com.codahale.metrics.Meter
replayOpsMeter
protected SolrMetricsContext
solrMetricsContext
protected List<Long>
startingVersions
protected UpdateLog.State
state
static Runnable
testing_logReplayFinishHook
static Runnable
testing_logReplayHook
protected TransactionLog
tlog
static String
TLOG_NAME
protected Path
tlogDir
protected String[]
tlogFiles
protected UpdateHandler
uhandler
static int
UPDATE_INPLACE
static int
VERSION_IDX
The index of the _version_ value in an entry from the transaction log.protected VersionInfo
versionInfo
-
Constructor Summary
Constructors Constructor Description UpdateLog()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
add(AddUpdateCommand cmd)
void
add(AddUpdateCommand cmd, boolean clearCaches)
protected void
addOldLog(TransactionLog oldLog, boolean removeOld)
Future<UpdateLog.RecoveryInfo>
applyBufferedUpdates()
Returns the Future to wait on, or null if no replay was neededlong
applyPartialUpdates(org.apache.lucene.util.BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, org.apache.solr.common.SolrDocumentBase<?,?> latestPartialDoc)
Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc.void
bufferUpdates()
void
clearLog(SolrCore core, PluginInfo ulogPluginInfo)
Clears the logs on the file system.void
close(boolean committed)
void
close(boolean committed, boolean deleteOnClose)
void
commitAndSwitchToNewTlog(CommitUpdateCommand cuc)
Block updates, append a commit at current tlog, then copy over updates to a new tlog.static AddUpdateCommand
convertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List<?> entry, int operation, long version)
Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.void
copyOverBufferingUpdates(CommitUpdateCommand cuc)
Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state.void
copyOverOldUpdates(long commitVersion)
void
copyOverOldUpdates(long commitVersion, TransactionLog oldTlog)
Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlogvoid
delete(DeleteUpdateCommand cmd)
void
deleteAll()
currently for testing onlyprotected void
deleteBufferLogs()
void
deleteByQuery(DeleteUpdateCommand cmd)
static void
deleteFile(Path file)
boolean
dropBufferedUpdates()
Returns true if we were able to drop buffered updates and return to the ACTIVE stateprotected void
ensureBufferTlog()
protected void
ensureLog()
boolean
existOldBufferLog()
void
finish(UpdateLog.SyncLevel syncLevel)
long
getCurrentLogSizeFromStream()
Long
getCurrentMaxVersion()
List<UpdateLog.DBQ>
getDBQNewer(long version)
long
getLastLogId()
String
getLogDir()
String[]
getLogList(File directory)
int
getMaxNumLogsToKeep()
int
getNumRecordsToKeep()
int
getNumVersionBuckets()
UpdateLog.RecentUpdates
getRecentUpdates()
The RecentUpdates object returned must be closed after useSolrMetricsContext
getSolrMetricsContext()
Implementations should return the context used inSolrMetricProducer.initializeMetrics(SolrMetricsContext, String)
to ensure proper cleanup of metrics at the end of the life-cycle of this component.List<Long>
getStartingVersions()
UpdateLog.State
getState()
protected String
getTlogDir(SolrCore core, PluginInfo info)
long
getTotalLogsNumber()
long
getTotalLogsSize()
VersionInfo
getVersionInfo()
boolean
hasUncommittedChanges()
void
init(PluginInfo info)
void
init(UpdateHandler uhandler, SolrCore core)
void
initializeMetrics(SolrMetricsContext parentContext, String scope)
Initialize metrics specific to this producer.Object
lookup(org.apache.lucene.util.BytesRef indexedId)
Long
lookupVersion(org.apache.lucene.util.BytesRef indexedId)
protected void
newMap()
TransactionLog
newTransactionLog(Path tlogFile, Collection<String> globalStrings, boolean openExisting)
Returns a newTransactionLog
.protected static int
objToInt(Object obj, int def)
void
openRealtimeSearcher()
Opens a new realtime searcher and clears the id caches.void
postCommit(CommitUpdateCommand cmd)
void
postSoftCommit(CommitUpdateCommand cmd)
void
preCommit(CommitUpdateCommand cmd)
void
preSoftCommit(CommitUpdateCommand cmd)
Future<UpdateLog.RecoveryInfo>
recoverFromCurrentLog()
Replay current tlog, so all updates will be written to index.Future<UpdateLog.RecoveryInfo>
recoverFromLog()
void
seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher)
protected Long
seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher, VersionInfo versions)
Used to seed all version buckets with the max value of the version field in the index.String
toString()
protected void
trackDeleteByQuery(String q, long version)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.solr.metrics.SolrMetricProducer
close
-
-
-
-
Field Detail
-
LOG_FILENAME_PATTERN
public static String LOG_FILENAME_PATTERN
-
TLOG_NAME
public static String TLOG_NAME
-
BUFFER_TLOG_NAME
public static String BUFFER_TLOG_NAME
-
ADD
public static final int ADD
- See Also:
- Constant Field Values
-
DELETE
public static final int DELETE
- See Also:
- Constant Field Values
-
DELETE_BY_QUERY
public static final int DELETE_BY_QUERY
- See Also:
- Constant Field Values
-
COMMIT
public static final int COMMIT
- See Also:
- Constant Field Values
-
UPDATE_INPLACE
public static final int UPDATE_INPLACE
- See Also:
- Constant Field Values
-
OPERATION_MASK
public static final int OPERATION_MASK
- See Also:
- Constant Field Values
-
FLAGS_IDX
public static final int FLAGS_IDX
The index of the flags value in an entry from the transaction log.- See Also:
- Constant Field Values
-
VERSION_IDX
public static final int VERSION_IDX
The index of the _version_ value in an entry from the transaction log.- See Also:
- Constant Field Values
-
PREV_POINTER_IDX
public static final int PREV_POINTER_IDX
The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.- See Also:
- Constant Field Values
-
PREV_VERSION_IDX
public static final int PREV_VERSION_IDX
The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.- See Also:
- Constant Field Values
-
id
protected long id
-
state
protected volatile UpdateLog.State state
-
bufferTlog
protected TransactionLog bufferTlog
-
tlog
protected TransactionLog tlog
-
prevTlog
protected TransactionLog prevTlog
-
prevTlogOnPrecommit
protected TransactionLog prevTlogOnPrecommit
-
logs
protected final Deque<TransactionLog> logs
-
newestLogsOnStartup
protected LinkedList<TransactionLog> newestLogsOnStartup
-
numOldRecords
protected int numOldRecords
-
map
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> map
-
prevMap
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> prevMap
-
prevMap2
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> prevMap2
-
prevMapLog
protected TransactionLog prevMapLog
-
prevMapLog2
protected TransactionLog prevMapLog2
-
numDeletesToKeep
protected final int numDeletesToKeep
- See Also:
- Constant Field Values
-
numDeletesByQueryToKeep
protected final int numDeletesByQueryToKeep
- See Also:
- Constant Field Values
-
numRecordsToKeep
protected int numRecordsToKeep
-
maxNumLogsToKeep
protected int maxNumLogsToKeep
-
numVersionBuckets
protected int numVersionBuckets
-
maxVersionFromIndex
protected Long maxVersionFromIndex
-
existOldBufferLog
protected boolean existOldBufferLog
-
oldDeletes
protected LinkedHashMap<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> oldDeletes
-
deleteByQueries
protected LinkedList<UpdateLog.DBQ> deleteByQueries
-
tlogFiles
protected String[] tlogFiles
-
tlogDir
protected Path tlogDir
-
globalStrings
protected Collection<String> globalStrings
-
dataDir
protected String dataDir
-
lastDataDir
protected String lastDataDir
-
versionInfo
protected VersionInfo versionInfo
-
defaultSyncLevel
protected UpdateLog.SyncLevel defaultSyncLevel
-
uhandler
protected volatile UpdateHandler uhandler
-
cancelApplyBufferUpdate
protected volatile boolean cancelApplyBufferUpdate
-
bufferedOpsGauge
protected com.codahale.metrics.Gauge<Integer> bufferedOpsGauge
-
applyingBufferedOpsMeter
protected com.codahale.metrics.Meter applyingBufferedOpsMeter
-
replayOpsMeter
protected com.codahale.metrics.Meter replayOpsMeter
-
copyOverOldUpdatesMeter
protected com.codahale.metrics.Meter copyOverOldUpdatesMeter
-
solrMetricsContext
protected SolrMetricsContext solrMetricsContext
-
testing_logReplayHook
public static Runnable testing_logReplayHook
-
testing_logReplayFinishHook
public static Runnable testing_logReplayFinishHook
-
recoveryInfo
protected UpdateLog.RecoveryInfo recoveryInfo
-
-
Method Detail
-
getTotalLogsSize
public long getTotalLogsSize()
-
getCurrentLogSizeFromStream
public long getCurrentLogSizeFromStream()
- Returns:
- the current transaction log's size (based on its output stream)
-
getTotalLogsNumber
public long getTotalLogsNumber()
-
getVersionInfo
public VersionInfo getVersionInfo()
-
getNumRecordsToKeep
public int getNumRecordsToKeep()
-
getMaxNumLogsToKeep
public int getMaxNumLogsToKeep()
-
getNumVersionBuckets
public int getNumVersionBuckets()
-
objToInt
protected static int objToInt(Object obj, int def)
-
init
public void init(PluginInfo info)
- Specified by:
init
in interfacePluginInfoInitialized
-
init
public void init(UpdateHandler uhandler, SolrCore core)
-
initializeMetrics
public void initializeMetrics(SolrMetricsContext parentContext, String scope)
Description copied from interface:SolrMetricProducer
Initialize metrics specific to this producer.- Specified by:
initializeMetrics
in interfaceSolrMetricProducer
- Parameters:
parentContext
- parent metrics context. If this component has the same life-cycle as the parent it can simply use the parent context, otherwise it should obtain a child context usingSolrMetricsContext.getChildContext(Object)
passingthis
as the child object.scope
- component scope
-
getSolrMetricsContext
public SolrMetricsContext getSolrMetricsContext()
Description copied from interface:SolrMetricProducer
Implementations should return the context used inSolrMetricProducer.initializeMetrics(SolrMetricsContext, String)
to ensure proper cleanup of metrics at the end of the life-cycle of this component. This should be the child context if one was created, or null if the parent context was used.- Specified by:
getSolrMetricsContext
in interfaceSolrMetricProducer
-
newTransactionLog
public TransactionLog newTransactionLog(Path tlogFile, Collection<String> globalStrings, boolean openExisting)
Returns a newTransactionLog
. Sub-classes can override this method to change the implementation of the transaction log.
-
getLogDir
public String getLogDir()
-
existOldBufferLog
public boolean existOldBufferLog()
-
addOldLog
protected void addOldLog(TransactionLog oldLog, boolean removeOld)
-
getLastLogId
public long getLastLogId()
-
add
public void add(AddUpdateCommand cmd)
-
add
public void add(AddUpdateCommand cmd, boolean clearCaches)
-
delete
public void delete(DeleteUpdateCommand cmd)
-
deleteByQuery
public void deleteByQuery(DeleteUpdateCommand cmd)
-
openRealtimeSearcher
public void openRealtimeSearcher()
Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)
-
deleteAll
public void deleteAll()
currently for testing only
-
trackDeleteByQuery
protected void trackDeleteByQuery(String q, long version)
-
getDBQNewer
public List<UpdateLog.DBQ> getDBQNewer(long version)
-
newMap
protected void newMap()
-
hasUncommittedChanges
public boolean hasUncommittedChanges()
-
preCommit
public void preCommit(CommitUpdateCommand cmd)
-
postCommit
public void postCommit(CommitUpdateCommand cmd)
-
preSoftCommit
public void preSoftCommit(CommitUpdateCommand cmd)
-
postSoftCommit
public void postSoftCommit(CommitUpdateCommand cmd)
-
applyPartialUpdates
public long applyPartialUpdates(org.apache.lucene.util.BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, org.apache.solr.common.SolrDocumentBase<?,?> latestPartialDoc)
Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.- Parameters:
id
- Binary representation of the unique key fieldprevPointer
- Pointer to the previous entry in the ulog, based on which the current in-place update was made.prevVersion
- Version of the previous entry in the ulog, based on which the current in-place update was made.onlyTheseFields
- When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.latestPartialDoc
- Partial document that is to be populated- Returns:
- Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
-
lookup
public Object lookup(org.apache.lucene.util.BytesRef indexedId)
-
lookupVersion
public Long lookupVersion(org.apache.lucene.util.BytesRef indexedId)
-
finish
public void finish(UpdateLog.SyncLevel syncLevel)
-
recoverFromLog
public Future<UpdateLog.RecoveryInfo> recoverFromLog()
-
recoverFromCurrentLog
public Future<UpdateLog.RecoveryInfo> recoverFromCurrentLog()
Replay current tlog, so all updates will be written to index. This is must do task for a tlog replica become a new leader.- Returns:
- future of this task
-
copyOverBufferingUpdates
public void copyOverBufferingUpdates(CommitUpdateCommand cuc)
Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state. So any updates which hasn't made it to the index is preserved in the current tlog, this also make RTG work- Parameters:
cuc
- any updates that have version larger than the version of cuc will be copied over
-
commitAndSwitchToNewTlog
public void commitAndSwitchToNewTlog(CommitUpdateCommand cuc)
Block updates, append a commit at current tlog, then copy over updates to a new tlog. So any updates which hasn't made it to the index is preserved in the current tlog- Parameters:
cuc
- any updates that have version larger than the version of cuc will be copied over
-
copyOverOldUpdates
public void copyOverOldUpdates(long commitVersion)
-
copyOverOldUpdates
public void copyOverOldUpdates(long commitVersion, TransactionLog oldTlog)
Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog- Parameters:
commitVersion
- any updates that have version larger than the commitVersion will be copied over
-
ensureBufferTlog
protected void ensureBufferTlog()
-
deleteBufferLogs
protected void deleteBufferLogs()
-
ensureLog
protected void ensureLog()
-
close
public void close(boolean committed)
-
close
public void close(boolean committed, boolean deleteOnClose)
-
getRecentUpdates
public UpdateLog.RecentUpdates getRecentUpdates()
The RecentUpdates object returned must be closed after use
-
bufferUpdates
public void bufferUpdates()
-
dropBufferedUpdates
public boolean dropBufferedUpdates()
Returns true if we were able to drop buffered updates and return to the ACTIVE state
-
applyBufferedUpdates
public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
Returns the Future to wait on, or null if no replay was needed
-
getState
public UpdateLog.State getState()
-
convertTlogEntryToAddUpdateCommand
public static AddUpdateCommand convertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List<?> entry, int operation, long version)
Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.- Parameters:
req
- The request to use as the owner of the new AddUpdateCommandentry
- Entry from the transaction log that contains the document to be addedoperation
- The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entryversion
- Version already obtained from the entry.
-
deleteFile
public static void deleteFile(Path file)
-
getTlogDir
protected String getTlogDir(SolrCore core, PluginInfo info)
-
clearLog
public void clearLog(SolrCore core, PluginInfo ulogPluginInfo)
Clears the logs on the file system. Only call before init.- Parameters:
core
- the SolrCoreulogPluginInfo
- the init info for the UpdateHandler
-
getCurrentMaxVersion
public Long getCurrentMaxVersion()
-
seedBucketsWithHighestVersion
protected Long seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher, VersionInfo versions)
Used to seed all version buckets with the max value of the version field in the index.
-
seedBucketsWithHighestVersion
public void seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher)
-
-