Package org.apache.solr.update
Class UpdateLog
- java.lang.Object
-
- org.apache.solr.update.UpdateLog
-
- All Implemented Interfaces:
AutoCloseable,SolrMetricProducer,PluginInfoInitialized
public class UpdateLog extends Object implements PluginInfoInitialized, SolrMetricProducer
This holds references to the transaction logs. It also keeps a map of unique key to location in log (along with the update's version). This map is only cleared on soft or hard commit- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classUpdateLog.DBQHolds the query and the version for a DeleteByQuery commandprotected static classUpdateLog.DeleteUpdatestatic classUpdateLog.LogPtrclassUpdateLog.RecentUpdatesstatic classUpdateLog.RecoveryInfostatic classUpdateLog.Statestatic classUpdateLog.SyncLevelprotected static classUpdateLog.Update
-
Field Summary
Fields Modifier and Type Field Description static intADDprotected com.codahale.metrics.MeterapplyingBufferedOpsMeterstatic StringBUFFER_TLOG_NAMEprotected com.codahale.metrics.Gauge<Integer>bufferedOpsGaugeprotected TransactionLogbufferTlogprotected booleancancelApplyBufferUpdatestatic intCOMMITprotected com.codahale.metrics.MetercopyOverOldUpdatesMeterprotected StringdataDirprotected UpdateLog.SyncLeveldefaultSyncLevelstatic intDELETEstatic intDELETE_BY_QUERYprotected LinkedList<UpdateLog.DBQ>deleteByQueriesprotected booleanexistOldBufferLogstatic intFLAGS_IDXThe index of the flags value in an entry from the transaction log.protected Collection<String>globalStringsprotected longidstatic StringLOG_FILENAME_PATTERNprotected Deque<TransactionLog>logsprotected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>mapprotected intmaxNumLogsToKeepprotected Deque<TransactionLog>newestLogsOnStartupprotected intnumDeletesByQueryToKeepprotected intnumDeletesToKeepprotected intnumOldRecordsprotected intnumRecordsToKeepprotected LinkedHashMap<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>oldDeletesstatic intOPERATION_MASKstatic intPREV_POINTER_IDXThe index of the previous pointer in an entry from the transaction log.static intPREV_VERSION_IDXThe index of the previous version in an entry from the transaction log.protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>prevMapprotected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr>prevMap2protected TransactionLogprevMapLogprotected TransactionLogprevMapLog2protected TransactionLogprevTlogprotected TransactionLogprevTlogOnPrecommitprotected UpdateLog.RecoveryInforecoveryInfoprotected CloseablereleaseTlogDirprotected com.codahale.metrics.MeterreplayOpsMeterprotected SolrMetricsContextsolrMetricsContextprotected List<Long>startingVersionsprotected UpdateLog.Statestatestatic Runnabletesting_logReplayFinishHookstatic Runnabletesting_logReplayHookprotected TransactionLogtlogstatic StringTLOG_NAMEprotected PathtlogDirprotected String[]tlogFilesprotected UpdateHandleruhandlerstatic intUPDATE_INPLACEstatic intVERSION_IDXThe index of the _version_ value in an entry from the transaction log.protected VersionInfoversionInfo
-
Constructor Summary
Constructors Constructor Description UpdateLog()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description voidadd(AddUpdateCommand cmd)voidadd(AddUpdateCommand cmd, boolean clearCaches)protected voidaddOldLog(TransactionLog oldLog, boolean removeOld)Future<UpdateLog.RecoveryInfo>applyBufferedUpdates()Returns the Future to wait on, or null if no replay was neededlongapplyPartialUpdates(org.apache.lucene.util.BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, org.apache.solr.common.SolrDocumentBase<?,?> latestPartialDoc)Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc.voidbufferUpdates()voidclearLog()Clears the logs on the file system.voidclose(boolean committed)voidclose(boolean committed, boolean deleteOnClose)voidcommitAndSwitchToNewTlog(CommitUpdateCommand cuc)Block updates, append a commit at current tlog, then copy over updates to a new tlog.static AddUpdateCommandconvertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List<?> entry, int operation, long version)Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.voidcopyOverBufferingUpdates(CommitUpdateCommand cuc)Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state.voidcopyOverOldUpdates(long commitVersion)voidcopyOverOldUpdates(long commitVersion, TransactionLog oldTlog)Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlogvoiddelete(DeleteUpdateCommand cmd)voiddeleteAll()currently for testing onlyprotected voiddeleteBufferLogs()voiddeleteByQuery(DeleteUpdateCommand cmd)static voiddeleteFile(Path file)booleandropBufferedUpdates()Returns true if we were able to drop buffered updates and return to the ACTIVE stateprotected voidensureBufferTlog()protected voidensureLog()Ensures a transaction log is ready.booleanexistOldBufferLog()voidfinish(UpdateLog.SyncLevel syncLevel)longgetCurrentLogSizeFromStream()List<UpdateLog.DBQ>getDBQNewer(long version)longgetLastLogId()UpdateLocksgetLocks()String[]getLogList(File directory)intgetMaxNumLogsToKeep()intgetNumRecordsToKeep()UpdateLog.RecentUpdatesgetRecentUpdates()The RecentUpdates object returned must be closed after useSolrMetricsContextgetSolrMetricsContext()Implementations should return the context used inSolrMetricProducer.initializeMetrics(SolrMetricsContext, String)to ensure proper cleanup of metrics at the end of the life-cycle of this component.List<Long>getStartingVersions()UpdateLog.StategetState()StringgetTlogDir()longgetTotalLogsNumber()longgetTotalLogsSize()StringgetUlogDir()VersionInfogetVersionInfo()booleanhasUncommittedChanges()voidinit(PluginInfo info)voidinit(UpdateHandler uhandler, SolrCore core)This must be called when a new log is created, or for an existing log whenever the core or update handler changes.voidinitializeMetrics(SolrMetricsContext parentContext, String scope)Initialize metrics specific to this producer.protected voidinitTlogDir(SolrCore core)Based ondataDir(whose initialization must be complete before this method is called), this method is responsible for, in order: resolving the tlog dir (in an implementation-dependent way) clearing any existing entries (if applicable) by callingmaybeClearLog(SolrCore)actually creating tlog files (and handling any existing tlog files, if necessary)Objectlookup(org.apache.lucene.util.BytesRef indexedId)LonglookupVersion(org.apache.lucene.util.BytesRef indexedId)protected voidmaybeClearLog(SolrCore core)protected voidnewMap()TransactionLognewTransactionLog(Path tlogFile, Collection<String> globalStrings, boolean openExisting)Returns a newTransactionLog.protected static intobjToInt(Object obj, int def)voidopenRealtimeSearcher()Opens a new realtime searcher and clears the id caches.voidpostCommit(CommitUpdateCommand cmd)voidpostSoftCommit(CommitUpdateCommand cmd)voidpreCommit(CommitUpdateCommand cmd)voidpreSoftCommit(CommitUpdateCommand cmd)Future<UpdateLog.RecoveryInfo>recoverFromCurrentLog()Replay current tlog, so all updates will be written to index.Future<UpdateLog.RecoveryInfo>recoverFromLog()protected StringresolveDataDir(SolrCore core, String path)Resolves any relative path wrt the highest core-scoped level (whatever that means for a particular implementation).StringtoString()protected voidtrackDeleteByQuery(String q, long version)static PathulogToTlogDir(String coreName, Path ulogDirPath, Path instancePath, String coreDataDir)-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.solr.metrics.SolrMetricProducer
close
-
-
-
-
Field Detail
-
LOG_FILENAME_PATTERN
public static String LOG_FILENAME_PATTERN
-
TLOG_NAME
public static String TLOG_NAME
-
BUFFER_TLOG_NAME
public static String BUFFER_TLOG_NAME
-
ADD
public static final int ADD
- See Also:
- Constant Field Values
-
DELETE
public static final int DELETE
- See Also:
- Constant Field Values
-
DELETE_BY_QUERY
public static final int DELETE_BY_QUERY
- See Also:
- Constant Field Values
-
COMMIT
public static final int COMMIT
- See Also:
- Constant Field Values
-
UPDATE_INPLACE
public static final int UPDATE_INPLACE
- See Also:
- Constant Field Values
-
OPERATION_MASK
public static final int OPERATION_MASK
- See Also:
- Constant Field Values
-
FLAGS_IDX
public static final int FLAGS_IDX
The index of the flags value in an entry from the transaction log.- See Also:
- Constant Field Values
-
VERSION_IDX
public static final int VERSION_IDX
The index of the _version_ value in an entry from the transaction log.- See Also:
- Constant Field Values
-
PREV_POINTER_IDX
public static final int PREV_POINTER_IDX
The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.- See Also:
- Constant Field Values
-
PREV_VERSION_IDX
public static final int PREV_VERSION_IDX
The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.- See Also:
- Constant Field Values
-
id
protected long id
-
state
protected volatile UpdateLog.State state
-
bufferTlog
protected TransactionLog bufferTlog
-
tlog
protected TransactionLog tlog
-
prevTlog
protected TransactionLog prevTlog
-
prevTlogOnPrecommit
protected TransactionLog prevTlogOnPrecommit
-
logs
protected final Deque<TransactionLog> logs
-
newestLogsOnStartup
protected Deque<TransactionLog> newestLogsOnStartup
-
numOldRecords
protected int numOldRecords
-
map
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> map
-
prevMap
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> prevMap
-
prevMap2
protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> prevMap2
-
prevMapLog
protected TransactionLog prevMapLog
-
prevMapLog2
protected TransactionLog prevMapLog2
-
numDeletesToKeep
protected final int numDeletesToKeep
- See Also:
- Constant Field Values
-
numDeletesByQueryToKeep
protected final int numDeletesByQueryToKeep
- See Also:
- Constant Field Values
-
numRecordsToKeep
protected int numRecordsToKeep
-
maxNumLogsToKeep
protected int maxNumLogsToKeep
-
existOldBufferLog
protected boolean existOldBufferLog
-
oldDeletes
protected LinkedHashMap<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> oldDeletes
-
deleteByQueries
protected LinkedList<UpdateLog.DBQ> deleteByQueries
-
tlogFiles
protected String[] tlogFiles
-
tlogDir
protected Path tlogDir
-
releaseTlogDir
protected Closeable releaseTlogDir
-
globalStrings
protected Collection<String> globalStrings
-
dataDir
protected String dataDir
-
versionInfo
protected VersionInfo versionInfo
-
defaultSyncLevel
protected UpdateLog.SyncLevel defaultSyncLevel
-
uhandler
protected volatile UpdateHandler uhandler
-
cancelApplyBufferUpdate
protected volatile boolean cancelApplyBufferUpdate
-
bufferedOpsGauge
protected com.codahale.metrics.Gauge<Integer> bufferedOpsGauge
-
applyingBufferedOpsMeter
protected com.codahale.metrics.Meter applyingBufferedOpsMeter
-
replayOpsMeter
protected com.codahale.metrics.Meter replayOpsMeter
-
copyOverOldUpdatesMeter
protected com.codahale.metrics.Meter copyOverOldUpdatesMeter
-
solrMetricsContext
protected SolrMetricsContext solrMetricsContext
-
testing_logReplayHook
public static Runnable testing_logReplayHook
-
testing_logReplayFinishHook
public static Runnable testing_logReplayFinishHook
-
recoveryInfo
protected UpdateLog.RecoveryInfo recoveryInfo
-
-
Method Detail
-
getTotalLogsSize
public long getTotalLogsSize()
-
getCurrentLogSizeFromStream
public long getCurrentLogSizeFromStream()
- Returns:
- the current transaction log's size (based on its output stream)
-
getTotalLogsNumber
public long getTotalLogsNumber()
-
getVersionInfo
public VersionInfo getVersionInfo()
-
getLocks
public UpdateLocks getLocks()
-
getNumRecordsToKeep
public int getNumRecordsToKeep()
-
getMaxNumLogsToKeep
public int getMaxNumLogsToKeep()
-
objToInt
protected static int objToInt(Object obj, int def)
-
ulogToTlogDir
public static Path ulogToTlogDir(String coreName, Path ulogDirPath, Path instancePath, String coreDataDir)
-
init
public void init(PluginInfo info)
- Specified by:
initin interfacePluginInfoInitialized
-
init
public void init(UpdateHandler uhandler, SolrCore core)
This must be called when a new log is created, or for an existing log whenever the core or update handler changes. It is called from the ctor of the specifiedUpdateHandler, so the specified uhandler will not yet be completely constructed.This method must be called after
init(PluginInfo)is called.
-
maybeClearLog
protected final void maybeClearLog(SolrCore core)
-
resolveDataDir
protected String resolveDataDir(SolrCore core, String path)
Resolves any relative path wrt the highest core-scoped level (whatever that means for a particular implementation). For most filesystems, this will be the core instanceDir, but there are other cases; e.g., HdfsUpdateLog will resolve paths relative to the core dataDir.If the input path is already absolute, it will be returned unmodified.
This method should return the final, absolute, normalized path that defines the location of the ulog dataDir. It should not bother to resolve the tlog dir, nor do any work associated with initializing the tlog dir or its contents. Tlog dir initialization takes place later (if necessary) in
initTlogDir(SolrCore).
-
initTlogDir
protected void initTlogDir(SolrCore core)
Based ondataDir(whose initialization must be complete before this method is called), this method is responsible for, in order:- resolving the tlog dir (in an implementation-dependent way)
- clearing any existing entries (if applicable) by calling
maybeClearLog(SolrCore) - actually creating tlog files (and handling any existing tlog files, if necessary)
Note: implementations of this method must call
maybeClearLog(SolrCore)after resolving the tlog dir, and before creating any (or handling any existing) tlog files.
-
initializeMetrics
public void initializeMetrics(SolrMetricsContext parentContext, String scope)
Description copied from interface:SolrMetricProducerInitialize metrics specific to this producer.- Specified by:
initializeMetricsin interfaceSolrMetricProducer- Parameters:
parentContext- parent metrics context. If this component has the same life-cycle as the parent it can simply use the parent context, otherwise it should obtain a child context usingSolrMetricsContext.getChildContext(Object)passingthisas the child object.scope- component scope
-
getSolrMetricsContext
public SolrMetricsContext getSolrMetricsContext()
Description copied from interface:SolrMetricProducerImplementations should return the context used inSolrMetricProducer.initializeMetrics(SolrMetricsContext, String)to ensure proper cleanup of metrics at the end of the life-cycle of this component. This should be the child context if one was created, or null if the parent context was used.- Specified by:
getSolrMetricsContextin interfaceSolrMetricProducer
-
newTransactionLog
public TransactionLog newTransactionLog(Path tlogFile, Collection<String> globalStrings, boolean openExisting)
Returns a newTransactionLog. Sub-classes can override this method to change the implementation of the transaction log.
-
getTlogDir
public String getTlogDir()
-
getUlogDir
public String getUlogDir()
-
existOldBufferLog
public boolean existOldBufferLog()
-
addOldLog
protected void addOldLog(TransactionLog oldLog, boolean removeOld)
-
getLastLogId
public long getLastLogId()
-
add
public void add(AddUpdateCommand cmd)
-
add
public void add(AddUpdateCommand cmd, boolean clearCaches)
-
delete
public void delete(DeleteUpdateCommand cmd)
-
deleteByQuery
public void deleteByQuery(DeleteUpdateCommand cmd)
-
openRealtimeSearcher
public void openRealtimeSearcher()
Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)
-
deleteAll
public void deleteAll()
currently for testing only
-
trackDeleteByQuery
protected void trackDeleteByQuery(String q, long version)
-
getDBQNewer
public List<UpdateLog.DBQ> getDBQNewer(long version)
-
newMap
protected void newMap()
-
hasUncommittedChanges
public boolean hasUncommittedChanges()
-
preCommit
public void preCommit(CommitUpdateCommand cmd)
-
postCommit
public void postCommit(CommitUpdateCommand cmd)
-
preSoftCommit
public void preSoftCommit(CommitUpdateCommand cmd)
-
postSoftCommit
public void postSoftCommit(CommitUpdateCommand cmd)
-
applyPartialUpdates
public long applyPartialUpdates(org.apache.lucene.util.BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, org.apache.solr.common.SolrDocumentBase<?,?> latestPartialDoc)Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.- Parameters:
id- Binary representation of the unique key fieldprevPointer- Pointer to the previous entry in the ulog, based on which the current in-place update was made.prevVersion- Version of the previous entry in the ulog, based on which the current in-place update was made.onlyTheseFields- When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.latestPartialDoc- Partial document that is to be populated- Returns:
- Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
-
lookup
public Object lookup(org.apache.lucene.util.BytesRef indexedId)
-
lookupVersion
public Long lookupVersion(org.apache.lucene.util.BytesRef indexedId)
-
finish
public void finish(UpdateLog.SyncLevel syncLevel)
-
recoverFromLog
public Future<UpdateLog.RecoveryInfo> recoverFromLog()
-
recoverFromCurrentLog
public Future<UpdateLog.RecoveryInfo> recoverFromCurrentLog()
Replay current tlog, so all updates will be written to index. This is must do task for a tlog replica become a new leader.- Returns:
- future of this task
-
copyOverBufferingUpdates
public void copyOverBufferingUpdates(CommitUpdateCommand cuc)
Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state. So any updates which hasn't made it to the index is preserved in the current tlog, this also make RTG work- Parameters:
cuc- any updates that have version larger than the version of cuc will be copied over
-
commitAndSwitchToNewTlog
public void commitAndSwitchToNewTlog(CommitUpdateCommand cuc)
Block updates, append a commit at current tlog, then copy over updates to a new tlog. So any updates which hasn't made it to the index is preserved in the current tlog- Parameters:
cuc- any updates that have version larger than the version of cuc will be copied over
-
copyOverOldUpdates
public void copyOverOldUpdates(long commitVersion)
-
copyOverOldUpdates
public void copyOverOldUpdates(long commitVersion, TransactionLog oldTlog)Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog- Parameters:
commitVersion- any updates that have version larger than the commitVersion will be copied over
-
ensureBufferTlog
protected void ensureBufferTlog()
-
deleteBufferLogs
protected void deleteBufferLogs()
-
ensureLog
protected void ensureLog()
Ensures a transaction log is ready. It is either the current one, or a new one. This method must be called with the synchronization monitor on thisUpdateLog.
-
close
public void close(boolean committed)
-
close
public void close(boolean committed, boolean deleteOnClose)
-
getRecentUpdates
public UpdateLog.RecentUpdates getRecentUpdates()
The RecentUpdates object returned must be closed after use
-
bufferUpdates
public void bufferUpdates()
-
dropBufferedUpdates
public boolean dropBufferedUpdates()
Returns true if we were able to drop buffered updates and return to the ACTIVE state
-
applyBufferedUpdates
public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
Returns the Future to wait on, or null if no replay was needed
-
getState
public UpdateLog.State getState()
-
convertTlogEntryToAddUpdateCommand
public static AddUpdateCommand convertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List<?> entry, int operation, long version)
Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.- Parameters:
req- The request to use as the owner of the new AddUpdateCommandentry- Entry from the transaction log that contains the document to be addedoperation- The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entryversion- Version already obtained from the entry.
-
deleteFile
public static void deleteFile(Path file)
-
clearLog
public void clearLog()
Clears the logs on the file system. Only call before init.
-
-