UpdateLog (Solr 7.1.0 API)

java.lang.Object
- org.apache.solr.update.UpdateLog

All Implemented Interfaces:

SolrMetricProducer, PluginInfoInitialized

Direct Known Subclasses:

CdcrUpdateLog, HdfsUpdateLog
```
public class UpdateLog
extends Object
implements PluginInfoInitialized, SolrMetricProducer
```
This holds references to the transaction logs and pointers for the document IDs to their exact positions in the transaction logs.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`UpdateLog.DBQ` Holds the query and the version for a DeleteByQuery command
`static class`	`UpdateLog.LogPtr`
`class`	`UpdateLog.RecentUpdates`
`static class`	`UpdateLog.RecoveryInfo`
`static class`	`UpdateLog.State`
`static class`	`UpdateLog.SyncLevel`

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`ADD`
`protected com.codahale.metrics.Meter`	`applyingBufferedOpsMeter`
`protected com.codahale.metrics.Gauge<Integer>`	`bufferedOpsGauge`
`protected boolean`	`cancelApplyBufferUpdate`
`static int`	`COMMIT`
`protected com.codahale.metrics.Meter`	`copyOverOldUpdatesMeter`
`protected String`	`dataDir`
`protected UpdateLog.SyncLevel`	`defaultSyncLevel`
`static int`	`DELETE`
`static int`	`DELETE_BY_QUERY`
`protected LinkedList<UpdateLog.DBQ>`	`deleteByQueries`
`static int`	`FLAG_GAP`
`static int`	`FLAGS_IDX` The index of the flags value in an entry from the transaction log.
`protected Collection<String>`	`globalStrings`
`protected String`	`lastDataDir`
`static String`	`LOG_FILENAME_PATTERN`
`protected Deque<TransactionLog>`	`logs`
`protected Map<BytesRef,UpdateLog.LogPtr>`	`map`
`protected int`	`maxNumLogsToKeep`
`protected Long`	`maxVersionFromIndex`
`protected LinkedList<TransactionLog>`	`newestLogsOnStartup`
`protected int`	`numDeletesByQueryToKeep`
`protected int`	`numDeletesToKeep`
`protected int`	`numOldRecords`
`protected int`	`numRecordsToKeep`
`protected int`	`numVersionBuckets`
`protected LinkedHashMap<BytesRef,UpdateLog.LogPtr>`	`oldDeletes`
`static int`	`OPERATION_MASK`
`protected int`	`operationFlags`
`static int`	`PREV_POINTER_IDX` The index of the previous pointer in an entry from the transaction log.
`static int`	`PREV_VERSION_IDX` The index of the previous version in an entry from the transaction log.
`protected Map<BytesRef,UpdateLog.LogPtr>`	`prevMap`
`protected Map<BytesRef,UpdateLog.LogPtr>`	`prevMap2`
`protected TransactionLog`	`prevMapLog`
`protected TransactionLog`	`prevMapLog2`
`protected TransactionLog`	`prevTlog`
`protected UpdateLog.RecoveryInfo`	`recoveryInfo`
`protected com.codahale.metrics.Meter`	`replayOpsMeter`
`protected UpdateLog.State`	`state`
`static Runnable`	`testing_logReplayFinishHook`
`static Runnable`	`testing_logReplayHook`
`protected TransactionLog`	`tlog`
`static String`	`TLOG_NAME`
`protected File`	`tlogDir`
`protected String[]`	`tlogFiles`
`static int`	`UPDATE_INPLACE`
`static int`	`VERSION_IDX` The index of the _version_ value in an entry from the transaction log.
`protected VersionInfo`	`versionInfo`

Constructor Summary

Constructors
Constructor and Description

UpdateLog()

Constructors
Constructor and Description
`UpdateLog()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`add(AddUpdateCommand cmd)`
`void`	`add(AddUpdateCommand cmd, boolean clearCaches)`
`protected void`	`addOldLog(TransactionLog oldLog, boolean removeOld)`
`Future<UpdateLog.RecoveryInfo>`	`applyBufferedUpdates()` Returns the Future to wait on, or null if no replay was needed
`long`	`applyPartialUpdates(BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, SolrDocumentBase latestPartialDoc)` Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc.
`void`	`bufferUpdates()`
`void`	`cancelApplyBufferedUpdates()`
`void`	`clearLog(SolrCore core, PluginInfo ulogPluginInfo)` Clears the logs on the file system.
`void`	`close(boolean committed)`
`void`	`close(boolean committed, boolean deleteOnClose)`
`static AddUpdateCommand`	`convertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List entry, int operation, long version)` Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.
`protected void`	`copyAndSwitchToNewTlog(CommitUpdateCommand cuc)`
`void`	`copyOverBufferingUpdates(CommitUpdateCommand cuc)` Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state.
`void`	`copyOverOldUpdates(CommitUpdateCommand cuc)` Block updates, append a commit at current tlog, then copy over updates to a new tlog.
`void`	`copyOverOldUpdates(long commitVersion)` Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog
`void`	`delete(DeleteUpdateCommand cmd)`
`void`	`deleteAll()` currently for testing only
`void`	`deleteByQuery(DeleteUpdateCommand cmd)`
`static void`	`deleteFile(File file)`
`boolean`	`dropBufferedUpdates()` Returns true if we were able to drop buffered updates and return to the ACTIVE state
`protected void`	`ensureLog()`
`void`	`finish(UpdateLog.SyncLevel syncLevel)`
`Long`	`getCurrentMaxVersion()`
`List<UpdateLog.DBQ>`	`getDBQNewer(long version)`
`org.apache.hadoop.fs.FileSystem`	`getFs()`
`long`	`getLastLogId()`
`String`	`getLogDir()`
`String[]`	`getLogList(File directory)`
`int`	`getMaxNumLogsToKeep()`
`int`	`getNumRecordsToKeep()`
`int`	`getNumVersionBuckets()`
`UpdateLog.RecentUpdates`	`getRecentUpdates()` The RecentUpdates object returned must be closed after use
`int`	`getStartingOperation()`
`List<Long>`	`getStartingVersions()`
`UpdateLog.State`	`getState()`
`protected String`	`getTlogDir(SolrCore core, PluginInfo info)`
`long`	`getTotalLogsNumber()`
`long`	`getTotalLogsSize()`
`VersionInfo`	`getVersionInfo()`
`boolean`	`hasUncommittedChanges()`
`void`	`init(PluginInfo info)`
`void`	`init(UpdateHandler uhandler, SolrCore core)`
`void`	`initializeMetrics(SolrMetricManager manager, String registry, String scope)` Initializes metrics specific to this producer
`Object`	`lookup(BytesRef indexedId)`
`Long`	`lookupVersion(BytesRef indexedId)`
`protected void`	`newMap()`
`TransactionLog`	`newTransactionLog(File tlogFile, Collection<String> globalStrings, boolean openExisting)` Returns a new `TransactionLog`.
`protected static int`	`objToInt(Object obj, int def)`
`void`	`openRealtimeSearcher()` Opens a new realtime searcher and clears the id caches.
`void`	`postCommit(CommitUpdateCommand cmd)`
`void`	`postSoftCommit(CommitUpdateCommand cmd)`
`void`	`preCommit(CommitUpdateCommand cmd)`
`void`	`preSoftCommit(CommitUpdateCommand cmd)`
`Future<UpdateLog.RecoveryInfo>`	`recoverFromCurrentLog()` Replay current tlog, so all updates will be written to index.
`Future<UpdateLog.RecoveryInfo>`	`recoverFromLog()`
`void`	`seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher)`
`protected Long`	`seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher, VersionInfo versions)` Used to seed all version buckets with the max value of the version field in the index.
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG_FILENAME_PATTERN

public static String LOG_FILENAME_PATTERN

TLOG_NAME
```
public static String TLOG_NAME
```

ADD
```
public static final int ADD
```
See Also:

Constant Field Values

DELETE
```
public static final int DELETE
```
See Also:

Constant Field Values

DELETE_BY_QUERY
```
public static final int DELETE_BY_QUERY
```
See Also:

Constant Field Values

COMMIT
```
public static final int COMMIT
```
See Also:

Constant Field Values

UPDATE_INPLACE
```
public static final int UPDATE_INPLACE
```
See Also:

Constant Field Values

FLAG_GAP
```
public static final int FLAG_GAP
```
See Also:

Constant Field Values

OPERATION_MASK
```
public static final int OPERATION_MASK
```
See Also:

Constant Field Values

FLAGS_IDX
```
public static final int FLAGS_IDX
```
The index of the flags value in an entry from the transaction log.

See Also:

Constant Field Values

VERSION_IDX
```
public static final int VERSION_IDX
```
The index of the _version_ value in an entry from the transaction log.

See Also:

Constant Field Values

PREV_POINTER_IDX
```
public static final int PREV_POINTER_IDX
```
The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.

See Also:

Constant Field Values

PREV_VERSION_IDX
```
public static final int PREV_VERSION_IDX
```
The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.

See Also:

Constant Field Values

state
```
protected UpdateLog.State state
```

operationFlags
```
protected int operationFlags
```

tlog
```
protected TransactionLog tlog
```

prevTlog
```
protected TransactionLog prevTlog
```

logs

protected final Deque<TransactionLog> logs

newestLogsOnStartup

protected LinkedList<TransactionLog> newestLogsOnStartup

numOldRecords
```
protected int numOldRecords
```

map

protected Map<BytesRef,UpdateLog.LogPtr> map

prevMap

protected Map<BytesRef,UpdateLog.LogPtr> prevMap

prevMap2

protected Map<BytesRef,UpdateLog.LogPtr> prevMap2

prevMapLog
```
protected TransactionLog prevMapLog
```

prevMapLog2
```
protected TransactionLog prevMapLog2
```

numDeletesToKeep
```
protected final int numDeletesToKeep
```
See Also:

Constant Field Values

numDeletesByQueryToKeep
```
protected final int numDeletesByQueryToKeep
```
See Also:

Constant Field Values

numRecordsToKeep
```
protected int numRecordsToKeep
```

maxNumLogsToKeep
```
protected int maxNumLogsToKeep
```

numVersionBuckets
```
protected int numVersionBuckets
```

maxVersionFromIndex
```
protected Long maxVersionFromIndex
```

oldDeletes

protected LinkedHashMap<BytesRef,UpdateLog.LogPtr> oldDeletes

deleteByQueries

protected LinkedList<UpdateLog.DBQ> deleteByQueries

tlogFiles
```
protected String[] tlogFiles
```

tlogDir
```
protected File tlogDir
```

globalStrings

protected Collection<String> globalStrings

dataDir
```
protected String dataDir
```

lastDataDir
```
protected String lastDataDir
```

versionInfo
```
protected VersionInfo versionInfo
```

defaultSyncLevel

protected UpdateLog.SyncLevel defaultSyncLevel

cancelApplyBufferUpdate

protected volatile boolean cancelApplyBufferUpdate

bufferedOpsGauge

protected com.codahale.metrics.Gauge<Integer> bufferedOpsGauge

applyingBufferedOpsMeter

protected com.codahale.metrics.Meter applyingBufferedOpsMeter

replayOpsMeter

protected com.codahale.metrics.Meter replayOpsMeter

copyOverOldUpdatesMeter

protected com.codahale.metrics.Meter copyOverOldUpdatesMeter

testing_logReplayHook

public static Runnable testing_logReplayHook

testing_logReplayFinishHook

public static Runnable testing_logReplayFinishHook

recoveryInfo

protected UpdateLog.RecoveryInfo recoveryInfo

Constructor Detail
- UpdateLog
```
public UpdateLog()
```

Method Detail
- getFs
```
public org.apache.hadoop.fs.FileSystem getFs()
```
- getTotalLogsSize
```
public long getTotalLogsSize()
```
- getTotalLogsNumber
```
public long getTotalLogsNumber()
```
- getVersionInfo
```
public VersionInfo getVersionInfo()
```
- getNumRecordsToKeep
```
public int getNumRecordsToKeep()
```
- getMaxNumLogsToKeep
```
public int getMaxNumLogsToKeep()
```
- getNumVersionBuckets
```
public int getNumVersionBuckets()
```
- objToInt
```
protected static int objToInt(Object obj,
                              int def)
```
- init
```
public void init(PluginInfo info)
```
  Specified by:
  
  init in interface PluginInfoInitialized
- init
```
public void init(UpdateHandler uhandler,
                 SolrCore core)
```
- initializeMetrics
```
public void initializeMetrics(SolrMetricManager manager,
                              String registry,
                              String scope)
```
  Description copied from interface: SolrMetricProducer
  
  Initializes metrics specific to this producer
  
  Specified by:
  
  initializeMetrics in interface SolrMetricProducer
  
  Parameters:
  
  manager - an instance of SolrMetricManager
  
  registry - registry name where metrics are registered
  
  scope - scope of the metrics (eg. handler name) to separate metrics of instances of the same component executing in different contexts
- newTransactionLog
```
public TransactionLog newTransactionLog(File tlogFile,
                                        Collection<String> globalStrings,
                                        boolean openExisting)
```
  Returns a new TransactionLog. Sub-classes can override this method to change the implementation of the transaction log.
- getLogDir
```
public String getLogDir()
```
- getStartingVersions
```
public List<Long> getStartingVersions()
```
- getStartingOperation
```
public int getStartingOperation()
```
- addOldLog
```
protected void addOldLog(TransactionLog oldLog,
                         boolean removeOld)
```
- getLogList
```
public String[] getLogList(File directory)
```
- getLastLogId
```
public long getLastLogId()
```
- add
```
public void add(AddUpdateCommand cmd)
```
- add
```
public void add(AddUpdateCommand cmd,
                boolean clearCaches)
```
- delete
```
public void delete(DeleteUpdateCommand cmd)
```
- deleteByQuery
```
public void deleteByQuery(DeleteUpdateCommand cmd)
```
- openRealtimeSearcher
```
public void openRealtimeSearcher()
```
  Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)
- deleteAll
```
public void deleteAll()
```
  currently for testing only
- getDBQNewer
```
public List<UpdateLog.DBQ> getDBQNewer(long version)
```
- newMap
```
protected void newMap()
```
- hasUncommittedChanges
```
public boolean hasUncommittedChanges()
```
- preCommit
```
public void preCommit(CommitUpdateCommand cmd)
```
- postCommit
```
public void postCommit(CommitUpdateCommand cmd)
```
- preSoftCommit
```
public void preSoftCommit(CommitUpdateCommand cmd)
```
- postSoftCommit
```
public void postSoftCommit(CommitUpdateCommand cmd)
```
- applyPartialUpdates
```
public long applyPartialUpdates(BytesRef id,
                                long prevPointer,
                                long prevVersion,
                                Set<String> onlyTheseFields,
                                SolrDocumentBase latestPartialDoc)
```
  Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.
  
  Parameters:
  
  id - Binary representation of the unique key field
  
  prevPointer - Pointer to the previous entry in the ulog, based on which the current in-place update was made.
  
  prevVersion - Version of the previous entry in the ulog, based on which the current in-place update was made.
  
  onlyTheseFields - When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.
  
  latestPartialDoc - Partial document that is to be populated
  
  Returns:
  
  Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
- lookup
```
public Object lookup(BytesRef indexedId)
```
- lookupVersion
```
public Long lookupVersion(BytesRef indexedId)
```
- finish
```
public void finish(UpdateLog.SyncLevel syncLevel)
```
- recoverFromLog
```
public Future<UpdateLog.RecoveryInfo> recoverFromLog()
```
- recoverFromCurrentLog
```
public Future<UpdateLog.RecoveryInfo> recoverFromCurrentLog()
```
  Replay current tlog, so all updates will be written to index. This is must do task for a tlog replica become a new leader.
  
  Returns:
  
  future of this task
- copyOverBufferingUpdates
```
public void copyOverBufferingUpdates(CommitUpdateCommand cuc)
```
  Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state. So any updates which hasn't made it to the index is preserved in the current tlog, this also make RTG work
  
  Parameters:
  
  cuc - any updates that have version larger than the version of cuc will be copied over
- copyOverOldUpdates
```
public void copyOverOldUpdates(CommitUpdateCommand cuc)
```
  Block updates, append a commit at current tlog, then copy over updates to a new tlog. So any updates which hasn't made it to the index is preserved in the current tlog
  
  Parameters:
  
  cuc - any updates that have version larger than the version of cuc will be copied over
- copyAndSwitchToNewTlog
```
protected void copyAndSwitchToNewTlog(CommitUpdateCommand cuc)
```
- copyOverOldUpdates
```
public void copyOverOldUpdates(long commitVersion)
```
  Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog
  
  Parameters:
  
  commitVersion - any updates that have version larger than the commitVersion will be copied over
- ensureLog
```
protected void ensureLog()
```
- close
```
public void close(boolean committed)
```
- close
```
public void close(boolean committed,
                  boolean deleteOnClose)
```
- getRecentUpdates
```
public UpdateLog.RecentUpdates getRecentUpdates()
```
  The RecentUpdates object returned must be closed after use
- bufferUpdates
```
public void bufferUpdates()
```
- dropBufferedUpdates
```
public boolean dropBufferedUpdates()
```
  Returns true if we were able to drop buffered updates and return to the ACTIVE state
- applyBufferedUpdates
```
public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
```
  Returns the Future to wait on, or null if no replay was needed
- getState
```
public UpdateLog.State getState()
```
- toString
```
public String toString()
```
  Overrides:
  
  toString in class Object
- convertTlogEntryToAddUpdateCommand
```
public static AddUpdateCommand convertTlogEntryToAddUpdateCommand(SolrQueryRequest req,
                                                                  List entry,
                                                                  int operation,
                                                                  long version)
```
  Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.
  
  Parameters:
  
  req - The request to use as the owner of the new AddUpdateCommand
  
  entry - Entry from the transaction log that contains the document to be added
  
  operation - The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entry
  
  version - Version already obtained from the entry.
- cancelApplyBufferedUpdates
```
public void cancelApplyBufferedUpdates()
```
- deleteFile
```
public static void deleteFile(File file)
```
- getTlogDir
```
protected String getTlogDir(SolrCore core,
                            PluginInfo info)
```
- clearLog
```
public void clearLog(SolrCore core,
                     PluginInfo ulogPluginInfo)
```
  Clears the logs on the file system. Only call before init.
  
  Parameters:
  
  core - the SolrCore
  
  ulogPluginInfo - the init info for the UpdateHandler
- getCurrentMaxVersion
```
public Long getCurrentMaxVersion()
```
- seedBucketsWithHighestVersion
```
protected Long seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher,
                                             VersionInfo versions)
```
  Used to seed all version buckets with the max value of the version field in the index.
- seedBucketsWithHighestVersion
```
public void seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher)
```

Class UpdateLog

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG_FILENAME_PATTERN

TLOG_NAME

ADD

DELETE

DELETE_BY_QUERY

COMMIT

UPDATE_INPLACE

FLAG_GAP

OPERATION_MASK

FLAGS_IDX

VERSION_IDX

PREV_POINTER_IDX

PREV_VERSION_IDX

state

operationFlags

tlog

prevTlog

logs

newestLogsOnStartup

numOldRecords

map

prevMap

prevMap2

prevMapLog

prevMapLog2

numDeletesToKeep

numDeletesByQueryToKeep

numRecordsToKeep

maxNumLogsToKeep

numVersionBuckets

maxVersionFromIndex

oldDeletes

deleteByQueries

tlogFiles

tlogDir

globalStrings

dataDir

lastDataDir

versionInfo

defaultSyncLevel

cancelApplyBufferUpdate

bufferedOpsGauge

applyingBufferedOpsMeter

replayOpsMeter

copyOverOldUpdatesMeter

testing_logReplayHook

testing_logReplayFinishHook

recoveryInfo

Constructor Detail

UpdateLog

Method Detail

getFs

getTotalLogsSize

getTotalLogsNumber

getVersionInfo

getNumRecordsToKeep

getMaxNumLogsToKeep

getNumVersionBuckets

objToInt

init

init

initializeMetrics

newTransactionLog

getLogDir

getStartingVersions

getStartingOperation

addOldLog

getLogList

getLastLogId

add

add

delete

deleteByQuery