UpdateLog (Solr 6.6.1 API)

java.lang.Object
- org.apache.solr.update.UpdateLog

All Implemented Interfaces:

SolrMetricProducer, PluginInfoInitialized

Direct Known Subclasses:

CdcrUpdateLog, HdfsUpdateLog
```
public class UpdateLog
extends Object
implements PluginInfoInitialized, SolrMetricProducer
```
This holds references to the transaction logs and pointers for the document IDs to their exact positions in the transaction logs.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`UpdateLog.DBQ` Holds the query and the version for a DeleteByQuery command
`static class`	`UpdateLog.LogPtr`
`class`	`UpdateLog.RecentUpdates`
`static class`	`UpdateLog.RecoveryInfo`
`static class`	`UpdateLog.State`
`static class`	`UpdateLog.SyncLevel`

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`ADD`
`protected com.codahale.metrics.Meter`	`applyingBufferedOpsMeter`
`protected com.codahale.metrics.Gauge<Integer>`	`bufferedOpsGauge`
`protected boolean`	`cancelApplyBufferUpdate`
`static int`	`COMMIT`
`protected String`	`dataDir`
`protected UpdateLog.SyncLevel`	`defaultSyncLevel`
`static int`	`DELETE`
`static int`	`DELETE_BY_QUERY`
`protected LinkedList<UpdateLog.DBQ>`	`deleteByQueries`
`static int`	`FLAG_GAP`
`static int`	`FLAGS_IDX` The index of the flags value in an entry from the transaction log.
`protected Collection<String>`	`globalStrings`
`protected String`	`lastDataDir`
`static String`	`LOG_FILENAME_PATTERN`
`protected Deque<TransactionLog>`	`logs`
`protected Map<BytesRef,UpdateLog.LogPtr>`	`map`
`protected int`	`maxNumLogsToKeep`
`protected Long`	`maxVersionFromIndex`
`protected LinkedList<TransactionLog>`	`newestLogsOnStartup`
`protected int`	`numDeletesByQueryToKeep`
`protected int`	`numDeletesToKeep`
`protected int`	`numOldRecords`
`protected int`	`numRecordsToKeep`
`protected int`	`numVersionBuckets`
`protected LinkedHashMap<BytesRef,UpdateLog.LogPtr>`	`oldDeletes`
`static int`	`OPERATION_MASK`
`protected int`	`operationFlags`
`static int`	`PREV_POINTER_IDX` The index of the previous pointer in an entry from the transaction log.
`static int`	`PREV_VERSION_IDX` The index of the previous version in an entry from the transaction log.
`protected Map<BytesRef,UpdateLog.LogPtr>`	`prevMap`
`protected Map<BytesRef,UpdateLog.LogPtr>`	`prevMap2`
`protected TransactionLog`	`prevMapLog`
`protected TransactionLog`	`prevMapLog2`
`protected TransactionLog`	`prevTlog`
`protected UpdateLog.RecoveryInfo`	`recoveryInfo`
`protected com.codahale.metrics.Meter`	`replayOpsMeter`
`protected UpdateLog.State`	`state`
`static Runnable`	`testing_logReplayFinishHook`
`static Runnable`	`testing_logReplayHook`
`protected TransactionLog`	`tlog`
`static String`	`TLOG_NAME`
`protected File`	`tlogDir`
`protected String[]`	`tlogFiles`
`static int`	`UPDATE_INPLACE`
`static int`	`VERSION_IDX` The index of the _version_ value in an entry from the transaction log.
`protected VersionInfo`	`versionInfo`

Constructor Summary

Constructors
Constructor and Description

UpdateLog()

Constructors
Constructor and Description
`UpdateLog()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`add(AddUpdateCommand cmd)`
`void`	`add(AddUpdateCommand cmd, boolean clearCaches)`
`protected void`	`addOldLog(TransactionLog oldLog, boolean removeOld)`
`Future<UpdateLog.RecoveryInfo>`	`applyBufferedUpdates()` Returns the Future to wait on, or null if no replay was needed
`long`	`applyPartialUpdates(BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, SolrDocumentBase latestPartialDoc)` Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc.
`void`	`bufferUpdates()`
`void`	`cancelApplyBufferedUpdates()`
`void`	`clearLog(SolrCore core, PluginInfo ulogPluginInfo)` Clears the logs on the file system.
`void`	`close(boolean committed)`
`void`	`close(boolean committed, boolean deleteOnClose)`
`static AddUpdateCommand`	`convertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List entry, int operation, long version)` Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.
`void`	`delete(DeleteUpdateCommand cmd)`
`void`	`deleteAll()` currently for testing only
`void`	`deleteByQuery(DeleteUpdateCommand cmd)`
`static void`	`deleteFile(File file)`
`boolean`	`dropBufferedUpdates()` Returns true if we were able to drop buffered updates and return to the ACTIVE state
`protected void`	`ensureLog()`
`void`	`finish(UpdateLog.SyncLevel syncLevel)`
`Long`	`getCurrentMaxVersion()`
`List<UpdateLog.DBQ>`	`getDBQNewer(long version)`
`org.apache.hadoop.fs.FileSystem`	`getFs()`
`long`	`getLastLogId()`
`String`	`getLogDir()`
`String[]`	`getLogList(File directory)`
`int`	`getMaxNumLogsToKeep()`
`int`	`getNumRecordsToKeep()`
`int`	`getNumVersionBuckets()`
`UpdateLog.RecentUpdates`	`getRecentUpdates()` The RecentUpdates object returned must be closed after use
`int`	`getStartingOperation()`
`List<Long>`	`getStartingVersions()`
`UpdateLog.State`	`getState()`
`protected String`	`getTlogDir(SolrCore core, PluginInfo info)`
`long`	`getTotalLogsNumber()`
`long`	`getTotalLogsSize()`
`VersionInfo`	`getVersionInfo()`
`boolean`	`hasUncommittedChanges()`
`void`	`init(PluginInfo info)`
`void`	`init(UpdateHandler uhandler, SolrCore core)`
`void`	`initializeMetrics(SolrMetricManager manager, String registry, String scope)` Initializes metrics specific to this producer
`Object`	`lookup(BytesRef indexedId)`
`Long`	`lookupVersion(BytesRef indexedId)`
`protected void`	`newMap()`
`TransactionLog`	`newTransactionLog(File tlogFile, Collection<String> globalStrings, boolean openExisting)` Returns a new `TransactionLog`.
`protected static int`	`objToInt(Object obj, int def)`
`void`	`openRealtimeSearcher()` Opens a new realtime searcher and clears the id caches.
`void`	`postCommit(CommitUpdateCommand cmd)`
`void`	`postSoftCommit(CommitUpdateCommand cmd)`
`void`	`preCommit(CommitUpdateCommand cmd)`
`void`	`preSoftCommit(CommitUpdateCommand cmd)`
`Future<UpdateLog.RecoveryInfo>`	`recoverFromLog()`
`void`	`seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher)`
`protected Long`	`seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher, VersionInfo versions)` Used to seed all version buckets with the max value of the version field in the index.
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG_FILENAME_PATTERN

public static String LOG_FILENAME_PATTERN

TLOG_NAME
```
public static String TLOG_NAME
```

ADD
```
public static final int ADD
```
See Also:

Constant Field Values

DELETE
```
public static final int DELETE
```
See Also:

Constant Field Values

DELETE_BY_QUERY
```
public static final int DELETE_BY_QUERY
```
See Also:

Constant Field Values

COMMIT
```
public static final int COMMIT
```
See Also:

Constant Field Values

UPDATE_INPLACE
```
public static final int UPDATE_INPLACE
```
See Also:

Constant Field Values

FLAG_GAP
```
public static final int FLAG_GAP
```
See Also:

Constant Field Values

OPERATION_MASK
```
public static final int OPERATION_MASK
```
See Also:

Constant Field Values

FLAGS_IDX
```
public static final int FLAGS_IDX
```
The index of the flags value in an entry from the transaction log.

See Also:

Constant Field Values

VERSION_IDX
```
public static final int VERSION_IDX
```
The index of the _version_ value in an entry from the transaction log.

See Also:

Constant Field Values

PREV_POINTER_IDX
```
public static final int PREV_POINTER_IDX
```
The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.

See Also:

Constant Field Values

PREV_VERSION_IDX
```
public static final int PREV_VERSION_IDX
```
The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.

See Also:

Constant Field Values

state
```
protected UpdateLog.State state
```

operationFlags
```
protected int operationFlags
```

tlog
```
protected TransactionLog tlog
```

prevTlog
```
protected TransactionLog prevTlog
```

logs

protected final Deque<TransactionLog> logs

newestLogsOnStartup

protected LinkedList<TransactionLog> newestLogsOnStartup

numOldRecords
```
protected int numOldRecords
```

map

protected Map<BytesRef,UpdateLog.LogPtr> map

prevMap

protected Map<BytesRef,UpdateLog.LogPtr> prevMap

prevMap2

protected Map<BytesRef,UpdateLog.LogPtr> prevMap2

prevMapLog
```
protected TransactionLog prevMapLog
```

prevMapLog2
```
protected TransactionLog prevMapLog2
```

numDeletesToKeep
```
protected final int numDeletesToKeep
```
See Also:

Constant Field Values

numDeletesByQueryToKeep
```
protected final int numDeletesByQueryToKeep
```
See Also:

Constant Field Values

numRecordsToKeep
```
protected int numRecordsToKeep
```

maxNumLogsToKeep
```
protected int maxNumLogsToKeep
```

numVersionBuckets
```
protected int numVersionBuckets
```

maxVersionFromIndex
```
protected Long maxVersionFromIndex
```

oldDeletes

protected LinkedHashMap<BytesRef,UpdateLog.LogPtr> oldDeletes

deleteByQueries

protected LinkedList<UpdateLog.DBQ> deleteByQueries

tlogFiles
```
protected String[] tlogFiles
```

tlogDir
```
protected File tlogDir
```

globalStrings

protected Collection<String> globalStrings

dataDir
```
protected String dataDir
```

lastDataDir
```
protected String lastDataDir
```

versionInfo
```
protected VersionInfo versionInfo
```

defaultSyncLevel

protected UpdateLog.SyncLevel defaultSyncLevel

cancelApplyBufferUpdate

protected volatile boolean cancelApplyBufferUpdate

bufferedOpsGauge

protected com.codahale.metrics.Gauge<Integer> bufferedOpsGauge

applyingBufferedOpsMeter

protected com.codahale.metrics.Meter applyingBufferedOpsMeter

replayOpsMeter

protected com.codahale.metrics.Meter replayOpsMeter

testing_logReplayHook

public static Runnable testing_logReplayHook

testing_logReplayFinishHook

public static Runnable testing_logReplayFinishHook

recoveryInfo

protected UpdateLog.RecoveryInfo recoveryInfo

Constructor Detail
- UpdateLog
```
public UpdateLog()
```

Method Detail

getFs

public org.apache.hadoop.fs.FileSystem getFs()

getTotalLogsSize
```
public long getTotalLogsSize()
```

getTotalLogsNumber
```
public long getTotalLogsNumber()
```

getVersionInfo
```
public VersionInfo getVersionInfo()
```

getNumRecordsToKeep
```
public int getNumRecordsToKeep()
```

getMaxNumLogsToKeep
```
public int getMaxNumLogsToKeep()
```

getNumVersionBuckets
```
public int getNumVersionBuckets()
```

objToInt

protected static int objToInt(Object obj,
                              int def)

init
```
public void init(PluginInfo info)
```
Specified by:

init in interface PluginInfoInitialized

init

public void init(UpdateHandler uhandler,
                 SolrCore core)

initializeMetrics
```
public void initializeMetrics(SolrMetricManager manager,
                              String registry,
                              String scope)
```
Description copied from interface: SolrMetricProducer

Initializes metrics specific to this producer

Specified by:

initializeMetrics in interface SolrMetricProducer

Parameters:

manager - an instance of SolrMetricManager

registry - registry name where metrics are registered

scope - scope of the metrics (eg. handler name) to separate metrics of instances of the same component executing in different contexts

newTransactionLog

public TransactionLog newTransactionLog(File tlogFile,
                                        Collection<String> globalStrings,
                                        boolean openExisting)

Returns a new TransactionLog. Sub-classes can override this method to change the implementation of the transaction log.

getLogDir
```
public String getLogDir()
```

getStartingVersions

public List<Long> getStartingVersions()

getStartingOperation
```
public int getStartingOperation()
```

addOldLog

protected void addOldLog(TransactionLog oldLog,
                         boolean removeOld)

getLogList

public String[] getLogList(File directory)

getLastLogId
```
public long getLastLogId()
```

add

public void add(AddUpdateCommand cmd)

add

public void add(AddUpdateCommand cmd,
                boolean clearCaches)

delete

public void delete(DeleteUpdateCommand cmd)

deleteByQuery

public void deleteByQuery(DeleteUpdateCommand cmd)

openRealtimeSearcher
```
public void openRealtimeSearcher()
```
Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)

deleteAll
```
public void deleteAll()
```
currently for testing only

getDBQNewer

public List<UpdateLog.DBQ> getDBQNewer(long version)

newMap
```
protected void newMap()
```

hasUncommittedChanges

public boolean hasUncommittedChanges()

preCommit

public void preCommit(CommitUpdateCommand cmd)

postCommit

public void postCommit(CommitUpdateCommand cmd)

preSoftCommit

public void preSoftCommit(CommitUpdateCommand cmd)

postSoftCommit

public void postSoftCommit(CommitUpdateCommand cmd)

applyPartialUpdates
```
public long applyPartialUpdates(BytesRef id,
                                long prevPointer,
                                long prevVersion,
                                Set<String> onlyTheseFields,
                                SolrDocumentBase latestPartialDoc)
```
Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.

Parameters:

id - Binary representation of the unique key field

prevPointer - Pointer to the previous entry in the ulog, based on which the current in-place update was made.

prevVersion - Version of the previous entry in the ulog, based on which the current in-place update was made.

onlyTheseFields - When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.

latestPartialDoc - Partial document that is to be populated

Returns:

Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.

lookup

public Object lookup(BytesRef indexedId)

lookupVersion

public Long lookupVersion(BytesRef indexedId)

finish

public void finish(UpdateLog.SyncLevel syncLevel)

recoverFromLog

public Future<UpdateLog.RecoveryInfo> recoverFromLog()

ensureLog
```
protected void ensureLog()
```

close
```
public void close(boolean committed)
```

close

public void close(boolean committed,
                  boolean deleteOnClose)

getRecentUpdates
```
public UpdateLog.RecentUpdates getRecentUpdates()
```
The RecentUpdates object returned must be closed after use

bufferUpdates
```
public void bufferUpdates()
```

dropBufferedUpdates
```
public boolean dropBufferedUpdates()
```
Returns true if we were able to drop buffered updates and return to the ACTIVE state

applyBufferedUpdates
```
public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
```
Returns the Future to wait on, or null if no replay was needed

getState
```
public UpdateLog.State getState()
```

toString
```
public String toString()
```
Overrides:

toString in class Object

convertTlogEntryToAddUpdateCommand
```
public static AddUpdateCommand convertTlogEntryToAddUpdateCommand(SolrQueryRequest req,
                                                                  List entry,
                                                                  int operation,
                                                                  long version)
```
Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.

Parameters:

req - The request to use as the owner of the new AddUpdateCommand

entry - Entry from the transaction log that contains the document to be added

operation - The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entry

version - Version already obtained from the entry.

cancelApplyBufferedUpdates

public void cancelApplyBufferedUpdates()

deleteFile

public static void deleteFile(File file)

getTlogDir

protected String getTlogDir(SolrCore core,
                            PluginInfo info)

clearLog
```
public void clearLog(SolrCore core,
                     PluginInfo ulogPluginInfo)
```
Clears the logs on the file system. Only call before init.

Parameters:

core - the SolrCore

ulogPluginInfo - the init info for the UpdateHandler

getCurrentMaxVersion
```
public Long getCurrentMaxVersion()
```

seedBucketsWithHighestVersion

protected Long seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher,
                                             VersionInfo versions)

Used to seed all version buckets with the max value of the version field in the index.

seedBucketsWithHighestVersion

public void seedBucketsWithHighestVersion(SolrIndexSearcher newSearcher)

Class UpdateLog

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG_FILENAME_PATTERN

TLOG_NAME

ADD

DELETE

DELETE_BY_QUERY

COMMIT

UPDATE_INPLACE

FLAG_GAP

OPERATION_MASK

FLAGS_IDX

VERSION_IDX

PREV_POINTER_IDX

PREV_VERSION_IDX

state

operationFlags

tlog

prevTlog

logs

newestLogsOnStartup

numOldRecords

map

prevMap

prevMap2

prevMapLog

prevMapLog2

numDeletesToKeep

numDeletesByQueryToKeep

numRecordsToKeep

maxNumLogsToKeep

numVersionBuckets

maxVersionFromIndex

oldDeletes

deleteByQueries

tlogFiles

tlogDir

globalStrings

dataDir

lastDataDir

versionInfo

defaultSyncLevel

cancelApplyBufferUpdate

bufferedOpsGauge

applyingBufferedOpsMeter

replayOpsMeter

testing_logReplayHook

testing_logReplayFinishHook

recoveryInfo

Constructor Detail

UpdateLog

Method Detail

getFs

getTotalLogsSize

getTotalLogsNumber

getVersionInfo

getNumRecordsToKeep

getMaxNumLogsToKeep

getNumVersionBuckets

objToInt

init

init

initializeMetrics

newTransactionLog

getLogDir

getStartingVersions

getStartingOperation

addOldLog

getLogList

getLastLogId

add

add

delete

deleteByQuery

openRealtimeSearcher