Package org.apache.solr.update

Class UpdateLog

java.lang.Object
org.apache.solr.update.UpdateLog
All Implemented Interfaces:
AutoCloseable, SolrMetricProducer, PluginInfoInitialized
public class UpdateLog extends Object implements PluginInfoInitialized, SolrMetricProducer
This holds references to the transaction logs. It also keeps a map of unique key to location in log (along with the update's version). This map is only cleared on soft or hard commit
WARNING: This API is experimental and might change in incompatible ways in the next release.

  • Field Details

    • LOG_FILENAME_PATTERN

      public static String LOG_FILENAME_PATTERN

    • TLOG_NAME

      public static String TLOG_NAME

    • BUFFER_TLOG_NAME

      public static String BUFFER_TLOG_NAME

    • ADD

      public static final int ADD
      See Also:

    • DELETE

      public static final int DELETE
      See Also:

    • DELETE_BY_QUERY

      public static final int DELETE_BY_QUERY
      See Also:

    • COMMIT

      public static final int COMMIT
      See Also:

    • UPDATE_INPLACE

      public static final int UPDATE_INPLACE
      See Also:

    • OPERATION_MASK

      public static final int OPERATION_MASK
      See Also:

    • FLAGS_IDX

      public static final int FLAGS_IDX
      The index of the flags value in an entry from the transaction log.
      See Also:

    • VERSION_IDX

      public static final int VERSION_IDX
      The index of the _version_ value in an entry from the transaction log.
      See Also:

    • PREV_POINTER_IDX

      public static final int PREV_POINTER_IDX
      The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.
      See Also:

    • PREV_VERSION_IDX

      public static final int PREV_VERSION_IDX
      The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.
      See Also:

    • id

      protected long id

    • state

      protected volatile UpdateLog.State state

    • bufferTlog

      protected TransactionLog bufferTlog

    • tlog

      protected TransactionLog tlog

    • prevTlog

      protected TransactionLog prevTlog

    • prevTlogOnPrecommit

      protected TransactionLog prevTlogOnPrecommit

    • logs

      protected final Deque<TransactionLog> logs

    • newestLogsOnStartup

      protected Deque<TransactionLog> newestLogsOnStartup

    • numOldRecords

      protected int numOldRecords

    • map

      protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> map

    • prevMap

      protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> prevMap

    • prevMap2

      protected Map<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> prevMap2

    • prevMapLog

      protected TransactionLog prevMapLog

    • prevMapLog2

      protected TransactionLog prevMapLog2

    • numDeletesToKeep

      protected final int numDeletesToKeep
      See Also:

    • numDeletesByQueryToKeep

      protected final int numDeletesByQueryToKeep
      See Also:

    • numRecordsToKeep

      protected int numRecordsToKeep

    • maxNumLogsToKeep

      protected int maxNumLogsToKeep

    • existOldBufferLog

      protected boolean existOldBufferLog

    • oldDeletes

      protected LinkedHashMap<org.apache.lucene.util.BytesRef,UpdateLog.LogPtr> oldDeletes

    • deleteByQueries

      protected LinkedList<UpdateLog.DBQ> deleteByQueries

    • tlogFiles

      protected String[] tlogFiles

    • tlogDir

      protected Path tlogDir

    • releaseTlogDir

      protected Closeable releaseTlogDir

    • globalStrings

      protected Collection<String> globalStrings

    • dataDir

      protected String dataDir

    • versionInfo

      protected VersionInfo versionInfo

    • defaultSyncLevel

      protected UpdateLog.SyncLevel defaultSyncLevel

    • uhandler

      protected volatile UpdateHandler uhandler

    • cancelApplyBufferUpdate

      protected volatile boolean cancelApplyBufferUpdate

    • startingVersions

      protected List<Long> startingVersions

    • applyingBufferedOpsCounter

      protected AttributedLongCounter applyingBufferedOpsCounter

    • replayOpsCounter

      protected AttributedLongCounter replayOpsCounter

    • copyOverOldUpdatesCounter

      protected AttributedLongCounter copyOverOldUpdatesCounter

    • toClose

      protected List<AutoCloseable> toClose

    • solrMetricsContext

      protected SolrMetricsContext solrMetricsContext

    • testing_logReplayHook

      public static Runnable testing_logReplayHook

    • testing_logReplayFinishHook

      public static Runnable testing_logReplayFinishHook

    • recoveryInfo

      protected UpdateLog.RecoveryInfo recoveryInfo

  • Constructor Details

    • UpdateLog

      public UpdateLog()

  • Method Details

    • getTotalLogsSize

      public long getTotalLogsSize()

    • getCurrentLogSizeFromStream

      public long getCurrentLogSizeFromStream()
      Returns:
      the current transaction log's size (based on its output stream)

    • getTotalLogsNumber

      public long getTotalLogsNumber()

    • getVersionInfo

      public VersionInfo getVersionInfo()

    • getLocks

      public UpdateLocks getLocks()

    • getNumRecordsToKeep

      public int getNumRecordsToKeep()

    • getMaxNumLogsToKeep

      public int getMaxNumLogsToKeep()

    • objToInt

      protected static int objToInt(Object obj, int def)

    • ulogToTlogDir

      public static Path ulogToTlogDir(String coreName, Path ulogDirPath, Path instancePath, String coreDataDir)

    • init

      public void init(PluginInfo info)
      Specified by:
      init in interface PluginInfoInitialized

    • init

      public void init(UpdateHandler uhandler, SolrCore core)
      This must be called when a new log is created, or for an existing log whenever the core or update handler changes. It is called from the ctor of the specified UpdateHandler, so the specified uhandler will not yet be completely constructed.

      This method must be called after init(PluginInfo) is called.

    • maybeClearLog

      protected final void maybeClearLog(SolrCore core)

    • resolveDataDir

      protected String resolveDataDir(SolrCore core, String path)
      Resolves any relative path wrt the highest core-scoped level (whatever that means for a particular implementation). For most filesystems, this will be the core instanceDir, but that is not a hard and fast rule.

      If the input path is already absolute, it will be returned unmodified.

      This method should return the final, absolute, normalized path that defines the location of the ulog dataDir. It should not bother to resolve the tlog dir, nor do any work associated with initializing the tlog dir or its contents. Tlog dir initialization takes place later (if necessary) in initTlogDir(SolrCore).

    • initTlogDir

      protected void initTlogDir(SolrCore core)
      Based on dataDir (whose initialization must be complete before this method is called), this method is responsible for, in order:
      1. resolving the tlog dir (in an implementation-dependent way)
      2. clearing any existing entries (if applicable) by calling maybeClearLog(SolrCore)
      3. actually creating tlog files (and handling any existing tlog files, if necessary)

      Note: implementations of this method must call maybeClearLog(SolrCore) after resolving the tlog dir, and before creating any (or handling any existing) tlog files.

    • initializeMetrics

      public void initializeMetrics(SolrMetricsContext parentContext, io.opentelemetry.api.common.Attributes attributes)
      Description copied from interface: SolrMetricProducer
      Implementation should initialize all metrics to a SolrMetricsContext Registry/MeterProvider with Attributes as the common set of attributes that will be attached to every metric that is initialized for that class/component
      Specified by:
      initializeMetrics in interface SolrMetricProducer
      Parameters:
      parentContext - The registry that the component will initialize metrics to
      attributes - Base set of attributes that will be bound to all metrics for that component

    • getSolrMetricsContext

      public SolrMetricsContext getSolrMetricsContext()
      Description copied from interface: SolrMetricProducer
      Implementations should return the context used in SolrMetricProducer.initializeMetrics(SolrMetricsContext, Attributes) to ensure proper cleanup of metrics at the end of the life-cycle of this component. This should be the child context if one was created, or null if the parent context was used.
      Specified by:
      getSolrMetricsContext in interface SolrMetricProducer

    • newTransactionLog

      public TransactionLog newTransactionLog(Path tlogFile, Collection<String> globalStrings, boolean openExisting)
      Returns a new TransactionLog. Sub-classes can override this method to change the implementation of the transaction log.

    • getTlogDir

      public String getTlogDir()

    • getUlogDir

      public String getUlogDir()

    • getStartingVersions

      public List<Long> getStartingVersions()

    • existOldBufferLog

      public boolean existOldBufferLog()

    • addOldLog

      protected void addOldLog(TransactionLog oldLog, boolean removeOld)

    • getLogList

      public String[] getLogList(Path directory)

    • getLastLogId

      public long getLastLogId()

    • add

      public void add(AddUpdateCommand cmd)

    • add

      public void add(AddUpdateCommand cmd, boolean clearCaches)

    • delete

      public void delete(DeleteUpdateCommand cmd)

    • deleteByQuery

      public void deleteByQuery(DeleteUpdateCommand cmd)

    • openRealtimeSearcher

      public void openRealtimeSearcher()
      Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)

    • deleteAll

      public void deleteAll()
      currently for testing only

    • trackDeleteByQuery

      protected void trackDeleteByQuery(String q, long version)

    • getDBQNewer

      public List<UpdateLog.DBQ> getDBQNewer(long version)

    • newMap

      protected void newMap()

    • hasUncommittedChanges

      public boolean hasUncommittedChanges()

    • preCommit

      public void preCommit(CommitUpdateCommand cmd)

    • postCommit

      public void postCommit(CommitUpdateCommand cmd)

    • preSoftCommit

      public void preSoftCommit(CommitUpdateCommand cmd)

    • postSoftCommit

      public void postSoftCommit(CommitUpdateCommand cmd)

    • applyPartialUpdates

      public long applyPartialUpdates(org.apache.lucene.util.BytesRef id, long prevPointer, long prevVersion, Set<String> onlyTheseFields, org.apache.solr.common.SolrDocumentBase<?,?> latestPartialDoc)
      Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.
      Parameters:
      id - Binary representation of the unique key field
      prevPointer - Pointer to the previous entry in the ulog, based on which the current in-place update was made.
      prevVersion - Version of the previous entry in the ulog, based on which the current in-place update was made.
      onlyTheseFields - When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.
      latestPartialDoc - Partial document that is to be populated
      Returns:
      Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.

    • lookup

      public Object lookup(org.apache.lucene.util.BytesRef indexedId)

    • lookupVersion

      public Long lookupVersion(org.apache.lucene.util.BytesRef indexedId)

    • finish

      public void finish(UpdateLog.SyncLevel syncLevel)

    • recoverFromLog

      public Future<UpdateLog.RecoveryInfo> recoverFromLog()

    • recoverFromCurrentLog

      public Future<UpdateLog.RecoveryInfo> recoverFromCurrentLog()
      Replay current tlog, so all updates will be written to index. This is must do task for a tlog replica become a new leader.
      Returns:
      future of this task

    • copyOverBufferingUpdates

      public void copyOverBufferingUpdates(CommitUpdateCommand cuc)
      Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state. So any updates which hasn't made it to the index is preserved in the current tlog, this also make RTG work
      Parameters:
      cuc - any updates that have version larger than the version of cuc will be copied over

    • commitAndSwitchToNewTlog

      public void commitAndSwitchToNewTlog(CommitUpdateCommand cuc)
      Block updates, append a commit at current tlog, then copy over updates to a new tlog. So any updates which hasn't made it to the index is preserved in the current tlog
      Parameters:
      cuc - any updates that have version larger than the version of cuc will be copied over

    • copyOverOldUpdates

      public void copyOverOldUpdates(long commitVersion)

    • copyOverOldUpdates

      public void copyOverOldUpdates(long commitVersion, TransactionLog oldTlog)
      Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog
      Parameters:
      commitVersion - any updates that have version larger than the commitVersion will be copied over

    • ensureBufferTlog

      protected void ensureBufferTlog()

    • deleteBufferLogs

      protected void deleteBufferLogs()

    • ensureLog

      protected void ensureLog()
      Ensures a transaction log is ready. It is either the current one, or a new one. This method must be called with the synchronization monitor on this UpdateLog.

    • close

      public void close(boolean committed)

    • close

      public void close(boolean committed, boolean deleteOnClose)

    • getRecentUpdates

      public UpdateLog.RecentUpdates getRecentUpdates()
      The RecentUpdates object returned must be closed after use

    • bufferUpdates

      public void bufferUpdates()

    • dropBufferedUpdates

      public boolean dropBufferedUpdates()
      Returns true if we were able to drop buffered updates and return to the ACTIVE state

    • applyBufferedUpdates

      public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
      Returns the Future to wait on, or null if no replay was needed

    • getState

      public UpdateLog.State getState()

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • convertTlogEntryToAddUpdateCommand

      public static AddUpdateCommand convertTlogEntryToAddUpdateCommand(SolrQueryRequest req, List<?> entry, int operation, long version)
      Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.
      Parameters:
      req - The request to use as the owner of the new AddUpdateCommand
      entry - Entry from the transaction log that contains the document to be added
      operation - The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entry
      version - Version already obtained from the entry.

    • deleteFile

      public static void deleteFile(Path file)

    • clearLog

      public void clearLog()
      Clears the logs on the file system. Only call before init.