Class UpdateLog

    • Field Detail

      • LOG_FILENAME_PATTERN

        public static String LOG_FILENAME_PATTERN
      • TLOG_NAME

        public static String TLOG_NAME
      • BUFFER_TLOG_NAME

        public static String BUFFER_TLOG_NAME
      • FLAGS_IDX

        public static final int FLAGS_IDX
        The index of the flags value in an entry from the transaction log.
        See Also:
        Constant Field Values
      • VERSION_IDX

        public static final int VERSION_IDX
        The index of the _version_ value in an entry from the transaction log.
        See Also:
        Constant Field Values
      • PREV_POINTER_IDX

        public static final int PREV_POINTER_IDX
        The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.
        See Also:
        Constant Field Values
      • PREV_VERSION_IDX

        public static final int PREV_VERSION_IDX
        The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.
        See Also:
        Constant Field Values
      • id

        protected long id
      • numOldRecords

        protected int numOldRecords
      • numRecordsToKeep

        protected int numRecordsToKeep
      • maxNumLogsToKeep

        protected int maxNumLogsToKeep
      • numVersionBuckets

        protected int numVersionBuckets
      • existOldBufferLog

        protected boolean existOldBufferLog
      • tlogFiles

        protected String[] tlogFiles
      • tlogDir

        protected Path tlogDir
      • releaseTlogDir

        protected Closeable releaseTlogDir
      • dataDir

        protected String dataDir
      • cancelApplyBufferUpdate

        protected volatile boolean cancelApplyBufferUpdate
      • startingVersions

        protected List<Long> startingVersions
      • bufferedOpsGauge

        protected com.codahale.metrics.Gauge<Integer> bufferedOpsGauge
      • applyingBufferedOpsMeter

        protected com.codahale.metrics.Meter applyingBufferedOpsMeter
      • replayOpsMeter

        protected com.codahale.metrics.Meter replayOpsMeter
      • copyOverOldUpdatesMeter

        protected com.codahale.metrics.Meter copyOverOldUpdatesMeter
      • testing_logReplayHook

        public static Runnable testing_logReplayHook
      • testing_logReplayFinishHook

        public static Runnable testing_logReplayFinishHook
    • Constructor Detail

      • UpdateLog

        public UpdateLog()
    • Method Detail

      • getTotalLogsSize

        public long getTotalLogsSize()
      • getCurrentLogSizeFromStream

        public long getCurrentLogSizeFromStream()
        Returns:
        the current transaction log's size (based on its output stream)
      • getTotalLogsNumber

        public long getTotalLogsNumber()
      • getNumRecordsToKeep

        public int getNumRecordsToKeep()
      • getMaxNumLogsToKeep

        public int getMaxNumLogsToKeep()
      • getNumVersionBuckets

        public int getNumVersionBuckets()
      • objToInt

        protected static int objToInt​(Object obj,
                                      int def)
      • ulogToTlogDir

        public static Path ulogToTlogDir​(String coreName,
                                         Path ulogDirPath,
                                         Path instancePath,
                                         String coreDataDir)
      • init

        public void init​(UpdateHandler uhandler,
                         SolrCore core)
        This must be called when a new log is created, or for an existing log whenever the core or update handler changes. It is called from the ctor of the specified UpdateHandler, so the specified uhandler will not yet be completely constructed.

        This method must be called after init(PluginInfo) is called.

      • maybeClearLog

        protected final void maybeClearLog​(SolrCore core)
      • resolveDataDir

        protected String resolveDataDir​(SolrCore core,
                                        String path)
        Resolves any relative path wrt the highest core-scoped level (whatever that means for a particular implementation). For most filesystems, this will be the core instanceDir, but there are other cases; e.g., HdfsUpdateLog will resolve paths relative to the core dataDir.

        If the input path is already absolute, it will be returned unmodified.

        This method should return the final, absolute, normalized path that defines the location of the ulog dataDir. It should not bother to resolve the tlog dir, nor do any work associated with initializing the tlog dir or its contents. Tlog dir initialization takes place later (if necessary) in initTlogDir(SolrCore).

      • initTlogDir

        protected void initTlogDir​(SolrCore core)
        Based on dataDir (whose initialization must be complete before this method is called), this method is responsible for, in order:
        1. resolving the tlog dir (in an implementation-dependent way)
        2. clearing any existing entries (if applicable) by calling maybeClearLog(SolrCore)
        3. actually creating tlog files (and handling any existing tlog files, if necessary)

        Note: implementations of this method must call maybeClearLog(SolrCore) after resolving the tlog dir, and before creating any (or handling any existing) tlog files.

      • newTransactionLog

        public TransactionLog newTransactionLog​(Path tlogFile,
                                                Collection<String> globalStrings,
                                                boolean openExisting)
        Returns a new TransactionLog. Sub-classes can override this method to change the implementation of the transaction log.
      • getTlogDir

        public String getTlogDir()
      • getUlogDir

        public String getUlogDir()
      • getStartingVersions

        public List<Long> getStartingVersions()
      • existOldBufferLog

        public boolean existOldBufferLog()
      • addOldLog

        protected void addOldLog​(TransactionLog oldLog,
                                 boolean removeOld)
      • getLogList

        public String[] getLogList​(File directory)
      • getLastLogId

        public long getLastLogId()
      • openRealtimeSearcher

        public void openRealtimeSearcher()
        Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)
      • deleteAll

        public void deleteAll()
        currently for testing only
      • trackDeleteByQuery

        protected void trackDeleteByQuery​(String q,
                                          long version)
      • newMap

        protected void newMap()
      • hasUncommittedChanges

        public boolean hasUncommittedChanges()
      • applyPartialUpdates

        public long applyPartialUpdates​(org.apache.lucene.util.BytesRef id,
                                        long prevPointer,
                                        long prevVersion,
                                        Set<String> onlyTheseFields,
                                        org.apache.solr.common.SolrDocumentBase<?,​?> latestPartialDoc)
        Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.
        Parameters:
        id - Binary representation of the unique key field
        prevPointer - Pointer to the previous entry in the ulog, based on which the current in-place update was made.
        prevVersion - Version of the previous entry in the ulog, based on which the current in-place update was made.
        onlyTheseFields - When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.
        latestPartialDoc - Partial document that is to be populated
        Returns:
        Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
      • lookup

        public Object lookup​(org.apache.lucene.util.BytesRef indexedId)
      • lookupVersion

        public Long lookupVersion​(org.apache.lucene.util.BytesRef indexedId)
      • recoverFromCurrentLog

        public Future<UpdateLog.RecoveryInfo> recoverFromCurrentLog()
        Replay current tlog, so all updates will be written to index. This is must do task for a tlog replica become a new leader.
        Returns:
        future of this task
      • copyOverBufferingUpdates

        public void copyOverBufferingUpdates​(CommitUpdateCommand cuc)
        Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state. So any updates which hasn't made it to the index is preserved in the current tlog, this also make RTG work
        Parameters:
        cuc - any updates that have version larger than the version of cuc will be copied over
      • commitAndSwitchToNewTlog

        public void commitAndSwitchToNewTlog​(CommitUpdateCommand cuc)
        Block updates, append a commit at current tlog, then copy over updates to a new tlog. So any updates which hasn't made it to the index is preserved in the current tlog
        Parameters:
        cuc - any updates that have version larger than the version of cuc will be copied over
      • copyOverOldUpdates

        public void copyOverOldUpdates​(long commitVersion)
      • copyOverOldUpdates

        public void copyOverOldUpdates​(long commitVersion,
                                       TransactionLog oldTlog)
        Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog
        Parameters:
        commitVersion - any updates that have version larger than the commitVersion will be copied over
      • ensureBufferTlog

        protected void ensureBufferTlog()
      • deleteBufferLogs

        protected void deleteBufferLogs()
      • ensureLog

        protected void ensureLog()
      • close

        public void close​(boolean committed)
      • close

        public void close​(boolean committed,
                          boolean deleteOnClose)
      • getRecentUpdates

        public UpdateLog.RecentUpdates getRecentUpdates()
        The RecentUpdates object returned must be closed after use
      • bufferUpdates

        public void bufferUpdates()
      • dropBufferedUpdates

        public boolean dropBufferedUpdates()
        Returns true if we were able to drop buffered updates and return to the ACTIVE state
      • applyBufferedUpdates

        public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
        Returns the Future to wait on, or null if no replay was needed
      • convertTlogEntryToAddUpdateCommand

        public static AddUpdateCommand convertTlogEntryToAddUpdateCommand​(SolrQueryRequest req,
                                                                          List<?> entry,
                                                                          int operation,
                                                                          long version)
        Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.
        Parameters:
        req - The request to use as the owner of the new AddUpdateCommand
        entry - Entry from the transaction log that contains the document to be added
        operation - The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entry
        version - Version already obtained from the entry.
      • deleteFile

        public static void deleteFile​(Path file)
      • clearLog

        public void clearLog()
        Clears the logs on the file system. Only call before init.