Class UpdateLog

    • Field Detail

      • LOG_FILENAME_PATTERN

        public static String LOG_FILENAME_PATTERN
      • TLOG_NAME

        public static String TLOG_NAME
      • BUFFER_TLOG_NAME

        public static String BUFFER_TLOG_NAME
      • FLAGS_IDX

        public static final int FLAGS_IDX
        The index of the flags value in an entry from the transaction log.
        See Also:
        Constant Field Values
      • VERSION_IDX

        public static final int VERSION_IDX
        The index of the _version_ value in an entry from the transaction log.
        See Also:
        Constant Field Values
      • PREV_POINTER_IDX

        public static final int PREV_POINTER_IDX
        The index of the previous pointer in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.
        See Also:
        Constant Field Values
      • PREV_VERSION_IDX

        public static final int PREV_VERSION_IDX
        The index of the previous version in an entry from the transaction log. This is only relevant if flags (indexed at FLAGS_IDX) includes UPDATE_INPLACE.
        See Also:
        Constant Field Values
      • numOldRecords

        protected int numOldRecords
      • numRecordsToKeep

        protected int numRecordsToKeep
      • maxNumLogsToKeep

        protected int maxNumLogsToKeep
      • numVersionBuckets

        protected int numVersionBuckets
      • maxVersionFromIndex

        protected Long maxVersionFromIndex
      • existOldBufferLog

        protected boolean existOldBufferLog
      • tlogFiles

        protected String[] tlogFiles
      • tlogDir

        protected File tlogDir
      • dataDir

        protected String dataDir
      • lastDataDir

        protected String lastDataDir
      • cancelApplyBufferUpdate

        protected volatile boolean cancelApplyBufferUpdate
      • bufferedOpsGauge

        protected com.codahale.metrics.Gauge<Integer> bufferedOpsGauge
      • applyingBufferedOpsMeter

        protected com.codahale.metrics.Meter applyingBufferedOpsMeter
      • replayOpsMeter

        protected com.codahale.metrics.Meter replayOpsMeter
      • copyOverOldUpdatesMeter

        protected com.codahale.metrics.Meter copyOverOldUpdatesMeter
      • registryName

        protected String registryName
      • testing_logReplayHook

        public static Runnable testing_logReplayHook
      • testing_logReplayFinishHook

        public static Runnable testing_logReplayFinishHook
    • Constructor Detail

      • UpdateLog

        public UpdateLog()
    • Method Detail

      • getFs

        public org.apache.hadoop.fs.FileSystem getFs()
      • getTotalLogsSize

        public long getTotalLogsSize()
      • getCurrentLogSizeFromStream

        public long getCurrentLogSizeFromStream()
        Returns:
        the current transaction log's size (based on its output stream)
      • getTotalLogsNumber

        public long getTotalLogsNumber()
      • getNumRecordsToKeep

        public int getNumRecordsToKeep()
      • getMaxNumLogsToKeep

        public int getMaxNumLogsToKeep()
      • getNumVersionBuckets

        public int getNumVersionBuckets()
      • objToInt

        protected static int objToInt​(Object obj,
                                      int def)
      • newTransactionLog

        public TransactionLog newTransactionLog​(File tlogFile,
                                                Collection<String> globalStrings,
                                                boolean openExisting)
        Returns a new TransactionLog. Sub-classes can override this method to change the implementation of the transaction log.
      • getLogDir

        public String getLogDir()
      • getStartingVersions

        public List<Long> getStartingVersions()
      • existOldBufferLog

        public boolean existOldBufferLog()
      • addOldLog

        protected void addOldLog​(TransactionLog oldLog,
                                 boolean removeOld)
      • getBufferLogList

        public String[] getBufferLogList​(File directory)
      • getLogList

        public String[] getLogList​(File directory)
      • getLastLogId

        public long getLastLogId()
      • openRealtimeSearcher

        public void openRealtimeSearcher()
        Opens a new realtime searcher and clears the id caches. This may also be called when we updates are being buffered (from PeerSync/IndexFingerprint)
      • deleteAll

        public void deleteAll()
        currently for testing only
      • newMap

        protected void newMap()
      • hasUncommittedChanges

        public boolean hasUncommittedChanges()
      • applyPartialUpdates

        public long applyPartialUpdates​(org.apache.lucene.util.BytesRef id,
                                        long prevPointer,
                                        long prevVersion,
                                        Set<String> onlyTheseFields,
                                        SolrDocumentBase latestPartialDoc)
        Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full document, or if there are no previous entries to follow in the update log.
        Parameters:
        id - Binary representation of the unique key field
        prevPointer - Pointer to the previous entry in the ulog, based on which the current in-place update was made.
        prevVersion - Version of the previous entry in the ulog, based on which the current in-place update was made.
        onlyTheseFields - When a non-null set of field names is passed in, the resolve process only attempts to populate the given fields in this set. When this set is null, it resolves all fields.
        latestPartialDoc - Partial document that is to be populated
        Returns:
        Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
      • lookup

        public Object lookup​(org.apache.lucene.util.BytesRef indexedId)
      • lookupVersion

        public Long lookupVersion​(org.apache.lucene.util.BytesRef indexedId)
      • recoverFromCurrentLog

        public Future<UpdateLog.RecoveryInfo> recoverFromCurrentLog()
        Replay current tlog, so all updates will be written to index. This is must do task for a tlog replica become a new leader.
        Returns:
        future of this task
      • copyOverBufferingUpdates

        public void copyOverBufferingUpdates​(CommitUpdateCommand cuc)
        Block updates, append a commit at current tlog, then copy over buffer updates to new tlog and bring back ulog to active state. So any updates which hasn't made it to the index is preserved in the current tlog, this also make RTG work
        Parameters:
        cuc - any updates that have version larger than the version of cuc will be copied over
      • commitAndSwitchToNewTlog

        public void commitAndSwitchToNewTlog​(CommitUpdateCommand cuc)
        Block updates, append a commit at current tlog, then copy over updates to a new tlog. So any updates which hasn't made it to the index is preserved in the current tlog
        Parameters:
        cuc - any updates that have version larger than the version of cuc will be copied over
      • copyOverOldUpdates

        public void copyOverOldUpdates​(long commitVersion)
      • copyOverOldUpdates

        public void copyOverOldUpdates​(long commitVersion,
                                       TransactionLog oldTlog)
        Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog
        Parameters:
        commitVersion - any updates that have version larger than the commitVersion will be copied over
      • ensureBufferTlog

        protected void ensureBufferTlog()
      • deleteBufferLogs

        protected void deleteBufferLogs()
      • ensureLog

        protected void ensureLog()
      • close

        public void close​(boolean committed)
      • close

        public void close​(boolean committed,
                          boolean deleteOnClose)
      • getRecentUpdates

        public UpdateLog.RecentUpdates getRecentUpdates()
        The RecentUpdates object returned must be closed after use
      • bufferUpdates

        public void bufferUpdates()
      • dropBufferedUpdates

        public boolean dropBufferedUpdates()
        Returns true if we were able to drop buffered updates and return to the ACTIVE state
      • applyBufferedUpdates

        public Future<UpdateLog.RecoveryInfo> applyBufferedUpdates()
        Returns the Future to wait on, or null if no replay was needed
      • convertTlogEntryToAddUpdateCommand

        public static AddUpdateCommand convertTlogEntryToAddUpdateCommand​(SolrQueryRequest req,
                                                                          List entry,
                                                                          int operation,
                                                                          long version)
        Given a entry from the transaction log containing a document, return a new AddUpdateCommand that can be applied to ADD the document or do an UPDATE_INPLACE.
        Parameters:
        req - The request to use as the owner of the new AddUpdateCommand
        entry - Entry from the transaction log that contains the document to be added
        operation - The value of the operation flag; this must be either ADD or UPDATE_INPLACE -- if it is UPDATE_INPLACE then the previous version will also be read from the entry
        version - Version already obtained from the entry.
      • deleteFile

        public static void deleteFile​(File file)
      • clearLog

        public void clearLog​(SolrCore core,
                             PluginInfo ulogPluginInfo)
        Clears the logs on the file system. Only call before init.
        Parameters:
        core - the SolrCore
        ulogPluginInfo - the init info for the UpdateHandler
      • getCurrentMaxVersion

        public Long getCurrentMaxVersion()
      • seedBucketsWithHighestVersion

        protected Long seedBucketsWithHighestVersion​(SolrIndexSearcher newSearcher,
                                                     VersionInfo versions)
        Used to seed all version buckets with the max value of the version field in the index.
      • seedBucketsWithHighestVersion

        public void seedBucketsWithHighestVersion​(SolrIndexSearcher newSearcher)