Cross Data Center Replication Operations

Monitoring

  1. Network and disk space monitoring are essential. Ensure that the system has plenty of available storage to queue up changes if there is a disconnect between the Source and Target. A network outage between the two data centers can cause your disk usage to grow. Some tips:
    1. Set a monitor for your disks to send alerts when the disk gets over a certain percentage (e.g., 70%).
    2. Run a test. With moderate indexing, how long can the system queue changes before you run out of disk space?
  2. Create a simple way to check the counts between the Source and the Target.
    1. Keep in mind that if indexing is running, the Source and Target may not match document for document. Set an alert to fire if the difference is greater than some percentage of the overall cloud size.

Upgrading and Patching Production

When rolling in upgrades to your indexer or application, you should shutdown the Source and the Target. Depending on your setup, you may want to pause/stop indexing, deploy the release or patch, then re-enable indexing. Then start the Target last.

  • There is no need to reissue the DISABLEBUFFERS or START commands. These are persisted.
  • After starting the Target, run a simple test. Add a test document to each of the Source clouds. Then check for it on the Target.
#send to the Source
curl http://<Source>/solr/cloud1/update -H 'Content-type:application/json' -d '[{"SKU":"ABC"}]'

#check the Target
curl "http://<Target>:8983/solr/<collection_name>/select?q=SKU:ABC&indent=true"