ceph-csi/docs/disaster-recovery.md

Failover and Failback In Disaster Recovery

RBD mirroring is an asynchronous replication of RBD images between multiple Ceph clusters. This capability is available in two modes:

• Journal-based: Every write to the RBD image is first recorded to the associated journal before modifying the actual image. The remote cluster will read from this associated journal and replay the updates to its local image.
• Snapshot-based: This mode uses periodically scheduled or manually created RBD image mirror-snapshots to replicate crash-consistent RBD images between clusters.

This documentation assumes that rbd mirroring is set up between two clusters. For more information on how to set up rbd mirroring, refer to ceph documentation.

Deploy the Volume Replication CRD

Volume Replication Operator is a kubernetes operator that provides common and reusable APIs for storage disaster recovery. It is based on csi-addons/spec specification and can be used by any storage provider.

Volume Replication Operator follows controller pattern and provides extended APIs for storage disaster recovery. The extended APIs are provided via Custom Resource Definition (CRD).

💡 For more information, please refer to the volume-replication-operator.

• Deploy the VolumeReplicationClass CRD

    kubectl create -f https://raw.githubusercontent.com/csi-addons/volume-replication-operator/release-v0.1/config/crd/bases/replication.storage.openshift.io_volumereplicationclasses.yaml

 customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created

• Deploy the VolumeReplication CRD

   kubectl create -f https://raw.githubusercontent.com/csi-addons/volume-replication-operator/release-v0.1/config/crd/bases/replication.storage.openshift.io_volumereplications.yaml

 customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created created

The VolumeReplicationClass and VolumeReplication CRDs are now created.

💡 Note: Use the latest available release for Volume Replication Operator. See releases for more information.

Add RBAC rules for Volume Replication Operator

Add the below mentioned rules to rbd-external-provisioner-runner ClusterRole in csi-provisioner-rbac.yaml

  - apiGroups: ["replication.storage.openshift.io"]
    resources: ["volumereplications", "volumereplicationclasses"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["replication.storage.openshift.io"]
    resources: ["volumereplications/finalizers"]
    verbs: ["update"]
  - apiGroups: ["replication.storage.openshift.io"]
    resources: ["volumereplications/status"]
    verbs: ["get", "patch", "update"]
  - apiGroups: ["replication.storage.openshift.io"]
    resources: ["volumereplicationclasses/status"]
    verbs: ["get"]

Deploy the Volume Replication Sidecar

To deploy volume-replication sidecar container in csi-rbdplugin-provisioner pod, add the following yaml to csi-rbdplugin-provisioner deployment.

        - name: volume-replication
          image: quay.io/csiaddons/volumereplication-operator:v0.1.0
          args :
            - "--metrics-bind-address=0"
            - "--leader-election-namespace=$(NAMESPACE)"
            - "--driver-name=rbd.csi.ceph.com"
            - "--csi-address=$(ADDRESS)"
            - "--rpc-timeout=150s"
            - "--health-probe-bind-address=:9998"
            - "--leader-elect=true"
          env:
            - name: ADDRESS
              value: unix:///csi/csi-provisioner.sock
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /csi

VolumeReplicationClass and VolumeReplication

VolumeReplicationClass

VolumeReplicationClass is a cluster scoped resource that contains driver related configuration parameters. It holds the storage admin information required for the volume replication operator.

VolumeReplication

VolumeReplication is a namespaced resource that contains references to storage object to be replicated and VolumeReplicationClass corresponding to the driver providing replication.

💡 For more information, please refer to the volume-replication-operator.

Let's say we have a PVC (rbd-pvc) in BOUND state; created using StorageClass with Retain reclaimPolicy.

kubectl get pvc --context=cluster-1

 NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
 rbd-pvc   Bound    pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec   1Gi        RWO            csi-rbd-sc   44s

• Create Volume Replication Class on cluster-1

  $cat <<EOF | kubectl --context=cluster1 apply -f -
  apiVersion: replication.storage.openshift.io/v1alpha1
  kind: VolumeReplicationClass
  metadata:
    name: rbd-volumereplicationclass
  spec:
    provisioner: rbd.csi.ceph.com
    parameters:
      mirroringMode: snapshot
      schedulingInterval: "12m"
      schedulingStartTime: "16:18:43"
      replication.storage.openshift.io/replication-secret-name: csi-rbd-secret
      replication.storage.openshift.io/replication-secret-namespace: default
  EOF
  

💡 Note: The schedulingInterval can be specified in formats of minutes, hours or days using suffix m,h and d respectively. The optional schedulingStartTime can be specified using the ISO 8601 time format.

• Once VolumeReplicationClass is created,create a Volume Replication for the PVC which we intend to replicate to secondary cluster.

  $cat <<EOF | kubectl --context=cluster-1 apply -f -
  apiVersion: replication.storage.openshift.io/v1alpha1
  kind: VolumeReplication
  metadata:
    name: pvc-volumereplication
  spec:
    volumeReplicationClass: rbd-volumereplicationclass
    replicationState: primary
    dataSource:
      apiGroup: ""
      kind: PersistentVolumeClaim
      name: rbd-pvc # Name of the PVC to which mirroring to be enabled.
  EOF
  

📝 VolumeReplication is a namespace scoped object. Thus, it should be created in the same namespace as of PVC.

replicationState is the state of the volume being referenced. Possible values are primary, secondary, and resync.

• primary denotes that the volume is primary.
• secondary denotes that the volume is secondary.
• resync denotes that the volume needs to be resynced.

To check VolumeReplication CR status:

kubectl get volumereplication pvc-volumereplication  --context=cluster-1 -oyaml

...
spec:
 dataSource:
   apiGroup: ""
   kind: PersistentVolumeClaim
   name: rbd-pvc
 replicationState: primary
 volumeReplicationClass: rbd-volumereplicationclass
status:
 conditions:
 - lastTransitionTime: "2021-05-04T07:39:00Z"
   message: ""
   observedGeneration: 1
   reason: Promoted
   status: "True"
   type: Completed
 - lastTransitionTime: "2021-05-04T07:39:00Z"
   message: ""
   observedGeneration: 1
   reason: Healthy
   status: "False"
   type: Degraded
 - lastTransitionTime: "2021-05-04T07:39:00Z"
   message: ""
   observedGeneration: 1
   reason: NotResyncing
   status: "False"
   type: Resyncing
 lastCompletionTime: "2021-05-04T07:39:00Z"
 lastStartTime: "2021-05-04T07:38:59Z"
 message: volume is marked primary
 observedGeneration: 1
 state: Primary

• Take a backup of PVC and PV object on primary cluster(cluster-1)

  • Take backup of the PVC rbd-pvc

    kubectl get pvc rbd-pvc -oyaml >pvc-backup.yaml
    

  • Take a backup of the PV, corresponding to the PVC

    kubectl get pv/pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec -oyaml >pv_backup.yaml
    

💡 We can also take backup using external tools like Velero. Refer velero documentation for more information.

• Restoring on the secondary cluster(cluster-2)

  • Create storageclass on the secondary cluster

    kubectl create -f examples/rbd/storageclass.yaml --context=cluster-2
  
   storageclass.storage.k8s.io/csi-rbd-sc created
  

  • Create VolumeReplicationClass on the secondary cluster

        cat <<EOF | kubectl --context=cluster-2 apply -f -
        apiVersion: replication.storage.openshift.io/v1alpha1
        kind: VolumeReplicationClass
        metadata:
          name: rbd-volumereplicationclass
        spec:
          provisioner: rbd.csi.ceph.com
          parameters:
            mirroringMode: snapshot
            replication.storage.openshift.io/replication-secret-name: csi-rbd-secret
            replication.storage.openshift.io/replication-secret-namespace: default
        EOF
  
  volumereplicationclass.replication.storage.openshift.io/rbd-volumereplicationclass created
  

  • If Persistent Volumes and Claims are created manually on the secondary cluster, remove the claimRef on the backed up PV objects in yaml files; so that the PV can get bound to the new claim on the secondary cluster.

    ...
    spec:
        accessModes:
        - ReadWriteOnce
        capacity:
          storage: 1Gi
        claimRef:
          apiVersion: v1
          kind: PersistentVolumeClaim
          name: rbd-pvc
          namespace: default
          resourceVersion: "64252"
          uid: 65dc0aac-5e15-4474-90f4-7a3532c621ec
        csi:
    ...
  

• Apply the Persistent Volume backup from the primary cluster

    kubectl create -f pv-backup.yaml --context=cluster-2

 persistentvolume/pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec created

• Apply the Persistent Volume claim from the restored backup

    kubectl create -f pvc-backup.yaml --context=cluster-2

 persistentvolumeclaim/rbd-pvc created

  kubectl get pvc --context=cluster-2

 NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
 rbd-pvc   Bound    pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec   1Gi        RWO            csi-rbd-sc   44s

Planned Migration

Use cases: Datacenter maintenance, Technology refresh, Disaster avoidance, etc.

Failover

The failover operation is the process of switching production to a backup facility (normally your recovery site). In the case of Failover, access to the image on the primary site should be stopped. The image should now be made primary on the secondary cluster so that the access can be resumed there.

📝 As mentioned in the pre-requisites, periodic or one time backup of the application should be available for restore on the secondary site (cluster-b).

Follow the below steps for planned migration of workload from primary cluster to secondary cluster:

• Scale down all the application pods which are using the mirrored PVC on the Primary Cluster
• Take a back up of PVC and PV object from the primary cluster. This can be done using some backup tools like velero.
• Update replicationState to secondary in VolumeReplication CR at Primary Site. When the operator sees this change, it will pass the information down to the driver via GRPC request to mark the dataSource as secondary.
• If you are manually recreating the PVC and PV on the secondary cluster, remove the claimRef section in the PV objects.
• Recreate the storageclass, PVC, and PV objects on the secondary site.
• As you are creating the static binding between PVC and PV, a new PV won’t be created here, the PVC will get bind to the existing PV.
• Create the VolumeReplicationClass on the secondary site.
• Create the VolumeReplications for all the PVC’s for which mirroring is enabled
  • replicationState should be primary for all the PVC’s on the secondary site.
• Check whether the image is marked primary on the secondary site by verifying it in VolumeReplication CR status.
• Once the Image is marked as primary, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.

📝 WARNING: In Async Disaster recovery use case, we don't get the complete data. We will only get the crash-consistent data based on the snapshot interval time.

Failback

To perform a failback operation to primary cluster in case of planned migration , just repeat the Failback steps in vice-versa.

📝 Remember: We can skip the backup-restore operations in case of failback if the required yamls are already present on the primary cluster. Any new PVCs will still need to be restored on the primary site.

Disaster Recovery

Use cases: Natural disasters, Power failures, System failures, and crashes, etc.

Failover (abrupt shutdown)

In case of Disaster recovery, create VolumeReplication CR at the Secondary Site. Since the connection to the Primary Site is lost, the operator automatically sends a GRPC request down to the driver to forcefully mark the dataSource as primary.

• If you are manually creating the PVC and PV on the secondary cluster, remove the claimRef section in the PV objects.
• Create the storageclass, PVC, and PV objects on the secondary site.
• As you are creating the static binding between PVC and PV, a new PV won’t be created here, the PVC will get bind to the existing PV.
• Create the VolumeReplicationClass and VolumeReplication CR on the secondary site.
• Check whether the image is primary on secondary site, by verifying in the VolumeReplication CR status.
• Once the Image is marked as primary, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.

Failback (post-disaster recovery)

Once the failed cluster is recovered on the primary site and you want to failback from secondary site, follow the below steps:

• Update the VolumeReplication CR replicationState from primary to secondary on the primary site.
• Scale down the applications on the secondary site.
• Update the VolumeReplication CR replicationState from primary to secondary in secondary site.
• On the primary site, verify that the VolumeReplication status is marked as volume ready to use
• Once the volume is marked to ready to use, change the replicationState state from secondary to primary in primary site.
• Scale up the applications again on the primary site.