rbd: Report errors when a resync maybe in progress

Currently we return a !ready status if an image
is not found when a replication resync is issued.

We also return a !ready just post issuing a resync.

The change is to ensure we return errors in these
cases for the caller to retry the operation till
we can determine we are actually resyncing, and then
return !ready with nil errors.

Part of addressing:
  https://github.com/csi-addons/volume-replication-operator/issues/101

Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
(cherry picked from commit 47dc9cf28d)

internal/rbd/replicationcontrollerserver.go

@@ -599,14 +599,13 @@ func (rs *ReplicationServer) ResyncVolume(ctx context.Context,
 
 	mirrorStatus, err := rbdVol.getImageMirroringStatus()
 	if err != nil {
-		// the image gets recreated after issuing resync in that case return
+		// the image gets recreated after issuing resync
-		// volume as not ready.
 		if errors.Is(err, ErrImageNotFound) {
-			resp := &replication.ResyncVolumeResponse{
+			// caller retries till RBD syncs an initial version of the image to
-				Ready: false,
+			// report its status in the resync call. Ideally, this line will not
-			}
+			// be executed as the error would get returned due to getImageMirroringInfo
-
+			// failing to find an image above.
-			return resp, nil
+			return nil, status.Error(codes.Aborted, err.Error())
 		}
 		util.ErrorLog(ctx, err.Error())
 
@@ -643,6 +642,11 @@ func (rs *ReplicationServer) ResyncVolume(ctx context.Context,
 
 			return nil, status.Error(codes.Internal, err.Error())
 		}
+
+		// If we issued a resync, return a non-final error as image needs to be recreated
+		// locally. Caller retries till RBD syncs an initial version of the image to
+		// report its status in the resync request.
+		return nil, status.Error(codes.Unavailable, "awaiting initial resync due to split brain")
 	}
 
 	// convert the last update time to UTC