Merge pull request #261 from j-griffith/default_multiwrite_blockmode

Default multiwrite blockmode
This commit is contained in:
Huamin Chen 2019-03-19 21:05:31 -04:00 committed by GitHub
commit fa54e5ca23
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 94 additions and 178 deletions

View File

@ -58,21 +58,6 @@ Parameter | Required | Description
`csi.storage.k8s.io/provisioner-secret-name`, `csi.storage.k8s.io/node-publish-secret-name` | for Kubernetes | name of the Kubernetes Secret object containing Ceph client credentials. Both parameters should have the same value `csi.storage.k8s.io/provisioner-secret-name`, `csi.storage.k8s.io/node-publish-secret-name` | for Kubernetes | name of the Kubernetes Secret object containing Ceph client credentials. Both parameters should have the same value
`csi.storage.k8s.io/provisioner-secret-namespace`, `csi.storage.k8s.io/node-publish-secret-namespace` | for Kubernetes | namespaces of the above Secret objects `csi.storage.k8s.io/provisioner-secret-namespace`, `csi.storage.k8s.io/node-publish-secret-namespace` | for Kubernetes | namespaces of the above Secret objects
`mounter`| no | if set to `rbd-nbd`, use `rbd-nbd` on nodes that have `rbd-nbd` and `nbd` kernel modules to map rbd images `mounter`| no | if set to `rbd-nbd`, use `rbd-nbd` on nodes that have `rbd-nbd` and `nbd` kernel modules to map rbd images
`fsType` | no | allows setting to `ext3 | ext-4 | xfs`, default is `ext-4`
`multiNodeWritable` | no | if set to `enabled` allows RBD volumes with MultiNode Access Modes to bypass watcher checks. By default multiple attachments of an RBD volume are NOT allowed. Even if this option is set in the StorageClass, it's ignored if a standard SingleNodeWriter Access Mode is requested
**Warning for multiNodeWritable:**
*NOTE* the `multiNodeWritable` setting is NOT safe for use by workloads
that are not designed to coordinate access. This does NOT add any sort
of a clustered filesystem or write syncronization, it's specifically for
special workloads that handle access coordination on their own
(ie Active/Passive scenarios).
Using this mode for general purposes *WILL RESULT IN DATA CORRUPTION*.
We attempt to limit exposure to trouble here but ignoring the Storage Class
setting unless your Volume explicitly asks for multi node access, and assume
you know what you're doing.
**Required secrets:** **Required secrets:**

View File

@ -115,52 +115,30 @@ kubectl create -f pvc-restore.yaml
kubectl create -f pod-restore.yaml kubectl create -f pod-restore.yaml
``` ```
## How to enable multi node attach support for RBD ## How to test RBD MULTI_NODE_MULTI_WRITER BLOCK feature
Requires feature-gates: `BlockVolume=true` `CSIBlockVolume=true`
*NOTE* The MULTI_NODE_MULTI_WRITER capability is only available for
Volumes that are of access_type `block`
*WARNING* This feature is strictly for workloads that know how to deal *WARNING* This feature is strictly for workloads that know how to deal
with concurrent acces to the Volume (eg Active/Passive applications). with concurrent access to the Volume (eg Active/Passive applications).
Using RWX modes on non clustered file systems with applications trying Using RWX modes on non clustered file systems with applications trying
to simultaneously access the Volume will likely result in data corruption! to simultaneously access the Volume will likely result in data corruption!
### Example process to test the multiNodeWritable feature Following are examples for issuing a request for a `Block`
`ReadWriteMany` Claim, and using the resultant Claim for a POD
Modify your current storage class, or create a new storage class specifically
for multi node writers by adding the `multiNodeWritable: "enabled"` entry to
your parameters. Here's an example:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd
provisioner: rbd.csi.ceph.com
parameters:
monitors: rook-ceph-mon-b.rook-ceph.svc.cluster.local:6789
pool: rbd
imageFormat: "2"
imageFeatures: layering
csiProvisionerSecretName: csi-rbd-secret
csiProvisionerSecretNamespace: default
csiNodePublishSecretName: csi-rbd-secret
csiNodePublishSecretNamespace: default
adminid: admin
userid: admin
fsType: xfs
multiNodeWritable: "enabled"
reclaimPolicy: Delete
```
Now, you can request Claims from the configured storage class that include
the `ReadWriteMany` access mode:
```yaml ```yaml
apiVersion: v1 apiVersion: v1
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
metadata: metadata:
name: pvc-1 name: block-pvc
spec: spec:
accessModes: accessModes:
- ReadWriteMany - ReadWriteMany
volumeMode: Block
resources: resources:
requests: requests:
storage: 1Gi storage: 1Gi
@ -173,108 +151,65 @@ Create a POD that uses this PVC:
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
metadata: metadata:
name: test-1 name: my-pod
spec: spec:
containers: containers:
- name: web-server - name: my-container
image: nginx image: debian
volumeMounts: command: ["/bin/bash", "-c"]
- name: mypvc args: [ "tail -f /dev/null" ]
mountPath: /var/lib/www/html volumeDevices:
- devicePath: /dev/rbdblock
name: my-volume
imagePullPolicy: IfNotPresent
volumes: volumes:
- name: mypvc - name: my-volume
persistentVolumeClaim: persistentVolumeClaim:
claimName: pvc-1 claimName: block-pvc
readOnly: false
```
Wait for the POD to enter Running state, write some data to ```
`/var/lib/www/html`
Now, we can create a second POD (ensure the POD is scheduled on a different Now, we can create a second POD (ensure the POD is scheduled on a different
node; multiwriter single node works without this feature) that also uses this node; multiwriter single node works without this feature) that also uses this
PVC at the same time PVC at the same time, again wait for the pod to enter running state, and verify
the block device is available.
```yaml ```yaml
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
metadata: metadata:
name: test-2 name: another-pod
spec: spec:
containers: containers:
- name: web-server - name: my-container
image: nginx image: debian
volumeMounts: command: ["/bin/bash", "-c"]
- name: mypvc
mountPath: /var/lib/www/html
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: pvc-1
readOnly: false
```
If you access the pod you can check that your data is avaialable at
`/var/lib/www/html`
## Testing Raw Block feature in kubernetes with RBD volumes
CSI block volume support is feature-gated and turned off by default. To run CSI
with block volume support enabled, a cluster administrator must enable the
feature for each Kubernetes component using the following feature gate flags:
--feature-gates=BlockVolume=true,CSIBlockVolume=true
these feature-gates must be enabled on both api-server and kubelet
### create a raw-block PVC
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: raw-block-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 1Gi
storageClassName: csi-rbd
```
create raw block pvc
```console
kubectl create -f raw-block-pvc.yaml
```
### create a pod to mount raw-block PVC
```yaml
---
apiVersion: v1
kind: Pod
metadata:
name: pod-with-raw-block-volume
spec:
containers:
- name: fc-container
image: fedora:26
command: ["/bin/sh", "-c"]
args: [ "tail -f /dev/null" ] args: [ "tail -f /dev/null" ]
volumeDevices: volumeDevices:
- name: data - devicePath: /dev/rbdblock
devicePath: /dev/xvda name: my-volume
imagePullPolicy: IfNotPresent
volumes: volumes:
- name: data - name: my-volume
persistentVolumeClaim: persistentVolumeClaim:
claimName: raw-block-pvc claimName: block-pvc
``` ```
Create a POD that uses raw block PVC Wait for the PODs to enter Running state, check that our block device
is available in the container at `/dev/rdbblock` in both containers:
```console ```bash
kubectl create -f raw-block-pod.yaml $ kubectl exec -it my-pod -- fdisk -l /dev/rbdblock
Disk /dev/rbdblock: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
```
```bash
$ kubectl exec -it another-pod -- fdisk -l /dev/rbdblock
Disk /dev/rbdblock: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
``` ```

View File

@ -35,7 +35,4 @@ parameters:
userid: kubernetes userid: kubernetes
# uncomment the following to use rbd-nbd as mounter on supported nodes # uncomment the following to use rbd-nbd as mounter on supported nodes
# mounter: rbd-nbd # mounter: rbd-nbd
# fsType: xfs
# uncomment the following line to enable multi-attach on RBD volumes
# multiNodeWritable: enabled
reclaimPolicy: Delete reclaimPolicy: Delete

View File

@ -21,7 +21,6 @@ import (
"os/exec" "os/exec"
"sort" "sort"
"strconv" "strconv"
"strings"
"syscall" "syscall"
csicommon "github.com/ceph/ceph-csi/pkg/csi-common" csicommon "github.com/ceph/ceph-csi/pkg/csi-common"
@ -94,16 +93,25 @@ func (cs *ControllerServer) validateVolumeReq(req *csi.CreateVolumeRequest) erro
func parseVolCreateRequest(req *csi.CreateVolumeRequest) (*rbdVolume, error) { func parseVolCreateRequest(req *csi.CreateVolumeRequest) (*rbdVolume, error) {
// TODO (sbezverk) Last check for not exceeding total storage capacity // TODO (sbezverk) Last check for not exceeding total storage capacity
// MultiNodeWriters are accepted but they're only for special cases, and we skip the watcher checks for them which isn't the greatest isMultiNode := false
// let's make sure we ONLY skip that if the user is requesting a MULTI Node accessible mode isBlock := false
disableMultiWriter := true for _, cap := range req.VolumeCapabilities {
for _, am := range req.VolumeCapabilities { // RO modes need to be handled indepedently (ie right now even if access mode is RO, they'll be RW upon attach)
if am.GetAccessMode().GetMode() != csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER { if cap.GetAccessMode().GetMode() == csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER {
disableMultiWriter = false isMultiNode = true
}
if cap.GetBlock() != nil {
isBlock = true
} }
} }
rbdVol, err := getRBDVolumeOptions(req.GetParameters(), disableMultiWriter) // We want to fail early if the user is trying to create a RWX on a non-block type device
if isMultiNode && !isBlock {
return nil, status.Error(codes.InvalidArgument, "multi node access modes are only supported on rbd `block` type volumes")
}
// if it's NOT SINGLE_NODE_WRITER and it's BLOCK we'll set the parameter to ignore the in-use checks
rbdVol, err := getRBDVolumeOptions(req.GetParameters(), (isMultiNode && isBlock))
if err != nil { if err != nil {
return nil, status.Error(codes.InvalidArgument, err.Error()) return nil, status.Error(codes.InvalidArgument, err.Error())
} }
@ -344,20 +352,11 @@ func (cs *ControllerServer) ListVolumes(ctx context.Context, req *csi.ListVolume
// ValidateVolumeCapabilities checks whether the volume capabilities requested // ValidateVolumeCapabilities checks whether the volume capabilities requested
// are supported. // are supported.
func (cs *ControllerServer) ValidateVolumeCapabilities(ctx context.Context, req *csi.ValidateVolumeCapabilitiesRequest) (*csi.ValidateVolumeCapabilitiesResponse, error) { func (cs *ControllerServer) ValidateVolumeCapabilities(ctx context.Context, req *csi.ValidateVolumeCapabilitiesRequest) (*csi.ValidateVolumeCapabilitiesResponse, error) {
params := req.GetParameters() for _, cap := range req.VolumeCapabilities {
multiWriter := params["multiNodeWritable"] if cap.GetAccessMode().GetMode() != csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER {
if strings.ToLower(multiWriter) == "enabled" { return &csi.ValidateVolumeCapabilitiesResponse{Message: ""}, nil
klog.V(3).Info("detected multiNodeWritable parameter in Storage Class, allowing multi-node access modes")
} else {
for _, cap := range req.VolumeCapabilities {
if cap.GetAccessMode().GetMode() != csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER {
return &csi.ValidateVolumeCapabilitiesResponse{Message: ""}, nil
}
} }
} }
return &csi.ValidateVolumeCapabilitiesResponse{ return &csi.ValidateVolumeCapabilitiesResponse{
Confirmed: &csi.ValidateVolumeCapabilitiesResponse_Confirmed{ Confirmed: &csi.ValidateVolumeCapabilitiesResponse_Confirmed{
VolumeCapabilities: req.VolumeCapabilities, VolumeCapabilities: req.VolumeCapabilities,

View File

@ -48,6 +48,7 @@ type NodeServer struct {
func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (*csi.NodePublishVolumeResponse, error) { func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (*csi.NodePublishVolumeResponse, error) {
targetPath := req.GetTargetPath() targetPath := req.GetTargetPath()
targetPathMutex.LockKey(targetPath) targetPathMutex.LockKey(targetPath)
disableInUseChecks := false
defer func() { defer func() {
if err := targetPathMutex.UnlockKey(targetPath); err != nil { if err := targetPathMutex.UnlockKey(targetPath); err != nil {
@ -71,18 +72,20 @@ func (ns *NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublis
return &csi.NodePublishVolumeResponse{}, nil return &csi.NodePublishVolumeResponse{}, nil
} }
// if our access mode is a simple SINGLE_NODE_WRITER we're going to ignore the SC directive and use the // MULTI_NODE_MULTI_WRITER is supported by default for Block access type volumes
// watcher still if req.VolumeCapability.AccessMode.Mode == csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER {
ignoreMultiWriterEnabled := true if isBlock {
if req.VolumeCapability.AccessMode.Mode != csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER { disableInUseChecks = true
ignoreMultiWriterEnabled = false } else {
klog.Warningf("MULTI_NODE_MULTI_WRITER currently only supported with volumes of access type `block`, invalid AccessMode for volume: %v", req.GetVolumeId())
return nil, status.Error(codes.InvalidArgument, "rbd: RWX access mode request is only valid for volumes with access type `block`")
}
} }
volOptions, err := getRBDVolumeOptions(req.GetVolumeContext(), ignoreMultiWriterEnabled) volOptions, err := getRBDVolumeOptions(req.GetVolumeContext(), disableInUseChecks)
if err != nil { if err != nil {
return nil, err return nil, err
} }
volOptions.VolName = volName volOptions.VolName = volName
// Mapping RBD image // Mapping RBD image
devicePath, err := attachRBDImage(volOptions, volOptions.UserID, req.GetSecrets()) devicePath, err := attachRBDImage(volOptions, volOptions.UserID, req.GetSecrets())

View File

@ -105,10 +105,12 @@ func (r *Driver) Run(driverName, nodeID, endpoint string, containerized bool, ca
csi.ControllerServiceCapability_RPC_CLONE_VOLUME, csi.ControllerServiceCapability_RPC_CLONE_VOLUME,
}) })
// TODO: JDG Should also look at remaining modes like MULT_NODE_READER (SINGLE_READER) // We only support the multi-writer option when using block, but it's a supported capability for the plugin in general
// In addition, we want to add the remaining modes like MULTI_NODE_READER_ONLY,
// MULTI_NODE_SINGLE_WRITER etc, but need to do some verification of RO modes first
// will work those as follow up features
r.cd.AddVolumeCapabilityAccessModes( r.cd.AddVolumeCapabilityAccessModes(
[]csi.VolumeCapability_AccessMode_Mode{ []csi.VolumeCapability_AccessMode_Mode{csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER}) csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER})
// Create GRPC servers // Create GRPC servers

View File

@ -258,6 +258,7 @@ func attachRBDImage(volOptions *rbdVolume, userID string, credentials map[string
Factor: rbdImageWatcherFactor, Factor: rbdImageWatcherFactor,
Steps: rbdImageWatcherSteps, Steps: rbdImageWatcherSteps,
} }
err = waitForrbdImage(backoff, volOptions, userID, credentials) err = waitForrbdImage(backoff, volOptions, userID, credentials)
if err != nil { if err != nil {
@ -313,16 +314,12 @@ func waitForrbdImage(backoff wait.Backoff, volOptions *rbdVolume, userID string,
if err != nil { if err != nil {
return false, fmt.Errorf("fail to check rbd image status with: (%v), rbd output: (%s)", err, rbdOutput) return false, fmt.Errorf("fail to check rbd image status with: (%v), rbd output: (%s)", err, rbdOutput)
} }
// In the case of multiattach we want to short circuit the retries when used (so r`if used; return used`) if (volOptions.DisableInUseChecks) && (used) {
// otherwise we're setting this to false which translates to !ok, which means backoff and try again klog.V(2).Info("valid multi-node attach requested, ignoring watcher in-use result")
// NOTE: we ONLY do this if an multi-node access mode is requested for this volume
if (strings.ToLower(volOptions.MultiNodeWritable) == "enabled") && (used) {
klog.V(2).Info("detected MultiNodeWritable enabled, ignoring watcher in-use result")
return used, nil return used, nil
} }
return !used, nil return !used, nil
}) })
// return error if rbd image has not become available for the specified timeout // return error if rbd image has not become available for the specified timeout
if err == wait.ErrWaitTimeout { if err == wait.ErrWaitTimeout {
return fmt.Errorf("rbd image %s is still being used", imagePath) return fmt.Errorf("rbd image %s is still being used", imagePath)

View File

@ -51,7 +51,7 @@ type rbdVolume struct {
AdminID string `json:"adminId"` AdminID string `json:"adminId"`
UserID string `json:"userId"` UserID string `json:"userId"`
Mounter string `json:"mounter"` Mounter string `json:"mounter"`
MultiNodeWritable string `json:"multiNodeWritable"` DisableInUseChecks bool `json:"disableInUseChecks"`
} }
type rbdSnapshot struct { type rbdSnapshot struct {
@ -227,7 +227,7 @@ func execCommand(command string, args []string) ([]byte, error) {
return cmd.CombinedOutput() return cmd.CombinedOutput()
} }
func getRBDVolumeOptions(volOptions map[string]string, ignoreMultiNodeWritable bool) (*rbdVolume, error) { func getRBDVolumeOptions(volOptions map[string]string, disableInUseChecks bool) (*rbdVolume, error) {
var ok bool var ok bool
rbdVol := &rbdVolume{} rbdVol := &rbdVolume{}
rbdVol.Pool, ok = volOptions["pool"] rbdVol.Pool, ok = volOptions["pool"]
@ -260,13 +260,11 @@ func getRBDVolumeOptions(volOptions map[string]string, ignoreMultiNodeWritable b
} }
} }
getCredsFromVol(rbdVol, volOptions)
klog.V(3).Infof("ignoreMultiNodeWritable flag in parse getRBDVolumeOptions is: %v", ignoreMultiNodeWritable) klog.V(3).Infof("setting disableInUseChecks on rbd volume to: %v", disableInUseChecks)
// If the volume we're working with is NOT requesting multi-node attach then don't treat it special, ignore the setting in the SC and just keep our watcher checks rbdVol.DisableInUseChecks = disableInUseChecks
if !ignoreMultiNodeWritable {
rbdVol.MultiNodeWritable = volOptions["multiNodeWritable"] getCredsFromVol(rbdVol, volOptions)
}
return rbdVol, nil return rbdVol, nil
} }