diff --git a/README.md b/README.md index 18cbe2cc0..e0b744820 100644 --- a/README.md +++ b/README.md @@ -2,21 +2,22 @@ ## Overview -RBD CSI plugin implements an interface between CSI enabled Container +Ceph CSI plugins implement an interface between CSI enabled Container Orchestrator and CEPH cluster. It allows dynamically provision CEPH volumes and attach it to workloads. -Current implementation of CSI RBD plugin was tested in Kubernetes environment, -but its code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic) +Current implementation of Ceph CSI plugins was tested in Kubernetes environment (requires Kubernetes 1.10+), +but the code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic) and should be able to run with any CSI enabled CO (Containers Orchestration). +[Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, and attacher for Ceph RBD and CephFS + +## RBD Plugin + An RBD CSI plugin is available to help simplify storage management. Once user creates PVC with the reference to a RBD storage class, rbd image and corresponding PV object gets dynamically created and becomes ready to be used by workloads. -[Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, and attacher for Ceph RBD and CephFS - -## RBD Plugin ### Configuration Requirements * Secret object with the authentication key for ceph cluster @@ -44,7 +45,7 @@ $ make rbdplugin To build a container: ``` -$ make container +$ make container-rbdplugin ``` By running: ``` @@ -73,29 +74,29 @@ Enable features `MountPropagation=true,CSIPersistentVolume=true` and runtime con #### Step 1: Create Secret ``` -$ kubectl create -f ./deploy/kubernetes/rbd-secrets.yaml +$ kubectl create -f ./deploy/rbd/kubernetes/rbd-secrets.yaml ``` **Important:** rbd-secrets.yaml, must be customized to match your ceph environment. #### Step 2: Create StorageClass ``` -$ kubectl create -f ./deploy/kubernetes/rbd-storage-class.yaml +$ kubectl create -f ./deploy/rbd/kubernetes/rbd-storage-class.yaml ``` **Important:** rbd-storage-class.yaml, must be customized to match your ceph environment. #### Step 3: Start CSI CEPH RBD plugin ``` -$ kubectl create -f ./deploy/kubernetes/rbdplugin.yaml +$ kubectl create -f ./deploy/rbd/kubernetes/rbdplugin.yaml ``` #### Step 4: Start CSI External Attacher ``` -$ kubectl create -f ./deploy/kubernetes/csi-attacher.yaml +$ kubectl create -f ./deploy/rbd/kubernetes/csi-attacher.yaml ``` #### Step 5: Start CSI External Provisioner ``` -$ kubectl create -f ./deploy/kubernetes/csi-provisioner.yaml +$ kubectl create -f ./deploy/rbd/kubernetes/csi-provisioner.yaml ``` **Important:** Deployment yaml files includes required Service Account definitions and required RBAC rules. @@ -116,7 +117,7 @@ default csi-provisioner-0 1/1 Runn #### Step 7: Create PVC ``` -$ kubectl create -f ./deploy/kubernetes/pvc.yaml +$ kubectl create -f ./deploy/rbd/kubernetes/pvc.yaml ``` #### Step 8: Check status of provisioner PV @@ -152,12 +153,150 @@ Source: #### Step 9: Create a test pod ```bash -# kubectl create -f ./deploy/pod.yaml +# kubectl create -f ./deploy/rbd/pod.yaml ``` ## CephFS plugin -TODO +A CephFS CSI plugin is available to help simplify storage management. +Once user creates PVC with the reference to a CephFS CSI storage class, corresponding +PV object gets dynamically created and becomes ready to be used by workloads. + +### Configuration Requirements + +* Secret object with the authentication key for ceph cluster +* StorageClass with csi-cephfsplugin (default CSI CephFS plugin name) as a provisioner name + and information about ceph cluster (mounter, monitors, user name, secret) +* CephFS root path, which is then mounted by either `ceph-fuse` or `mount.ceph` - configurable + through the StorageClass +* Service Accounts with required RBAC permissions + +### Feature Status + +### 1.10: Alpha + +**Important:** `CSIPersistentVolume` and `MountPropagation` +[feature gates must be enabled starting in 1.9](#enabling-the-alpha-feature-gates). +Also API server must run with running config set to: `storage.k8s.io/v1alpha1` + +* `kube-apiserver` must be launched with `--feature-gate=CSIPersistentVolume=true,MountPropagation=true` + and `--runtime-config=storage.k8s.io/v1alpha1=true` +* `kube-controller-manager` must be launched with `--feature-gate=CSIPersistentVolume=true` +* `kubelet` must be launched with `--feature-gate=CSIPersistentVolume=true,MountPropagation=true` + +### Compiling +CSI CephFS plugin can be compiled in a form of a binary file or in a form of a container. When compiled +as a binary file, it gets stored in \_output folder with the name cephfsplugin. When compiled as a container, +the resulting image is stored in a local docker's image store. + +To compile just a binary file: +``` +$ make cephfsplugin +``` + +To build a container: +``` +$ make cephfsplugin-container +``` +By running: +``` +$ docker images | grep cephfsplugin +``` +You should see the following line in the output: +``` +quay.io/cephcsi/cephfsplugin v0.2.0 79482e644593 4 minutes ago 305MB +``` + +### Testing + +#### Prerequisite + +##### Enable Mount Propagation in Docker + +Comment out `MountFlags=slave` in docker systemd service then restart docker service. +``` +# systemctl daemon-reload +# systemctl restart docker +``` + +##### Enable Kubernetes Feature Gates + +Enable features `MountPropagation=true,CSIPersistentVolume=true` and runtime config `storage.k8s.io/v1alpha1=true` + +#### Step 1: Create Secret +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/secret.yaml +``` +**Important:** secret.yaml, must be customized to match your ceph environment. + +#### Step 2: Create StorageClass +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/cephfs-storage-class.yaml +``` +**Important:** cephfs-storage-class.yaml, must be customized to match your ceph environment. + +#### Step 3: Start CSI CEPH CephFS plugin +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/cephfsplugin.yaml +``` + +#### Step 4: Start CSI External Attacher +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/csi-attacher.yaml +``` + +#### Step 5: Start CSI External Provisioner +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/csi-provisioner.yaml +``` +**Important:** Deployment yaml files includes required Service Account definitions and +required RBAC rules. + +#### Step 6: Check status of CSI CephFS plugin +``` +$ kubectl get pods | grep csi +csi-attacher-0 1/1 Running 0 6m +csi-cephfsplugin-hmqpk 2/2 Running 0 6m +csi-provisioner-0 1/1 Running 0 6m +``` + +#### Step 7: Create PVC +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/pvc.yaml +``` + +#### Step 8: Check status of provisioner PV +``` +$ kubectl get pv +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +kubernetes-dynamic-pv-715cef0b30d811e8 5Gi RWX Delete Bound default/csi-cephfs-pvc csi-cephfs 5s +``` + +``` +$ kubectl describe pv kubernetes-dynamic-pv-715cef0b30d811e8 +Name: kubernetes-dynamic-pv-715cef0b30d811e8 +Labels: +Annotations: pv.kubernetes.io/provisioned-by=csi-cephfsplugin +StorageClass: csi-cephfs +Status: Bound +Claim: default/csi-cephfs-pvc +Reclaim Policy: Delete +Access Modes: RWX +Capacity: 5Gi +Message: +Source: + Type: CSI (a Container Storage Interface (CSI) volume source) + Driver: ReadOnly: %v + + VolumeHandle: csi-cephfsplugin +%!(EXTRA string=csi-cephfs-7182b779-30d8-11e8-bf01-5254007d7491, bool=false)Events: +``` + +#### Step 9: Create a test pod + +``` +$ kubectl create -f ./deploy/cephfs/kubernetes/pod.yaml +``` ## Troubleshooting