diff --git a/README.md b/README.md index 8efe6d524..fbc40d344 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,65 @@ # Ceph CSI + +## Overview + +RBD CSI plugin implements an interface between CSI enabled Container +Orchestrator and CEPH cluster. It allows dynamically provision CEPH +volumes and attach it to workloads. +Current implementation of CSI RBD plugin was tested in Kubernetes environment, +but its code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic) +and should be able to run with any CSI enabled CO (Containers Orchestration). + +An RBD CSI plugin is available to help simplify storage management. +Once user creates PVC with the reference to a RBD storage class, rbd image and +corresponding PV object gets dynamically created and becomes ready to be used by +workloads. + [Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, and attacher for Ceph RBD and CephFS -# Prerequisite +## RBD Plugin +### Configuration Requirements -## Enable Mount Propagation in Docker +* Secret object with the authentication key for ceph cluster +* StorageClass with rbdplugin (default CSI RBD plugin name) as a provisioner name + and information about ceph cluster (monitors, pool, etc) +* Service Accounts with required RBAC permissions + +### Feature Status + +### 1.9: Alpha + +**Important:** `CSIPersistentVolume` and `MountPropagation` +[feature gates must be enabled starting in 1.9](#enabling-the-alpha-feature-gates). +Also API server must run with running config set to: `storage.k8s.io/v1alpha1` + +### Compiling +CSI RBD plugin can be compiled in a form of a binary file or in a form of a container. When compiled +as a binary file, it gets stored in \_output folder with the name rbdplugin. When compiled as a container, +the resulting image is stored in a local docker's image store. + +To compile just a binary file: +``` +$ make rbdplugin +``` + +To build a container: +``` +$ make container +``` +By running: +``` +$ docker images | grep rbdplugin +``` +You should see the following line in the output: +``` +csi_images/rbdplugin latest 248ddba297fa 30 seconds ago 431 MB +``` + +### Testing + +#### Prerequisite + +##### Enable Mount Propagation in Docker Comment out `MountFlags=slave` in docker systemd service then restart docker service. ```bash @@ -11,62 +67,98 @@ Comment out `MountFlags=slave` in docker systemd service then restart docker ser # systemctl restart docker ``` -## Enable Kubernetes Feature Gates +##### Enable Kubernetes Feature Gates Enable features `MountPropagation=true,CSIPersistentVolume=true` and runtime config `storage.k8s.io/v1alpha1=true` -# Build +#### Step 1: Create Secret +``` +$ kubectl create -f ./deploy/kubernetes/rbd-secrets.yaml +``` +**Important:** rbd-secrets.yaml, must be customized to match your ceph environment. + +#### Step 2: Create StorageClass +``` +$ kubectl create -f ./deploy/kubernetes/rbd-storage-class.yaml +``` +**Important:** rbd-secrets.yaml, must be customized to match your ceph environment. + +#### Step 3: Start CSI CEPH RBD plugin +``` +$ kubectl create -f ./deploy/kubernetes/rbdplugin.yaml +``` + +#### Step 4: Start CSI External Attacher +``` +$ kubectl create -f ./deploy/kubernetes/csi-attacher.yaml +``` + +#### Step 5: Start CSI External Provisioner +``` +$ kubectl create -f ./deploy/kubernetes/csi-provisioner.yaml +``` +**Important:** Deployment yaml files includes required Service Account definitions and +required RBAC rules. + +#### Step 6: Check status of CSI RBD plugin +``` +$ kubectl get pods | grep csi +``` + +The following output should be displayed: + +``` +NAMESPACE NAME READY STATUS RESTARTS AGE +default csi-attacher-0 1/1 Running 0 1d +default csi-nodeplugin-rbdplugin-qxqtl 2/2 Running 0 1d +default csi-provisioner-0 1/1 Running 0 1d +``` + +#### Step 7: Create PVC +``` +$ kubectl create -f ./deploy/kubernetes/pvc.yaml +``` + +#### Step 8: Check status of provisioner PV +``` +$ kubectl get pv +``` + +The following output should be displayed: + +``` +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea 5Gi RWO Delete Bound default/csi-pvc rbdv2 10s +``` + +``` +$ kubectl describe pv kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea +Name: kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea +Annotations: csi.volume.kubernetes.io/volume-attributes={"monitors":"192.168.80.233:6789","pool":"kubernetes"} + csiProvisionerIdentity=1516716490787-8081-rbdplugin <------ !!! + pv.kubernetes.io/provisioned-by=rbdplugin +StorageClass: rbdv2 <------ !!! +Status: Bound <------ !!! +Claim: default/csi-pvc <------ !!! +Reclaim Policy: Delete +Access Modes: RWO +VolumeMode: Filesystem +Capacity: 5Gi +Message: +Source: + Type: CSI <------ !!! +``` + +#### Step 9: Create a test pod ```bash -# make container +# kubectl create -f ./deploy/pod.yaml ``` -# Test +## CepgFS plugin -## Start rbdplugin and driver registrar +TODO -```bash -# kubectl create -f deploy/kubernetes/rbdplugin.yaml -``` +## Troubleshooting -### Start CSI external volume provisioner - -```bash -# kubectl create -f deploy/kubernetes/csi-provisioner.yaml -``` - -### Start CSI external volume attacher - -``` -# kubectl create -f deploy/kubernetes/csi-attacher.yaml -``` - -### Verify all componets are ready - -```bash -# kubectl get pod -NAME READY STATUS RESTARTS AGE -csi-attacher-0 1/1 Running 0 6s -csi-nodeplugin-rbdplugin-kwhhc 2/2 Running 0 6m -csi-provisioner-0 1/1 Running 0 1m -``` - -### Create a CSI storage class - -Replace Ceph mon and secrets in [examples/sc.yaml](examples/sc.yaml) - -```bash -# kubectl create -f examples/sc.yaml -``` - -### Create a PVC - -```bash -# kubectl create -f examples/pvc.yaml -``` - -### Create a Pod - -```bash -# kubectl create -f examples/pod.yaml -``` +Please submit an issue at:[Issues](https://github.com/ceph/ceph-csi/issues) diff --git a/examples/pvc.yaml b/deploy/kubernetes/pvc.yaml similarity index 51% rename from examples/pvc.yaml rename to deploy/kubernetes/pvc.yaml index 306d773fa..d3dd0058a 100644 --- a/examples/pvc.yaml +++ b/deploy/kubernetes/pvc.yaml @@ -1,11 +1,11 @@ -kind: PersistentVolumeClaim apiVersion: v1 +kind: PersistentVolumeClaim metadata: - name: rbd-pv + name: pvc spec: accessModes: - - ReadWriteOnce - storageClassName: rbd-csi-provisioner + - ReadWriteOnce resources: requests: - storage: 1Gi + storage: 5Gi + storageClassName: rbd diff --git a/deploy/kubernetes/rbd-secrets.yaml b/deploy/kubernetes/rbd-secrets.yaml new file mode 100644 index 000000000..160d9b87b --- /dev/null +++ b/deploy/kubernetes/rbd-secrets.yaml @@ -0,0 +1,19 @@ +apiVersion: v1 +kind: Secret +metadata: + name: ceph-secret-admin + namespace: kube-system +type: "kubernetes.io/rbd" +data: +#Please note this value is base64 encoded. + key: QVFDZUhPMVpJTFBQRFJBQTd6dzNkNzZicGxrdlR3em9vc3lidkE9PQo= +type: kubernetes.io/rbd +--- +apiVersion: v1 +kind: Secret +metadata: + name: ceph-secret-user +type: "kubernetes.io/rbd" +data: +#Please note this value is base64 encoded. + key: QVFDZDR1MVoxSDI0QnhBQWFxdmZIRnFuMSs0RFZlK1pRZ0ZmUEE9PQo= diff --git a/deploy/kubernetes/rbd-storage-class.yaml b/deploy/kubernetes/rbd-storage-class.yaml new file mode 100644 index 000000000..c5fb54c8c --- /dev/null +++ b/deploy/kubernetes/rbd-storage-class.yaml @@ -0,0 +1,13 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: rbd +provisioner: rbdplugin +parameters: + monitors: 192.168.80.233:6789 + pool: kubernetes + adminID: admin + adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA== + userID: kube + userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ== +reclaimPolicy: Delete diff --git a/examples/pod.yaml b/deploy/pod.yaml similarity index 100% rename from examples/pod.yaml rename to deploy/pod.yaml diff --git a/examples/sc.yaml b/examples/sc.yaml deleted file mode 100644 index 7a4b6ad7a..000000000 --- a/examples/sc.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: storage.k8s.io/v1beta1 -kind: StorageClass -metadata: - name: rbd-csi-provisioner -provisioner: rbdplugin -parameters: - monitors: 172.24.0.6:6789 - pool: rbd - adminID: admin - adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA== - userID: kube - userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==