Starting README

README.md

@@ -1,9 +1,65 @@
 # Ceph CSI
+
+## Overview
+
+RBD CSI plugin implements an interface between  CSI enabled Container
+Orchestrator and CEPH cluster. It allows dynamically provision CEPH
+volumes and attach it to workloads.
+Current implementation of CSI RBD plugin was tested in Kubernetes environment,
+but its code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic)
+and should be able to run with any CSI enabled CO (Containers Orchestration).
+
+An RBD CSI plugin is available to help simplify storage management.
+Once user creates PVC with the reference to a RBD storage class, rbd image and 
+corresponding PV object gets dynamically created and becomes ready to be used by
+workloads.
+
 [Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, and attacher for Ceph RBD and CephFS
 
-# Prerequisite
+## RBD Plugin
+### Configuration Requirements
 
-## Enable Mount Propagation in Docker 
+* Secret object with the authentication key for ceph cluster
+* StorageClass with rbdplugin (default CSI RBD plugin name) as a provisioner name
+  and information about ceph cluster (monitors, pool, etc) 
+* Service Accounts with required RBAC permissions   
+
+### Feature Status
+
+### 1.9: Alpha
+
+**Important:** `CSIPersistentVolume` and `MountPropagation`
+[feature gates must be enabled starting in 1.9](#enabling-the-alpha-feature-gates).
+Also API server must run with running config set to: `storage.k8s.io/v1alpha1` 
+
+### Compiling
+CSI RBD plugin can be compiled in a form of a binary file or in a form of a container. When compiled
+as a binary file, it gets stored in \_output folder with the name rbdplugin. When compiled as a container,
+the resulting image is stored in a local docker's image store. 
+
+To compile just a binary file:
+```
+$ make rbdplugin
+```
+
+To build a container:
+```
+$ make container
+```
+By running:
+```
+$ docker images | grep rbdplugin
+```
+You should see the following line in the output:
+```
+csi_images/rbdplugin   latest  248ddba297fa        30 seconds ago    431 MB
+```
+
+### Testing
+
+#### Prerequisite
+
+##### Enable Mount Propagation in Docker 
 
 Comment out `MountFlags=slave` in docker systemd service then restart docker service.
 ```bash
@@ -11,62 +67,98 @@ Comment out `MountFlags=slave` in docker systemd service then restart docker ser
 # systemctl restart docker
 ```
 
-## Enable Kubernetes Feature Gates
+##### Enable Kubernetes Feature Gates
 
 Enable features `MountPropagation=true,CSIPersistentVolume=true` and runtime config `storage.k8s.io/v1alpha1=true`
 
-# Build
+#### Step 1: Create Secret 
+```
+$ kubectl create -f ./deploy/kubernetes/rbd-secrets.yaml 
+```
+**Important:** rbd-secrets.yaml, must be customized to match your ceph environment.
+
+#### Step 2: Create StorageClass
+```
+$ kubectl create -f ./deploy/kubernetes/rbd-storage-class.yaml
+```
+**Important:** rbd-secrets.yaml, must be customized to match your ceph environment.
+
+#### Step 3: Start CSI CEPH RBD plugin
+```
+$ kubectl create -f ./deploy/kubernetes/rbdplugin.yaml
+```
+
+#### Step 4: Start CSI External Attacher
+```
+$ kubectl create -f ./deploy/kubernetes/csi-attacher.yaml
+```
+
+#### Step 5: Start CSI External Provisioner  
+```
+$ kubectl create -f ./deploy/kubernetes/csi-provisioner.yaml
+```
+**Important:** Deployment yaml files includes required Service Account definitions and
+required RBAC rules.
+
+#### Step 6: Check status of CSI RBD plugin  
+```
+$ kubectl get pods | grep csi 
+```
+
+The following output should be displayed:
+
+```
+NAMESPACE     NAME                                                READY     STATUS    RESTARTS   AGE          
+default       csi-attacher-0                                      1/1       Running   0          1d           
+default       csi-nodeplugin-rbdplugin-qxqtl                      2/2       Running   0          1d           
+default       csi-provisioner-0                                   1/1       Running   0          1d           
+```
+
+#### Step 7: Create PVC 
+```
+$ kubectl create -f ./deploy/kubernetes/pvc.yaml
+```
+
+#### Step 8: Check status of provisioner PV  
+```
+$ kubectl get pv
+```
+
+The following output should be displayed:
+
+```
+NAME                                                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS   REASON    AGE
+kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea   5Gi        RWO            Delete           Bound     default/csi-pvc   rbdv2                    10s
+```
+
+```
+$ kubectl describe pv kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
+Name:            kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
+Annotations:     csi.volume.kubernetes.io/volume-attributes={"monitors":"192.168.80.233:6789","pool":"kubernetes"}
+                 csiProvisionerIdentity=1516716490787-8081-rbdplugin  <------ !!!
+                 pv.kubernetes.io/provisioned-by=rbdplugin
+StorageClass:    rbdv2              <------ !!!
+Status:          Bound              <------ !!!
+Claim:           default/csi-pvc    <------ !!!
+Reclaim Policy:  Delete
+Access Modes:    RWO
+VolumeMode:      Filesystem
+Capacity:        5Gi
+Message:         
+Source:
+    Type:    CSI <------ !!!
+```
+
+#### Step 9: Create a test pod
 
 ```bash
-# make container
+# kubectl create -f ./deploy/pod.yaml
 ```
 
-# Test
+## CepgFS plugin
 
-## Start rbdplugin and driver registrar
+TODO 
 
-```bash
+## Troubleshooting
-# kubectl create -f deploy/kubernetes/rbdplugin.yaml
-```
 
-### Start CSI external volume provisioner
+Please submit an issue at:[Issues](https://github.com/ceph/ceph-csi/issues)
-
-```bash
-# kubectl create -f deploy/kubernetes/csi-provisioner.yaml
-```
-
-### Start CSI external volume attacher
-
-```
-# kubectl create -f deploy/kubernetes/csi-attacher.yaml
-```
-
-### Verify all componets are ready
-
-```bash
-# kubectl get pod
-NAME                             READY     STATUS    RESTARTS   AGE
-csi-attacher-0                   1/1       Running   0          6s
-csi-nodeplugin-rbdplugin-kwhhc   2/2       Running   0          6m
-csi-provisioner-0                1/1       Running   0          1m
-```
-
-### Create a CSI storage class
-
-Replace Ceph mon and secrets in [examples/sc.yaml](examples/sc.yaml)
-
-```bash
-# kubectl create -f examples/sc.yaml
-```
-
-### Create a PVC
-
-```bash
-# kubectl create -f examples/pvc.yaml
-```
-
-### Create a Pod
-
-```bash
-# kubectl create -f examples/pod.yaml
-```

deploy/kubernetes/pvc.yaml

@@ -1,11 +1,11 @@
-kind: PersistentVolumeClaim
 apiVersion: v1
+kind: PersistentVolumeClaim
 metadata:
-  name: rbd-pv 
+  name: pvc
 spec:
   accessModes:
   - ReadWriteOnce
-  storageClassName: rbd-csi-provisioner 
   resources:
     requests:
-      storage: 1Gi
+      storage: 5Gi
+  storageClassName: rbd

deploy/kubernetes/rbd-secrets.yaml

@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ceph-secret-admin
+  namespace: kube-system 
+type: "kubernetes.io/rbd"
+data:
+#Please note this value is base64 encoded.
+  key: QVFDZUhPMVpJTFBQRFJBQTd6dzNkNzZicGxrdlR3em9vc3lidkE9PQo=
+type: kubernetes.io/rbd
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ceph-secret-user
+type: "kubernetes.io/rbd"  
+data:
+#Please note this value is base64 encoded.
+  key: QVFDZDR1MVoxSDI0QnhBQWFxdmZIRnFuMSs0RFZlK1pRZ0ZmUEE9PQo=

deploy/kubernetes/rbd-storage-class.yaml

@@ -0,0 +1,13 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+   name: rbd
+provisioner: rbdplugin 
+parameters:
+    monitors: 192.168.80.233:6789
+    pool: kubernetes
+    adminID: admin
+    adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA==
+    userID: kube
+    userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
+reclaimPolicy: Delete 

examples/sc.yaml

@@ -1,12 +0,0 @@
-apiVersion: storage.k8s.io/v1beta1
-kind: StorageClass
-metadata:
-  name: rbd-csi-provisioner 
-provisioner: rbdplugin
-parameters:
-   monitors: 172.24.0.6:6789
-   pool: rbd
-   adminID: admin
-   adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA==
-   userID: kube
-   userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==