Starting README

This commit is contained in:
Serguei Bezverkhi 2018-01-20 15:04:38 -05:00
parent 86acb9ccd9
commit 29ec9358d4
6 changed files with 180 additions and 68 deletions

194
README.md
View File

@ -1,9 +1,65 @@
# Ceph CSI
## Overview
RBD CSI plugin implements an interface between CSI enabled Container
Orchestrator and CEPH cluster. It allows dynamically provision CEPH
volumes and attach it to workloads.
Current implementation of CSI RBD plugin was tested in Kubernetes environment,
but its code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic)
and should be able to run with any CSI enabled CO (Containers Orchestration).
An RBD CSI plugin is available to help simplify storage management.
Once user creates PVC with the reference to a RBD storage class, rbd image and
corresponding PV object gets dynamically created and becomes ready to be used by
workloads.
[Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, and attacher for Ceph RBD and CephFS
# Prerequisite
## RBD Plugin
### Configuration Requirements
## Enable Mount Propagation in Docker
* Secret object with the authentication key for ceph cluster
* StorageClass with rbdplugin (default CSI RBD plugin name) as a provisioner name
and information about ceph cluster (monitors, pool, etc)
* Service Accounts with required RBAC permissions
### Feature Status
### 1.9: Alpha
**Important:** `CSIPersistentVolume` and `MountPropagation`
[feature gates must be enabled starting in 1.9](#enabling-the-alpha-feature-gates).
Also API server must run with running config set to: `storage.k8s.io/v1alpha1`
### Compiling
CSI RBD plugin can be compiled in a form of a binary file or in a form of a container. When compiled
as a binary file, it gets stored in \_output folder with the name rbdplugin. When compiled as a container,
the resulting image is stored in a local docker's image store.
To compile just a binary file:
```
$ make rbdplugin
```
To build a container:
```
$ make container
```
By running:
```
$ docker images | grep rbdplugin
```
You should see the following line in the output:
```
csi_images/rbdplugin latest 248ddba297fa 30 seconds ago 431 MB
```
### Testing
#### Prerequisite
##### Enable Mount Propagation in Docker
Comment out `MountFlags=slave` in docker systemd service then restart docker service.
```bash
@ -11,62 +67,98 @@ Comment out `MountFlags=slave` in docker systemd service then restart docker ser
# systemctl restart docker
```
## Enable Kubernetes Feature Gates
##### Enable Kubernetes Feature Gates
Enable features `MountPropagation=true,CSIPersistentVolume=true` and runtime config `storage.k8s.io/v1alpha1=true`
# Build
#### Step 1: Create Secret
```
$ kubectl create -f ./deploy/kubernetes/rbd-secrets.yaml
```
**Important:** rbd-secrets.yaml, must be customized to match your ceph environment.
#### Step 2: Create StorageClass
```
$ kubectl create -f ./deploy/kubernetes/rbd-storage-class.yaml
```
**Important:** rbd-secrets.yaml, must be customized to match your ceph environment.
#### Step 3: Start CSI CEPH RBD plugin
```
$ kubectl create -f ./deploy/kubernetes/rbdplugin.yaml
```
#### Step 4: Start CSI External Attacher
```
$ kubectl create -f ./deploy/kubernetes/csi-attacher.yaml
```
#### Step 5: Start CSI External Provisioner
```
$ kubectl create -f ./deploy/kubernetes/csi-provisioner.yaml
```
**Important:** Deployment yaml files includes required Service Account definitions and
required RBAC rules.
#### Step 6: Check status of CSI RBD plugin
```
$ kubectl get pods | grep csi
```
The following output should be displayed:
```
NAMESPACE NAME READY STATUS RESTARTS AGE
default csi-attacher-0 1/1 Running 0 1d
default csi-nodeplugin-rbdplugin-qxqtl 2/2 Running 0 1d
default csi-provisioner-0 1/1 Running 0 1d
```
#### Step 7: Create PVC
```
$ kubectl create -f ./deploy/kubernetes/pvc.yaml
```
#### Step 8: Check status of provisioner PV
```
$ kubectl get pv
```
The following output should be displayed:
```
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea 5Gi RWO Delete Bound default/csi-pvc rbdv2 10s
```
```
$ kubectl describe pv kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
Name: kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
Annotations: csi.volume.kubernetes.io/volume-attributes={"monitors":"192.168.80.233:6789","pool":"kubernetes"}
csiProvisionerIdentity=1516716490787-8081-rbdplugin <------ !!!
pv.kubernetes.io/provisioned-by=rbdplugin
StorageClass: rbdv2 <------ !!!
Status: Bound <------ !!!
Claim: default/csi-pvc <------ !!!
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 5Gi
Message:
Source:
Type: CSI <------ !!!
```
#### Step 9: Create a test pod
```bash
# make container
# kubectl create -f ./deploy/pod.yaml
```
# Test
## CepgFS plugin
## Start rbdplugin and driver registrar
TODO
```bash
# kubectl create -f deploy/kubernetes/rbdplugin.yaml
```
## Troubleshooting
### Start CSI external volume provisioner
```bash
# kubectl create -f deploy/kubernetes/csi-provisioner.yaml
```
### Start CSI external volume attacher
```
# kubectl create -f deploy/kubernetes/csi-attacher.yaml
```
### Verify all componets are ready
```bash
# kubectl get pod
NAME READY STATUS RESTARTS AGE
csi-attacher-0 1/1 Running 0 6s
csi-nodeplugin-rbdplugin-kwhhc 2/2 Running 0 6m
csi-provisioner-0 1/1 Running 0 1m
```
### Create a CSI storage class
Replace Ceph mon and secrets in [examples/sc.yaml](examples/sc.yaml)
```bash
# kubectl create -f examples/sc.yaml
```
### Create a PVC
```bash
# kubectl create -f examples/pvc.yaml
```
### Create a Pod
```bash
# kubectl create -f examples/pod.yaml
```
Please submit an issue at:[Issues](https://github.com/ceph/ceph-csi/issues)

View File

@ -1,11 +1,11 @@
kind: PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pv
name: pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: rbd-csi-provisioner
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storage: 5Gi
storageClassName: rbd

View File

@ -0,0 +1,19 @@
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret-admin
namespace: kube-system
type: "kubernetes.io/rbd"
data:
#Please note this value is base64 encoded.
key: QVFDZUhPMVpJTFBQRFJBQTd6dzNkNzZicGxrdlR3em9vc3lidkE9PQo=
type: kubernetes.io/rbd
---
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret-user
type: "kubernetes.io/rbd"
data:
#Please note this value is base64 encoded.
key: QVFDZDR1MVoxSDI0QnhBQWFxdmZIRnFuMSs0RFZlK1pRZ0ZmUEE9PQo=

View File

@ -0,0 +1,13 @@
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rbd
provisioner: rbdplugin
parameters:
monitors: 192.168.80.233:6789
pool: kubernetes
adminID: admin
adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA==
userID: kube
userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
reclaimPolicy: Delete

View File

@ -1,12 +0,0 @@
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: rbd-csi-provisioner
provisioner: rbdplugin
parameters:
monitors: 172.24.0.6:6789
pool: rbd
adminID: admin
adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA==
userID: kube
userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==