mirror of
https://github.com/ceph/ceph-csi.git
synced 2024-12-22 13:00:19 +00:00
Huamin Chen
GitHub
commit
c4c51e0e39
194
README.md
194
README.md
@ -1,9 +1,65 @@
|
||||
# Ceph CSI
|
||||
|
||||
## Overview
|
||||
|
||||
RBD CSI plugin implements an interface between CSI enabled Container
|
||||
Orchestrator and CEPH cluster. It allows dynamically provision CEPH
|
||||
volumes and attach it to workloads.
|
||||
Current implementation of CSI RBD plugin was tested in Kubernetes environment,
|
||||
but its code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic)
|
||||
and should be able to run with any CSI enabled CO (Containers Orchestration).
|
||||
|
||||
An RBD CSI plugin is available to help simplify storage management.
|
||||
Once user creates PVC with the reference to a RBD storage class, rbd image and
|
||||
corresponding PV object gets dynamically created and becomes ready to be used by
|
||||
workloads.
|
||||
|
||||
[Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, and attacher for Ceph RBD and CephFS
|
||||
|
||||
# Prerequisite
|
||||
## RBD Plugin
|
||||
### Configuration Requirements
|
||||
|
||||
## Enable Mount Propagation in Docker
|
||||
* Secret object with the authentication key for ceph cluster
|
||||
* StorageClass with rbdplugin (default CSI RBD plugin name) as a provisioner name
|
||||
and information about ceph cluster (monitors, pool, etc)
|
||||
* Service Accounts with required RBAC permissions
|
||||
|
||||
### Feature Status
|
||||
|
||||
### 1.9: Alpha
|
||||
|
||||
**Important:** `CSIPersistentVolume` and `MountPropagation`
|
||||
[feature gates must be enabled starting in 1.9](#enabling-the-alpha-feature-gates).
|
||||
Also API server must run with running config set to: `storage.k8s.io/v1alpha1`
|
||||
|
||||
### Compiling
|
||||
CSI RBD plugin can be compiled in a form of a binary file or in a form of a container. When compiled
|
||||
as a binary file, it gets stored in \_output folder with the name rbdplugin. When compiled as a container,
|
||||
the resulting image is stored in a local docker's image store.
|
||||
|
||||
To compile just a binary file:
|
||||
```
|
||||
$ make rbdplugin
|
||||
```
|
||||
|
||||
To build a container:
|
||||
```
|
||||
$ make container
|
||||
```
|
||||
By running:
|
||||
```
|
||||
$ docker images | grep rbdplugin
|
||||
```
|
||||
You should see the following line in the output:
|
||||
```
|
||||
csi_images/rbdplugin latest 248ddba297fa 30 seconds ago 431 MB
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
#### Prerequisite
|
||||
|
||||
##### Enable Mount Propagation in Docker
|
||||
|
||||
Comment out `MountFlags=slave` in docker systemd service then restart docker service.
|
||||
```bash
|
||||
@ -11,62 +67,98 @@ Comment out `MountFlags=slave` in docker systemd service then restart docker ser
|
||||
# systemctl restart docker
|
||||
```
|
||||
|
||||
## Enable Kubernetes Feature Gates
|
||||
##### Enable Kubernetes Feature Gates
|
||||
|
||||
Enable features `MountPropagation=true,CSIPersistentVolume=true` and runtime config `storage.k8s.io/v1alpha1=true`
|
||||
|
||||
# Build
|
||||
#### Step 1: Create Secret
|
||||
```
|
||||
$ kubectl create -f ./deploy/kubernetes/rbd-secrets.yaml
|
||||
```
|
||||
**Important:** rbd-secrets.yaml, must be customized to match your ceph environment.
|
||||
|
||||
#### Step 2: Create StorageClass
|
||||
```
|
||||
$ kubectl create -f ./deploy/kubernetes/rbd-storage-class.yaml
|
||||
```
|
||||
**Important:** rbd-secrets.yaml, must be customized to match your ceph environment.
|
||||
|
||||
#### Step 3: Start CSI CEPH RBD plugin
|
||||
```
|
||||
$ kubectl create -f ./deploy/kubernetes/rbdplugin.yaml
|
||||
```
|
||||
|
||||
#### Step 4: Start CSI External Attacher
|
||||
```
|
||||
$ kubectl create -f ./deploy/kubernetes/csi-attacher.yaml
|
||||
```
|
||||
|
||||
#### Step 5: Start CSI External Provisioner
|
||||
```
|
||||
$ kubectl create -f ./deploy/kubernetes/csi-provisioner.yaml
|
||||
```
|
||||
**Important:** Deployment yaml files includes required Service Account definitions and
|
||||
required RBAC rules.
|
||||
|
||||
#### Step 6: Check status of CSI RBD plugin
|
||||
```
|
||||
$ kubectl get pods | grep csi
|
||||
```
|
||||
|
||||
The following output should be displayed:
|
||||
|
||||
```
|
||||
NAMESPACE NAME READY STATUS RESTARTS AGE
|
||||
default csi-attacher-0 1/1 Running 0 1d
|
||||
default csi-nodeplugin-rbdplugin-qxqtl 2/2 Running 0 1d
|
||||
default csi-provisioner-0 1/1 Running 0 1d
|
||||
```
|
||||
|
||||
#### Step 7: Create PVC
|
||||
```
|
||||
$ kubectl create -f ./deploy/kubernetes/pvc.yaml
|
||||
```
|
||||
|
||||
#### Step 8: Check status of provisioner PV
|
||||
```
|
||||
$ kubectl get pv
|
||||
```
|
||||
|
||||
The following output should be displayed:
|
||||
|
||||
```
|
||||
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
|
||||
kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea 5Gi RWO Delete Bound default/csi-pvc rbdv2 10s
|
||||
```
|
||||
|
||||
```
|
||||
$ kubectl describe pv kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
|
||||
Name: kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
|
||||
Annotations: csi.volume.kubernetes.io/volume-attributes={"monitors":"192.168.80.233:6789","pool":"kubernetes"}
|
||||
csiProvisionerIdentity=1516716490787-8081-rbdplugin <------ !!!
|
||||
pv.kubernetes.io/provisioned-by=rbdplugin
|
||||
StorageClass: rbdv2 <------ !!!
|
||||
Status: Bound <------ !!!
|
||||
Claim: default/csi-pvc <------ !!!
|
||||
Reclaim Policy: Delete
|
||||
Access Modes: RWO
|
||||
VolumeMode: Filesystem
|
||||
Capacity: 5Gi
|
||||
Message:
|
||||
Source:
|
||||
Type: CSI <------ !!!
|
||||
```
|
||||
|
||||
#### Step 9: Create a test pod
|
||||
|
||||
```bash
|
||||
# make container
|
||||
# kubectl create -f ./deploy/pod.yaml
|
||||
```
|
||||
|
||||
# Test
|
||||
## CepgFS plugin
|
||||
|
||||
## Start rbdplugin and driver registrar
|
||||
TODO
|
||||
|
||||
```bash
|
||||
# kubectl create -f deploy/kubernetes/rbdplugin.yaml
|
||||
```
|
||||
## Troubleshooting
|
||||
|
||||
### Start CSI external volume provisioner
|
||||
|
||||
```bash
|
||||
# kubectl create -f deploy/kubernetes/csi-provisioner.yaml
|
||||
```
|
||||
|
||||
### Start CSI external volume attacher
|
||||
|
||||
```
|
||||
# kubectl create -f deploy/kubernetes/csi-attacher.yaml
|
||||
```
|
||||
|
||||
### Verify all componets are ready
|
||||
|
||||
```bash
|
||||
# kubectl get pod
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
csi-attacher-0 1/1 Running 0 6s
|
||||
csi-nodeplugin-rbdplugin-kwhhc 2/2 Running 0 6m
|
||||
csi-provisioner-0 1/1 Running 0 1m
|
||||
```
|
||||
|
||||
### Create a CSI storage class
|
||||
|
||||
Replace Ceph mon and secrets in [examples/sc.yaml](examples/sc.yaml)
|
||||
|
||||
```bash
|
||||
# kubectl create -f examples/sc.yaml
|
||||
```
|
||||
|
||||
### Create a PVC
|
||||
|
||||
```bash
|
||||
# kubectl create -f examples/pvc.yaml
|
||||
```
|
||||
|
||||
### Create a Pod
|
||||
|
||||
```bash
|
||||
# kubectl create -f examples/pod.yaml
|
||||
```
|
||||
Please submit an issue at:[Issues](https://github.com/ceph/ceph-csi/issues)
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rbd-pv
|
||||
name: pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: rbd-csi-provisioner
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
storage: 5Gi
|
||||
storageClassName: rbd
|
||||
19
deploy/kubernetes/rbd-secrets.yaml
Normal file
19
deploy/kubernetes/rbd-secrets.yaml
Normal file
@ -0,0 +1,19 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ceph-secret-admin
|
||||
namespace: kube-system
|
||||
type: "kubernetes.io/rbd"
|
||||
data:
|
||||
#Please note this value is base64 encoded.
|
||||
key: QVFDZUhPMVpJTFBQRFJBQTd6dzNkNzZicGxrdlR3em9vc3lidkE9PQo=
|
||||
type: kubernetes.io/rbd
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ceph-secret-user
|
||||
type: "kubernetes.io/rbd"
|
||||
data:
|
||||
#Please note this value is base64 encoded.
|
||||
key: QVFDZDR1MVoxSDI0QnhBQWFxdmZIRnFuMSs0RFZlK1pRZ0ZmUEE9PQo=
|
||||
13
deploy/kubernetes/rbd-storage-class.yaml
Normal file
13
deploy/kubernetes/rbd-storage-class.yaml
Normal file
@ -0,0 +1,13 @@
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: rbd
|
||||
provisioner: rbdplugin
|
||||
parameters:
|
||||
monitors: 192.168.80.233:6789
|
||||
pool: kubernetes
|
||||
adminID: admin
|
||||
adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA==
|
||||
userID: kube
|
||||
userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
|
||||
reclaimPolicy: Delete
|
||||
@ -1,12 +0,0 @@
|
||||
apiVersion: storage.k8s.io/v1beta1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: rbd-csi-provisioner
|
||||
provisioner: rbdplugin
|
||||
parameters:
|
||||
monitors: 172.24.0.6:6789
|
||||
pool: rbd
|
||||
adminID: admin
|
||||
adminSecret: AQAmsGBap4EoBhAAET/Hc7fBqAZj/cy7cDcoQA==
|
||||
userID: kube
|
||||
userSecret: AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
|
||||
Loading…
Reference in New Issue
Block a user