mirrors/ceph-csi
1
0
Fork 0
mirror of https://github.com/ceph/ceph-csi.git synced 2024-12-22 21:10:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
CSI driver for Ceph
1,998 Commits 26 Branches 40 Tags 155 MiB
Go 95.1%
Shell 2.8%
Python 1%
Makefile 0.7%
Smarty 0.3%
Other 0.1%
07a916b84d
Go to file
Madhu Rajanna 07a916b84d rbd: mark image ready when image state is up+unknown 
To recover from split brain (up+error) state the image need to be
demoted and requested for resync on site-a and then the image on site-b
should gets demoted.The volume should be marked to ready=true when the
image state on both the clusters are up+unknown because during the last
snapshot syncing the data gets copied first and then image state on the
site-a changes to up+unknown.

If the image state on both the sites are up+unknown consider that
complete data is synced as the last snapshot
gets exchanged between the clusters.

* create 10 GB of file and validate the data after resync

* Do Failover when the site-a goes down
* Force promote the image and write data in GiB
* Once the site-a comes back, Demote the image and issue resync
* Demote the image on site-b
* The status will get reflected on the other site when the last
  snapshot sync happens
* The image will go to up+unknown state. and complete data will
  be copied to site a
* Promote the image on site-a and use it

```bash
csi-vol-5633715e-a7eb-11eb-bebb-0242ac110006:
  global_id:   e7f9ec55-06ab-46cb-a1ae-784be75ed96d
  state:       up+unknown
  description: remote image demoted
  service:     a on minicluster1
  last_update: 2021-04-28 07:11:56
  peer_sites:
    name: e47e29f4-96e8-44ed-b6c6-edf15c5a91d6-rook-ceph
    state: up+unknown
    description: remote image demoted
    last_update: 2021-04-28 07:11:41
 ```

* Do Failover when the site-a goes down
* Force promote the image on site-b and write data in GiB
* Demote the image on site-b
* Once the site-a comes back, Demote the image on site-a
* The images on the both site will go to split brain state

```bash
csi-vol-37effcb5-a7f1-11eb-bebb-0242ac110006:
  global_id:   115c3df9-3d4f-4c04-93a7-531b82155ddf
  state:       up+error
  description: split-brain
  service:     a on minicluster2
  last_update: 2021-04-28 07:25:41
  peer_sites:
    name: abbda0f0-0117-4425-8cb2-deb4c853da47-rook-ceph
    state: up+error
    description: split-brain
    last_update: 2021-04-28 07:25:26
```
* Issue resync
* The images cannot be resynced because when we issue resync
  on site a the image on site-b was in demoted state
* To recover from this state (promote and then demote the
  image on site-b after sometime)

```bash
csi-vol-37effcb5-a7f1-11eb-bebb-0242ac110006:
  global_id:   115c3df9-3d4f-4c04-93a7-531b82155ddf
  state:       up+unknown
  description: remote image demoted
  service:     a on minicluster1
  last_update: 2021-04-28 07:32:56
  peer_sites:
    name: e47e29f4-96e8-44ed-b6c6-edf15c5a91d6-rook-ceph
    state: up+unknown
    description: remote image demoted
    last_update: 2021-04-28 07:32:41
```
* Once the data is copied we can see that  the image state
  is moved to up+unknown on both sites
* Promote the image on site-a and use it

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-05-05 13:38:29 +00:00
.github doc: rename "master" branch to "devel" 2021-03-01 10:51:30 +05:30
assets feat: Adds Ceph logo as icon for Helm charts 2019-08-20 05:34:28 +00:00
charts helm: update external-snapshotter image to v4.0.0 2021-04-29 13:41:48 +00:00
cmd cleanup: refactor deeply nested if statement in cephcsi.go 2021-04-07 02:31:41 +00:00
deploy deploy: use serviceAccountName instead of serviceAccount in yamls 2021-04-06 09:00:35 +00:00
docs doc: correct the keys in certificate secrets 2021-04-29 08:51:29 +00:00
e2e rbd: delete encryption key from KMS 2021-04-30 08:05:47 +00:00
examples deploy: use serviceAccountName instead of serviceAccount in yamls 2021-04-06 09:00:35 +00:00
internal rbd: mark image ready when image state is up+unknown 2021-05-05 13:38:29 +00:00
scripts testing: enable nodeExpansion for k8s-storage rbd-rwo verification 2021-04-26 05:49:39 +00:00
troubleshooting/tools util: Fix tracevol to use --config for oc only 2020-08-25 10:04:19 +00:00
vendor rebase: rename kube-storage to csi-addons 2021-04-06 10:59:58 +00:00
.commitlintrc.yml ci: fix typo in commitlintrc.yml 2020-10-21 23:04:18 +00:00
.gitignore build: add .test-container-id to .gitignore 2020-08-18 14:34:08 +00:00
.mergify.yml ci: use mergify to add labels to PRs 2021-05-05 17:53:13 +05:30
.pre-commit-config.yaml ci: Add pre-commit hook to catch issues locally 2020-08-19 16:01:16 +00:00
.travis.yml ci: use "devel" branch instead of "master" 2021-03-01 10:51:30 +05:30
build.env e2e: update rook ceph cluster version 2021-05-03 13:12:46 +00:00
deploy.sh build: remove helm init from deploy.sh 2021-04-22 06:26:22 +00:00
go.mod rebase: rename kube-storage to csi-addons 2021-04-06 10:59:58 +00:00
go.sum rebase: rename kube-storage to csi-addons 2021-04-06 10:59:58 +00:00
LICENSE Removing appendix from license. 2019-08-09 15:16:46 +00:00
Makefile build: regenerate scripts/golangci.yml if build.env was updated 2021-03-17 13:38:46 +00:00
README.md doc: rename master to devel 2021-04-16 06:10:46 +00:00

README.md

Ceph CSI

Go Report Card Build Status

This repo contains Ceph Container Storage Interface (CSI) driver for RBD, CephFS and kubernetes sidecar deployment yamls of provisioner, attacher, resizer, driver-registrar and snapshotter for supporting CSI functionalities.

Overview

Ceph CSI plugins implement an interface between CSI enabled Container Orchestrator (CO) and Ceph cluster. It allows dynamically provisioning Ceph volumes and attaching them to workloads.

Independent CSI plugins are provided to support RBD and CephFS backed volumes,

  • For details about configuration and deployment of RBD plugin, please refer rbd doc and for CephFS plugin configuration and deployment please refer cephfs doc.
  • For example usage of RBD and CephFS CSI plugins, see examples in examples/.
  • Stale resource cleanup, please refer cleanup doc.

NOTE:

  • Ceph CSI Arm64 support is experimental.

Project status

Status: GA

Supported CO platforms

Ceph CSI drivers are currently developed and tested exclusively on Kubernetes environments. There is work in progress to make this CO independent and thus support other orchestration environments in the future.

NOTE:

  • csiv0.3 is deprecated with release of csi v1.1.0

Support Matrix

Ceph-CSI features and available versions

Plugin Features Feature Status CSI Driver Version CSI Spec Version Ceph Cluster Version Kubernetes Version
RBD Dynamically provision, de-provision Block mode RWO volume GA >= v1.0.0 >= v1.0.0 Nautilus (>=14.0.0) >= v1.14.0
Dynamically provision, de-provision Block mode RWX volume GA >= v1.0.0 >= v1.0.0 Nautilus (>=14.0.0) >= v1.14.0
Dynamically provision, de-provision File mode RWO volume GA >= v1.0.0 >= v1.0.0 Nautilus (>=14.0.0) >= v1.14.0
Provision File Mode ROX volume from snapshot Alpha >= v3.0.0 >= v1.0.0 Nautilus (>=v14.2.2) >= v1.17.0
Provision File Mode ROX volume from another volume Alpha >= v3.0.0 >= v1.0.0 Nautilus (>=v14.2.2) >= v1.16.0
Provision Block Mode ROX volume from snapshot Alpha >= v3.0.0 >= v1.0.0 Nautilus (>=v14.2.2) >= v1.17.0
Provision Block Mode ROX volume from another volume Alpha >= v3.0.0 >= v1.0.0 Nautilus (>=v14.2.2) >= v1.16.0
Creating and deleting snapshot Alpha >= v1.0.0 >= v1.0.0 Nautilus (>=14.0.0) >= v1.17.0
Provision volume from snapshot Alpha >= v1.0.0 >= v1.0.0 Nautilus (>=14.0.0) >= v1.17.0
Provision volume from another volume Alpha >= v1.0.0 >= v1.0.0 Nautilus (>=14.0.0) >= v1.16.0
Expand volume Beta >= v2.0.0 >= v1.1.0 Nautilus (>=14.0.0) >= v1.15.0
Metrics Support Beta >= v1.2.0 >= v1.1.0 Nautilus (>=14.0.0) >= v1.15.0
Topology Aware Provisioning Support Alpha >= v2.1.0 >= v1.1.0 Nautilus (>=14.0.0) >= v1.14.0
CephFS Dynamically provision, de-provision File mode RWO volume Beta >= v1.1.0 >= v1.0.0 Nautilus (>=14.2.2) >= v1.14.0
Dynamically provision, de-provision File mode RWX volume Beta >= v1.1.0 >= v1.0.0 Nautilus (>=v14.2.2) >= v1.14.0
Dynamically provision, de-provision File mode ROX volume Alpha >= v3.0.0 >= v1.0.0 Nautilus (>=v14.2.2) >= v1.14.0
Creating and deleting snapshot Alpha >= v3.1.0 >= v1.0.0 Octopus (>=v15.2.3) >= v1.17.0
Provision volume from snapshot Alpha >= v3.1.0 >= v1.0.0 Octopus (>=v15.2.3) >= v1.17.0
Provision volume from another volume Alpha >= v3.1.0 >= v1.0.0 Octopus (>=v15.2.3) >= v1.16.0
Expand volume Beta >= v2.0.0 >= v1.1.0 Nautilus (>=v14.2.2) >= v1.15.0
Metrics Beta >= v1.2.0 >= v1.1.0 Nautilus (>=v14.2.2) >= v1.15.0

NOTE: The Alpha status reflects possible non-backward compatible changes in the future, and is thus not recommended for production use.

CSI spec and Kubernetes version compatibility

Please refer to the matrix in the Kubernetes documentation.

Ceph CSI Container images and release compatibility

Ceph CSI Release/Branch Container image name Image Tag
devel (Branch) quay.io/cephcsi/cephcsi canary
v3.3.0 (Release) quay.io/cephcsi/cephcsi v3.3.0
v3.2.1 (Release) quay.io/cephcsi/cephcsi v3.2.1
v3.2.0 (Release) quay.io/cephcsi/cephcsi v3.2.0
v3.1.2 (Release) quay.io/cephcsi/cephcsi v3.1.2
v3.1.1 (Release) quay.io/cephcsi/cephcsi v3.1.1
v3.1.0 (Release) quay.io/cephcsi/cephcsi v3.1.0
v3.0.0 (Release) quay.io/cephcsi/cephcsi v3.0.0
v2.1.2 (Release) quay.io/cephcsi/cephcsi v2.1.2
v2.1.1 (Release) quay.io/cephcsi/cephcsi v2.1.1
v2.1.0 (Release) quay.io/cephcsi/cephcsi v2.1.0
v2.0.1 (Release) quay.io/cephcsi/cephcsi v2.0.1
v2.0.0 (Release) quay.io/cephcsi/cephcsi v2.0.0
v1.2.2 (Release) quay.io/cephcsi/cephcsi v1.2.2
v1.2.1 (Release) quay.io/cephcsi/cephcsi v1.2.1
v1.2.0 (Release) quay.io/cephcsi/cephcsi v1.2.0
v1.1.0 (Release) quay.io/cephcsi/cephcsi v1.1.0
v1.0.0 (Branch) quay.io/cephcsi/cephfsplugin v1.0.0
v1.0.0 (Branch) quay.io/cephcsi/rbdplugin v1.0.0

Contributing to this repo

Please follow development-guide and coding style guidelines if you are interested to contribute to this repo.

Troubleshooting

Please submit an issue at: Issues

Weekly Bug Triage call

We conduct weekly bug triage calls at our slack channel on Tuesdays. More details are available here

Dev standup

A regular dev standup takes place every other Monday,Tuesday,Thursday at 12:00 PM UTC. Convert to your local timezone by executing command date -d "12:00 UTC" on terminal

Any changes to the meeting schedule will be added to the agenda doc.

Anyone who wants to discuss the direction of the project, design and implementation reviews, or general questions with the broader community is welcome and encouraged to join.

Contact

Please use the following to reach members of the community: