mirror of
https://github.com/ceph/ceph-csi.git
synced 2025-06-13 02:33:34 +00:00
Removed config maps and replaced with rados omaps
Existing config maps are now replaced with rados omaps that help store information regarding the requested volume names and the rbd image names backing the same. Further to detect cluster, pool and which image a volume ID refers to, changes to volume ID encoding has been done as per provided design specification in the stateless ceph-csi proposal. Additional changes and updates, - Updated documentation - Updated manifests - Updated Helm chart - Addressed a few csi-test failures Signed-off-by: ShyamsundarR <srangana@redhat.com>
This commit is contained in:
ShyamsundarR
committed by
mergify[bot]
mergify[bot]
parent
f60a07ae82
commit
d02e50aa9b
@ -1,5 +1,7 @@
|
||||
# How to test RBD and CephFS plugins with Kubernetes 1.13
|
||||
|
||||
## Deploying Ceph-CSI services
|
||||
|
||||
Both `rbd` and `cephfs` directories contain `plugin-deploy.sh` and
|
||||
`plugin-teardown.sh` helper scripts. You can use those to help you
|
||||
deploy/teardown RBACs, sidecar containers and the plugin in one go.
|
||||
@ -7,16 +9,45 @@ By default, they look for the YAML manifests in
|
||||
`../../deploy/{rbd,cephfs}/kubernetes`.
|
||||
You can override this path by running `$ ./plugin-deploy.sh /path/to/my/manifests`.
|
||||
|
||||
## Creating CSI configuration for RBD based provisioning
|
||||
|
||||
**NOTE:** This section is not required for cephfs based provisioning, and SHOULD
|
||||
be skipped.
|
||||
|
||||
For RBD based provisioning, the CSI plugin requires configuration information
|
||||
regarding the Ceph cluster(s), that would host the RBD based block devices. This
|
||||
is provided by adding a per-cluster identifier (referred to as clusterID), and
|
||||
the required monitor details for the same, as in the provided [sample config
|
||||
map](./rbd/csi-config-map-sample.yaml).
|
||||
|
||||
Gather the following information from the Ceph cluster(s) of choice,
|
||||
|
||||
* Ceph monitor list
|
||||
* Typically in the output of `ceph mon dump`
|
||||
* Used to prepare a list of `monitors` in the CSI configuration file
|
||||
* Ceph Cluster fsid
|
||||
* If choosing to use the Ceph cluster fsid as the unique value of clusterID,
|
||||
* Output of `ceph fsid`
|
||||
* Alternatively, choose a `<cluster-id>` value that is distinct per Ceph
|
||||
cluster in use by this kubernetes cluster
|
||||
|
||||
Update the [sample config map](./rbd/csi-config-map-sample.yaml) with values
|
||||
from a Ceph cluster and replace `<cluster-id>` with the chosen clusterID, to
|
||||
create the manifest for the config map which can be updated in the cluster
|
||||
using the following command,
|
||||
|
||||
* `kubectl replace -f rbd/csi-config-map-sample.yaml`
|
||||
|
||||
Storage class and snapshot class, using `<cluster-id>` as the value for the
|
||||
option `clusterID`, can now be created on the cluster.
|
||||
|
||||
## Deploying the storage class
|
||||
|
||||
Once the plugin is successfully deployed, you'll need to customize
|
||||
`storageclass.yaml` and `secret.yaml` manifests to reflect your Ceph cluster
|
||||
setup.
|
||||
Please consult the documentation for info about available parameters.
|
||||
|
||||
**NOTE:** See section
|
||||
[Cluster ID based configuration](#cluster-id-based-configuration) if using
|
||||
the `clusterID` instead of `monitors` or `monValueFromSecret` option in the
|
||||
storage class for RBD based provisioning before proceeding.
|
||||
|
||||
After configuring the secrets, monitors, etc. you can deploy a
|
||||
testing Pod mounting a RBD image / CephFS volume:
|
||||
|
||||
@ -32,7 +63,7 @@ Other helper scripts:
|
||||
* `logs.sh` output of the plugin
|
||||
* `exec-bash.sh` logs into the plugin's container and runs bash
|
||||
|
||||
## How to test RBD Snapshot feature
|
||||
### How to test RBD Snapshot feature
|
||||
|
||||
Before continuing, make sure you enabled the required
|
||||
feature gate `VolumeSnapshotDataSource=true` in your Kubernetes cluster.
|
||||
@ -42,7 +73,7 @@ In the `examples/rbd` directory you will find two files related to snapshots:
|
||||
[snapshot.yaml](./rbd/snapshot.yaml).
|
||||
|
||||
Once you created your RBD volume, you'll need to customize at least
|
||||
`snapshotclass.yaml` and make sure the `monitors` and `pool` parameters match
|
||||
`snapshotclass.yaml` and make sure the `clusterid` and `pool` parameters match
|
||||
your Ceph cluster setup.
|
||||
If you followed the documentation to create the rbdplugin, you shouldn't
|
||||
have to edit any other file.
|
||||
@ -120,7 +151,7 @@ kubectl create -f pvc-restore.yaml
|
||||
kubectl create -f pod-restore.yaml
|
||||
```
|
||||
|
||||
## How to test RBD MULTI_NODE_MULTI_WRITER BLOCK feature
|
||||
### How to test RBD MULTI_NODE_MULTI_WRITER BLOCK feature
|
||||
|
||||
Requires feature-gates: `BlockVolume=true` `CSIBlockVolume=true`
|
||||
|
||||
@ -218,37 +249,3 @@ Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
|
||||
```
|
||||
|
||||
## Cluster ID based configuration
|
||||
|
||||
Before creating a storage class that uses the option `clusterID` to refer to a
|
||||
Ceph cluster, the following actions need to be completed.
|
||||
|
||||
Get the following information from the Ceph cluster,
|
||||
|
||||
* Admin ID and key, that has privileges to perform CRUD operations on the Ceph
|
||||
cluster and pools of choice
|
||||
* Key is typically the output of, `ceph auth get-key client.admin` where
|
||||
`admin` is the Admin ID
|
||||
* Used to substitute admin/user id and key values in the files below
|
||||
* Ceph monitor list
|
||||
* Typically in the output of `ceph mon dump`
|
||||
* Used to prepare comma separated MON list where required in the files below
|
||||
* Ceph Cluster fsid
|
||||
* If choosing to use the Ceph cluster fsid as the unique value of clusterID,
|
||||
* Output of `ceph fsid`
|
||||
* Used to substitute `<cluster-id>` references in the files below
|
||||
|
||||
Update the template
|
||||
[template-ceph-cluster-ID-secret.yaml](./rbd/template-ceph-cluster-ID-secret.yaml)
|
||||
with values from
|
||||
a Ceph cluster and replace `<cluster-id>` with the chosen clusterID to create
|
||||
the following secret,
|
||||
|
||||
* `kubectl create -f rbd/template-ceph-cluster-ID-secret.yaml`
|
||||
|
||||
Storage class and snapshot class, using `<cluster-id>` as the value for the
|
||||
option `clusterID`, can now be created on the cluster.
|
||||
|
||||
Remaining steps to test functionality remains the same as mentioned in the
|
||||
sections above.
|
||||
|
||||
32
examples/rbd/csi-config-map-sample.yaml
Normal file
32
examples/rbd/csi-config-map-sample.yaml
Normal file
@ -0,0 +1,32 @@
|
||||
---
|
||||
# This is a sample config map that helps define a Ceph cluster configuration
|
||||
# as required by the CSI plugins.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
# The <cluster-id> is used by the CSI plugin to uniquely identify and use a
|
||||
# Ceph cluster, the value MUST match the value provided as `clusterID` in the
|
||||
# StorageClass
|
||||
# The <MONValue#> fields are the various monitor addresses for the Ceph cluster
|
||||
# identified by the <cluster-id>
|
||||
# If a CSI plugin is using more than one Ceph cluster, repeat the section for
|
||||
# each such cluster in use.
|
||||
# To add more clusters or edit MON addresses in an existing config map, use
|
||||
# the `kubectl replace` command.
|
||||
# NOTE: Changes to the config map is automatically updated in the running pods,
|
||||
# thus restarting existing pods using the config map is NOT required on edits
|
||||
# to the config map.
|
||||
data:
|
||||
config.json: |-
|
||||
[
|
||||
{
|
||||
"clusterID": "<cluster-id>",
|
||||
"monitors": [
|
||||
"<MONValue1>",
|
||||
"<MONValue2>",
|
||||
...
|
||||
"<MONValueN>"
|
||||
]
|
||||
}
|
||||
]
|
||||
metadata:
|
||||
name: ceph-csi-config
|
||||
@ -9,6 +9,3 @@ data:
|
||||
admin: BASE64-ENCODED-PASSWORD
|
||||
# Key value corresponds to a user name defined in ceph cluster
|
||||
kubernetes: BASE64-ENCODED-PASSWORD
|
||||
# if monValueFromSecret is set to "monitors", uncomment the
|
||||
# following and set the mon there
|
||||
# monitors: BASE64-ENCODED-Comma-Delimited-Mons
|
||||
|
||||
@ -5,19 +5,16 @@ metadata:
|
||||
name: csi-rbdplugin-snapclass
|
||||
snapshotter: rbd.csi.ceph.com
|
||||
parameters:
|
||||
pool: rbd
|
||||
# Comma separated list of Ceph monitors
|
||||
# if using FQDN, make sure csi plugin's dns policy is appropriate.
|
||||
monitors: mon1:port,mon2:port,...
|
||||
# OR,
|
||||
# String representing a Ceph cluster to provision storage from.
|
||||
# Should be unique across all Ceph clusters in use for provisioning,
|
||||
# cannot be greater than 36 bytes in length, and should remain immutable for
|
||||
# the lifetime of the StorageClass in use.
|
||||
# If using clusterID, ensure to create a secret, as in
|
||||
# template-ceph-cluster-ID-secret.yaml, to accompany the string chosen to
|
||||
# represent the Ceph cluster in clusterID
|
||||
# clusterID: <cluster-id>
|
||||
# Ensure to create an entry in the config map named ceph-csi-config, based on
|
||||
# csi-config-map-sample.yaml, to accompany the string chosen to
|
||||
# represent the Ceph cluster in clusterID below
|
||||
clusterID: <cluster-id>
|
||||
|
||||
pool: rbd
|
||||
|
||||
csi.storage.k8s.io/snapshotter-secret-name: csi-rbd-secret
|
||||
csi.storage.k8s.io/snapshotter-secret-namespace: default
|
||||
|
||||
@ -5,23 +5,14 @@ metadata:
|
||||
name: csi-rbd-sc
|
||||
provisioner: rbd.csi.ceph.com
|
||||
parameters:
|
||||
# Comma separated list of Ceph monitors
|
||||
# if using FQDN, make sure csi plugin's dns policy is appropriate.
|
||||
monitors: mon1:port,mon2:port,...
|
||||
# OR,
|
||||
# String representing a Ceph cluster to provision storage from.
|
||||
# Should be unique across all Ceph clusters in use for provisioning,
|
||||
# cannot be greater than 36 bytes in length, and should remain immutable for
|
||||
# the lifetime of the StorageClass in use.
|
||||
# If using clusterID, ensure to create a secret, as in
|
||||
# template-ceph-cluster-ID-secret.yaml, to accompany the string chosen to
|
||||
# represent the Ceph cluster in clusterID
|
||||
# clusterID: <cluster-id>
|
||||
# OR,
|
||||
# if "monitors" parameter is not set, driver to get monitors from same
|
||||
# secret as admin/user credentials. "monValueFromSecret" provides the
|
||||
# key in the secret whose value is the mons
|
||||
# monValueFromSecret: "monitors"
|
||||
# Ensure to create an entry in the config map named ceph-csi-config, based on
|
||||
# csi-config-map-sample.yaml, to accompany the string chosen to
|
||||
# represent the Ceph cluster in clusterID below
|
||||
clusterID: <cluster-id>
|
||||
|
||||
# Ceph pool into which the RBD image shall be created
|
||||
pool: rbd
|
||||
@ -34,20 +25,15 @@ parameters:
|
||||
imageFeatures: layering
|
||||
|
||||
# The secrets have to contain Ceph admin credentials.
|
||||
# NOTE: If using "clusterID" instead of "monitors" above, the following
|
||||
# secrets MAY be added to the ceph-cluster-<cluster-id> secret and skipped
|
||||
# here
|
||||
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: default
|
||||
csi.storage.k8s.io/node-publish-secret-name: csi-rbd-secret
|
||||
csi.storage.k8s.io/node-publish-secret-namespace: default
|
||||
|
||||
# Ceph users for operating RBD
|
||||
# NOTE: If using "clusterID" instead of "monitors" above, the following
|
||||
# IDs MAY be added to the ceph-cluster-<cluster-id> secret and skipped
|
||||
# here
|
||||
adminid: admin
|
||||
userid: kubernetes
|
||||
|
||||
# uncomment the following to use rbd-nbd as mounter on supported nodes
|
||||
# mounter: rbd-nbd
|
||||
reclaimPolicy: Delete
|
||||
|
||||
@ -1,36 +0,0 @@
|
||||
---
|
||||
# This is a template secret that helps define a Ceph cluster configuration
|
||||
# as required by the CSI driver. This is used when a StorageClass has the
|
||||
# "clusterID" defined as one of the parameters, to provide the CSI instance
|
||||
# Ceph cluster configuration information.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
# The <cluster-id> is used by the CSI plugin to uniquely identify and use a
|
||||
# Ceph cluster, the value MUST match the value provided as `clusterID` in the
|
||||
# StorageClass
|
||||
name: ceph-cluster-<cluster-id>
|
||||
namespace: default
|
||||
data:
|
||||
# Base64 encoded and comma separated Ceph cluster monitor list
|
||||
# - Typically output of: `echo -n "mon1:port,mon2:port,..." | base64`
|
||||
monitors: <BASE64-ENCODED-MONLIST>
|
||||
# Base64 encoded and comma separated list of pool names from which volumes
|
||||
# can be provisioned
|
||||
pools: <BASE64-ENCODED-POOLIST>
|
||||
# Base64 encoded admin ID to use for provisioning
|
||||
# - Typically output of: `echo -n "<admin-id>" | base64`
|
||||
# Substitute the entire string including angle braces, with the base64 value
|
||||
adminid: <BASE64-ENCODED-ID>
|
||||
# Base64 encoded key of the provisioner admin ID
|
||||
# - Output of: `ceph auth get-key client.<admin-id> | base64`
|
||||
# Substitute the entire string including angle braces, with the base64 value
|
||||
adminkey: <BASE64-ENCODED-PASSWORD>
|
||||
# Base64 encoded user ID to use for publishing
|
||||
# - Typically output of: `echo -n "<admin-id>" | base64`
|
||||
# Substitute the entire string including angle braces, with the base64 value
|
||||
userid: <BASE64-ENCODED-ID>
|
||||
# Base64 encoded key of the publisher user ID
|
||||
# - Output of: `ceph auth get-key client.<admin-id> | base64`
|
||||
# Substitute the entire string including angle braces, with the base64 value
|
||||
userkey: <BASE64-ENCODED-PASSWORD>
|
||||
@ -1,33 +0,0 @@
|
||||
---
|
||||
# This is a patch to the existing daemonset deployment of CSI rbdplugin.
|
||||
#
|
||||
# This is to be used when using `clusterID` instead of monitors or
|
||||
# monValueFromSecret in the StorageClass to specify the Ceph cluster to
|
||||
# provision storage from, AND when the value of `--configroot` option to the
|
||||
# CSI pods is NOT "k8s_objects".
|
||||
#
|
||||
# This patch file, patches in the specified secret for the 'clusterID' as a
|
||||
# volume, instead of the Ceph CSI plugin actively fetching and using kubernetes
|
||||
# secrets.
|
||||
#
|
||||
# Post substituting the <cluster-id> in all places execute,
|
||||
# `kubectl patch daemonset csi-rbdplugin --patch\
|
||||
# "$(cat template-csi-rbdplugin-patch.yaml)"`
|
||||
# to patch the daemonset deployment.
|
||||
#
|
||||
# `kubectl patch statefulset csi-rbdplugin-provisioner --patch\
|
||||
# "$(cat template-csi-rbdplugin-patch.yaml)"`
|
||||
# to patch the statefulset deployment.
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: csi-rbdplugin
|
||||
volumeMounts:
|
||||
- name: ceph-cluster-<cluster-id>
|
||||
mountPath: "/etc/csi-config/ceph-cluster-<cluster-id>"
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: ceph-cluster-<cluster-id>
|
||||
secret:
|
||||
secretName: ceph-cluster-<cluster-id>
|
||||
Reference in New Issue
Block a user