mirrors/ceph-csi
1
0
Fork 0
mirror of https://github.com/ceph/ceph-csi.git synced 2025-06-13 10:33:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

doc: resturcture the doc folder

Browse Source 
This commit is the restructure the doc folder to
include sub-folders like csi-addons, rbd, cephfs
to contain related docs and other general docs can be
placed under doc folder.
This will enhance the doc structure will make it easier
for the users to search the docs as it get more populated.

Signed-off-by: yati1998 <ypadia@redhat.com>
This commit is contained in:
yati1998
2024-10-14 17:42:02 +05:30
committed by mergify[bot]
parent f9496dbe45
commit a6f1af49c6
8 changed files with 2 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
docs/design/proposals/ceph-mount-corruption.md Normal file
View File

@ -0,0 +1,92 @@
# Ceph mount corruption detection and recover
## ceph-fuse: detection of corrupted mounts and their recovery
Mounts managed by ceph-fuse may get corrupted by e.g. the ceph-fuse process
exiting abruptly, or its parent Node Plugin container being terminated, taking
down its child processes with it.
This may manifest in concerned workloads like so:
```
# mount | grep fuse
ceph-fuse on /cephfs-share type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
# ls /cephfs-share
ls: /cephfs-share: Socket not connected
```
or,
```
# stat /home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount: transport endpoint is not connected
```
This feature allows CSI CephFS plugin to be able to detect if a ceph-fuse mount
is corrupted during the volume publishing phase, and will attempt to recover it
for the newly scheduled pod. Pods that already reside on a node whose
ceph-fuse mountpoints were broken may still need to be restarted, however.
### ceph-fuse corruption detection
A mountpoint is deemed corrupted if `stat()`-ing it returns one of the
following errors:
* `ENOTCONN`
* `ESTALE`
* `EIO`
* `EACCES`
* `EHOSTDOWN`
### ceph-fuse recovery
Once a mountpoint corruption is detected, its recovery is performed by
remounting the volume associated with it.
Recovery is attempted only if `/csi/mountinfo` directory is made available to
CSI CephFS plugin (available by default in the Helm chart and Kubernetes
manifests).
## kernel client: detection of corrupted mounts and their recovery
Mounts managed by ceph-kernel may get corrupted e.g. if your network
connection is disrupted for a long enough time, the client will be forcibly
disconnected from the system. More details can be found
[here](https://docs.ceph.com/en/quincy/cephfs/troubleshooting/#disconnected-remounted-fs)
The above case may manifest in concerned workloads like so:
```
# mount | grep ceph
10.102.104.172:6789:/volumes/csi/csi-vol-7fed1ce7-97cf-43ef-9b84-2a49ab992515/d61be75e-74ae-428c-a5d1-48f79d1d3c8c on /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/bc0146ec2b5d9a9db62e698abbe0adcae19c0e01f5cf15d3d593ed33c7bc1a8d/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.102.104.172:6789:/volumes/csi/csi-vol-7fed1ce7-97cf-43ef-9b84-2a49ab992515/d61be75e-74ae-428c-a5d1-48f79d1d3c8c on /var/lib/kubelet/pods/8087df68-9756-4f38-86ef-6c81e1075607/volumes/kubernetes.io~csi/pvc-15e63d0a-77de-4886-8d0f-516f9fecbeb4/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)# ls /cephfs-share
sh-4.4# ls /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/bc0146ec2b5d9a9db62e698abbe0adcae19c0e01f5cf15d3d593ed33c7bc1a8d/globalmount
ls: cannot access '/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/bc0146ec2b5d9a9db62e698abbe0adcae19c0e01f5cf15d3d593ed33c7bc1a8d/globalmount': Permission denied
```
### kernel client corruption detection
A mountpoint is deemed corrupted if `stat()`-ing it returns one of the
following errors:
* `ENOTCONN`
* `ESTALE`
* `EIO`
* `EACCES`
* `EHOSTDOWN`
More details about the error codes can be found [here](https://www.gnu.org/software/libc/manual/html_node/Error-Codes.html)
For such mounts, The CephCSI nodeplugin returns volume_condition as
abnormal for `NodeGetVolumeStats` RPC call.
### kernel client recovery
Once a mountpoint corruption is detected,
Below are the two methods to recover from it.
* Reboot the node where the abnormal volume behavior is observed.
* Scale down all the applications using the CephFS PVC
on the node where abnormal mounts are present.
Once all the applications are deleted, scale up the application
to remount the CephFS PVC to application pods.

62
docs/design/proposals/cephfs-snapshot-backed-volumes.md Normal file
View File

@ -0,0 +1,62 @@
# Provisioning and mounting CephFS snapshot-backed volumes
Snapshot-backed volumes allow CephFS subvolume snapshots to be exposed as
regular read-only PVCs. No data cloning is performed and provisioning such
volumes is done in constant time.
For more details please refer to [Snapshots as shallow read-only volumes](./design/proposals/cephfs-snapshot-shallow-ro-vol.md)
design document.
## Prerequisites
Prerequisites for this feature are the same as for creating PVCs with snapshot
volume source. See [Create snapshot and Clone Volume](./snap-clone.md) for more
information.
## Usage
### Provisioning a snapshot-backed volume from a volume snapshot
For provisioning new snapshot-backed volumes, following configuration must be
set for PVCs:
* PersistentVolumeClaim:
* Set `storageClassName` to point to your existing cephFS storage class.
* Define `spec.dataSource` for your desired source volume snapshot.
* Set `spec.accessModes` to `ReadOnlyMany`. This is the only access mode that
is supported by this feature.
Note:- We can also disable shallowVolume by setting `backingSnapshot: "false"
in cephFS storageclass. If the value is set in the storageclass when requested
for `ReadOnlyMany` PVC a clone will get created in ceph cluster.
### Mounting snapshots from pre-provisioned volumes
Steps for defining a PersistentVolume and PersistentVolumeClaim for
pre-provisioned CephFS subvolumes are identical to those described in
[Static PVC with ceph-csi](./static-pvc.md), except one additional parameter
must be specified: `backingSnapshotID`. CephFS-CSI driver will retrieve the
snapshot identified by the given ID from within the specified subvolume, and
expose it to workloads in read-only mode. Volume access mode must be set to
`ReadOnlyMany`.
Note that the snapshot retrieval is done by traversing `<rootPath>/.snap` and
searching for a directory that contains `backingSnapshotID` value in its name.
The specified snapshot ID does not necessarily need to be the complete directory
name inside `<rootPath>/.snap`, however it must be complete enough to uniquely
identify that directory.
Example:
```
$ ls .snap
_f279df14-6729-4342-b82f-166f45204233_1099511628283
_a364870e-6729-4342-b82f-166f45204233_1099635085072
```
`f279df14-6729-4342-b82f-166f45204233` would be considered a valid value for
`backingSnapshotID` volume parameter, whereas `6729-4342-b82f-166f45204233`
would not, as it would be ambiguous.
If the given snapshot ID is ambiguous, or no such snapshot is found, mounting
the PVC will fail with INVALID_ARGUMENT error code.

99
docs/design/proposals/rbd-nbd.md Normal file
View File

@ -0,0 +1,99 @@
# RBD NBD Mounter
- [RBD NBD Mounter](#rbd-nbd-mounter)
- [Overview](#overview)
- [Configuration](#configuration)
- [Configuring logging path](#configuring-logging-path)
- [Status](#status)
- [Support Matrix](#support-matrix)
- [CSI spec and Kubernetes version compatibility](#csi-spec-and-kubernetes-version-compatibility)
## Overview
The RBD CSI plugin will provision new RBD images and attach and mount those
to workloads. Currently, the default mounter is krbd, which uses the kernel
rbd driver to mount the rbd images onto the application node. Here on
at ceph-csi we will also have a userspace way of mounting the RBD images,
via RBD-NBD.
[Rbd-nbd](https://docs.ceph.com/en/latest/man/8/rbd-nbd/) is a client for
RADOS block device (rbd) images like the existing rbd kernel module. It
will map an rbd image to an NBD (Network Block Device) device, allowing
access to it as a regular local block device.
It’s worth to make a note that the rbd-nbd processes will run on the
client-side, which is inside the `csi-rbdplugin` node plugin.
## Configuration
To use the rbd-nbd mounter for RBD-backed PVs, set `mounter` to `rbd-nbd`
in the StorageClass.
Please note that the minimum recommended kernel version to use rbd-nbd is
5.4 or higher.
### Configuring logging path
If you are using the default rbd nodePlugin DaemonSet and StorageClass
templates then `cephLogDir` will be `/var/log/ceph`, this directory will be
a host-path and the default log file path will be
`/var/log/ceph/rbd-nbd-<volID>.log`. rbd-nbd creates a log file per volume
under the `cephLogDir` path on NodeStage(map) and removed the same on
the respective NodeUnstage(unmap).
- There are different strategies to maintain the logs
- `remove`: delete log file on unmap/detach (default behaviour)
- `compress`: compress the log file to gzip on unmap/detach, in case there
exists a `.gz` file from previous map/unmap of the same volume, then
override the previous log with new log.
- `preserve`: preserve the log file in text format
You can tweak the log strategies through `cephLogStrategy` option from the
storageclass yaml
- In case if you need a customized log path, you should do below:
- Edit the DaemonSet templates to change the ceph log directory host-path
- If you are using helm charts, then you can use key `cephLogDirHostPath`
```
helm install --set cephLogDirHostPath=/var/log/ceph-csi/my-dir
```
- For standard templates edit [csi-rbdplugin.yaml](../deploy/rbd/kubernetes/csi-rbdplugin.yaml)
to update `hostPath` for `ceph-logdir`.
to update `pathPrefix` spec entries.
- Update the StorageClass with the customized log directory path
- Now update rbd StorageClass for `cephLogDir`, for example
```
cephLogDir: "/var/log/prod-A-logs"
```
`NOTE`:
- On uninstall make sure to delete `cephLogDir` on host manually to freeup
some space just in case if there are any uncleaned log files.
- In case if you do not need the rbd-nbd logging to persistent at all, then
simply update the StorageClass for `cephLogDir` to use a non-persistent path.
## Status
Rbd-nbd support status: **Alpha**
## Support Matrix
| Features | Feature Status | CSI Driver Version | Ceph Cluster Version | CSI Spec Version | Kubernetes Version |
| ---------------------------------------- | -------------- | ------------------ | -------------------- | ---------------- | ------------------ |
| Creating and deleting snapshot | Alpha | >= v3.4.0 | Pacific (>=16.0.0) | >= v1.0.0 | >= v1.17.0 |
| Creating and deleting clones | Alpha | >= v3.4.0 | Pacific (>=16.0.0) | >= v1.0.0 | >= v1.17.0 |
| Creating and deleting encrypted volumes | Alpha | >= v3.4.0 | Pacific (>=16.0.0) | >= v1.0.0 | >= v1.14.0 |
| Expand volumes | Alpha | >= v3.4.0 | Pacific (>=16.0.0) | >= v1.1.0 | >= v1.15.0 |
`NOTE`: The `Alpha` status reflects possible non-backward compatible
changes in the future, and is thus not recommended for production use.
### CSI spec and Kubernetes version compatibility
Please refer to the [matrix](https://kubernetes-csi.github.io/docs/#kubernetes-releases)
in the Kubernetes documentation.