mirror of
https://github.com/ceph/ceph-csi.git
synced 2024-12-18 19:10:21 +00:00
103 lines
4.9 KiB
Markdown
103 lines
4.9 KiB
Markdown
|
# RBD NBD VOLUME HEALER
|
|||
|
|
|
|||
|
|
- [RBD NBD VOLUME HEALER](#rbd-nbd-volume-healer)
|
|||
|
|
- [Rbd Nbd](#rbd-nbd)
|
|||
|
|
- [Advantages of userspace mounters](#advantages-of-userspace-mounters)
|
|||
|
|
- [Side effects of userspace mounters](#side-effects-of-userspace-mounters)
|
|||
|
|
- [Volume Healer](#volume-healer)
|
|||
|
|
- [More thoughts](#more-thoughts)
|
|||
|
|
|
|||
|
|
## Rbd nbd
|
|||
|
|
|
|||
|
|
The rbd CSI plugin will provision new rbd images and attach and mount those
|
|||
|
|
to workloads. Currently, the default mounter is krbd, which uses the kernel
|
|||
|
|
rbd driver to mount the rbd images onto the application pod. Here on
|
|||
|
|
at Ceph-CSI we will also have a userspace way of mounting the rbd images,
|
|||
|
|
via rbd-nbd.
|
|||
|
|
|
|||
|
|
[Rbd-nbd](https://docs.ceph.com/en/latest/man/8/rbd-nbd/) is a client for
|
|||
|
|
RADOS block device (rbd) images like the existing rbd kernel module. It
|
|||
|
|
will map an rbd image to an nbd (Network Block Device) device, allowing
|
|||
|
|
access to it as a regular local block device.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
It’s worth making a note that the rbd-nbd processes will run on the
|
|||
|
|
client-side, which is inside the `csi-rbdplugin` node plugin.
|
|||
|
|
|
|||
|
|
### Advantages of userspace mounters
|
|||
|
|
|
|||
|
|
- It is easier to add features to rbd-nbd as it is released regularly with
|
|||
|
|
Ceph, and more difficult and time consuming to add features to the kernel
|
|||
|
|
rbd module as that is part of the Linux kernel release schedule.
|
|||
|
|
- Container upgrades will be independent of the host node, which means if
|
|||
|
|
there are any new features with rbd-nbd, we don’t have to reboot the node
|
|||
|
|
as the changes will be shipped inside the container.
|
|||
|
|
- Because the container upgrades are host node independent, we will be a
|
|||
|
|
better citizen in K8s by switching to the userspace model.
|
|||
|
|
- Unlike krbd, rbd-nbd uses librbd user-space library that gets most of the
|
|||
|
|
development focus, and hence rbd-nbd will be feature-rich.
|
|||
|
|
- Being entirely kernel space impacts fault-tolerance as any kernel panic
|
|||
|
|
affects a whole node not only a single pod that is using rbd storage.
|
|||
|
|
Thanks to the rbd-nbd’s userspace design, we are less bothered here, the
|
|||
|
|
krbd is a complete kernel and vendor-specific driver which needs changes
|
|||
|
|
on every feature basis, on the other hand, rbd-nbd depends on NBD generic
|
|||
|
|
driver, while all the vendor-specific logic sits in the userspace. It's
|
|||
|
|
worth taking note that NBD generic driver is mostly unchanged much from
|
|||
|
|
years and consider it to be much stable. Also given NBD is a generic
|
|||
|
|
driver there will be many eyes on it compared to the rbd driver.
|
|||
|
|
|
|||
|
|
### Side effects of userspace mounters
|
|||
|
|
|
|||
|
|
Since the rbd-nbd processes run per volume map on the client side i.e.
|
|||
|
|
inside the `csi-rbdplugin` node plugin, a restart of the node plugin will
|
|||
|
|
terminate all the rbd-nbd processes, and there is no way to restore
|
|||
|
|
these processes back to life currently, which could lead to IO errors
|
|||
|
|
on all the application pods.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
This is where the Volume healer could help.
|
|||
|
|
|
|||
|
|
## Volume healer
|
|||
|
|
|
|||
|
|
Volume healer runs on the start of rbd node plugin and runs within the
|
|||
|
|
node plugin driver context.
|
|||
|
|
|
|||
|
|
Volume healer does the below,
|
|||
|
|
|
|||
|
|
- Get the Volume attachment list for the current node where it is running
|
|||
|
|
- Filter the volume attachments list through matching driver name and
|
|||
|
|
status attached
|
|||
|
|
- For each volume attachment get the respective PV information and check
|
|||
|
|
the criteria of PV Bound, mounter type
|
|||
|
|
- Build the StagingPath where rbd images PVC is mounted, based on the
|
|||
|
|
KUBELET path and PV object
|
|||
|
|
- Construct the NodeStageVolume() request and send Request to CSI Driver.
|
|||
|
|
- The NodeStageVolume() has a way to identify calls received from the
|
|||
|
|
healer and when executed from the healer context, it just runs in the
|
|||
|
|
minimal required form, where it fetches the previously mapped device to
|
|||
|
|
the image, and the respective secrets and finally ensures to bringup the
|
|||
|
|
respective process back to life. Thus enabling IO to continue.
|
|||
|
|
|
|||
|
|
### More thoughts
|
|||
|
|
|
|||
|
|
- Currently the NodeStageVolume() call is safeguarded by the global Ceph-CSI
|
|||
|
|
level lock (per volID) that needs to be acquired before doing any of the
|
|||
|
|
NodeStage, NodeUnstage, NodePublish, NodeUnPulish operations. Hence none
|
|||
|
|
of the operations happen in parallel.
|
|||
|
|
- Any issues if the NodeUnstage is issued by kubelet?
|
|||
|
|
- This can not be a problem as we take a lock at the Ceph-CSI level
|
|||
|
|
- If the NodeUnstage success, Ceph-CSI will return StagingPath not found
|
|||
|
|
error, we can then skip
|
|||
|
|
- If the NodeUnstage fails with an operation already going on, in the
|
|||
|
|
next NodeUnstage the volume gets unmounted
|
|||
|
|
- What if the PVC is deleted?
|
|||
|
|
- If the PVC is deleted, the volume attachment list might already got
|
|||
|
|
refreshed and entry will be skipped/deleted at the healer.
|
|||
|
|
- For any reason, If the request bails out with Error NotFound, skip the
|
|||
|
|
PVC, assuming it might have deleted or the NodeUnstage might have
|
|||
|
|
already happened.
|
|||
|
|
- The Volume healer currently works with rbd-nbd, but the design can
|
|||
|
|
accommodate other userspace mounters (may be ceph-fuse).
|