# Design Doc for RBD QoS using cgroup v2 ## Introduction The RBD QoS (Quality of Service) design aims to address the issue of IO noisy neighbor problems encountered in early Ceph deployments catering to OpenStack environments. These problems were effectively managed by implementing QEMU throttling at the virtio-blk/scsi level. To further enhance this, capacity-based IOPS were introduced, providing a more dynamic experience similar to public cloud environments. The challenge arises in virtual environments, where a noisy neighbor can lead to performance degradation for other instances sharing the same resources. Although it's uncommon to observe noisy neighbor issues in Kubernetes environments backed by Ceph storage, the possibility exists. The existing QoS support with rbd-nbd doesn't apply to krbd, and as rbd-nbd isn't suitable for container production workloads, a solution is needed for krbd. To mitigate resource starvation issues, setting QoS at the device level through cgroup v2 when enabled becomes crucial. This approach guarantees that I/O capacity isn't overcommitted and is fairly distributed among workloads. ## Dependency * cgroup v2 must be enabled on the Node * We might have Kubernetes dependency as well * Container runtime dependency that supports cgroupv2 ## Manual steps for implementing RBD QoS in a Kubernetes Cluster ```bash [$] ssh root@node1 sh-4.4# chroot /host sh-5.1# cat /proc/partitions major minor #blocks name 259 0 125829120 nvme0n1 259 1 1024 nvme0n1p1 259 2 130048 nvme0n1p2 259 3 393216 nvme0n1p3 259 4 125303791 nvme0n1p4 259 6 52428800 nvme2n1 7 0 536870912 loop0 259 5 536870912 nvme1n1 252 0 52428800 rbd0 sh-5.1# ``` Once the rbd device is mapped on the node we get the device's major and minor number we need to set the io limit on the device but we need to find the right cgroup file where we need to set the limit Kubernetes/Openshift creates a custom cgroup hierarchy for the pods it created but start is `/sys/fs/cgroup` folder ```bash sh-5.1# cd /sys/fs/cgroup/ sh-5.1# ls cgroup.controllers cgroup.subtree_control cpuset.mems.effective io.stat memory.reclaim sys-kernel-debug.mount cgroup.max.depth cgroup.threads dev-hugepages.mount kubepods.slice memory.stat sys-kernel-tracing.mount cgroup.max.descendants cpu.pressure dev-mqueue.mount machine.slice misc.capacity system.slice cgroup.procs cpu.stat init.scope memory.numa_stat sys-fs-fuse-connections.mount user.slice cgroup.stat cpuset.cpus.effective io.pressure memory.pressure sys-kernel-config.mount ``` `kubepods.slice` is the starting point and it contains multiple slices ```bash sh-5.1# cd kubepods.slice sh-5.1# ls cgroup.controllers cpuset.cpus hugetlb.2MB.rsvd.max memory.pressure cgroup.events cpuset.cpus.effective io.bfq.weight memory.reclaim cgroup.freeze cpuset.cpus.partition io.latency memory.stat cgroup.kill cpuset.mems io.max memory.swap.current cgroup.max.depth cpuset.mems.effective io.pressure memory.swap.events cgroup.max.descendants hugetlb.1GB.current io.stat memory.swap.high cgroup.procs hugetlb.1GB.events kubepods-besteffort.slice memory.swap.max cgroup.stat hugetlb.1GB.events.local kubepods-burstable.slice memory.zswap.current cgroup.subtree_control hugetlb.1GB.max kubepods-pod2b38830b_c2d6_4528_8935_b1c08511b1e3.slice memory.zswap.max cgroup.threads hugetlb.1GB.numa_stat memory.current misc.current cgroup.type hugetlb.1GB.rsvd.current memory.events misc.max cpu.idle hugetlb.1GB.rsvd.max memory.events.local pids.current cpu.max hugetlb.2MB.current memory.high pids.events cpu.max.burst hugetlb.2MB.events memory.low pids.max cpu.pressure hugetlb.2MB.events.local memory.max rdma.current cpu.stat hugetlb.2MB.max memory.min rdma.max cpu.weight hugetlb.2MB.numa_stat memory.numa_stat cpu.weight.nice hugetlb.2MB.rsvd.current memory.oom.group ``` Based on the QoS of the pod, either our application pod will end up in the above `kubepods-besteffort.slice` or `kubepods-burstable.slice` or `kubepods.slice` (Guaranteed QoS) cgroup. The 3 QoS classes are defined [here](https://kubernetes.io/docs/concepts/workloads/pods/pod-QoS/#quality-of-service-classes) To identify the right cgroup file, we need pod UUID and container UUID from the `pod yaml` output ```bash [$]kubectl get po csi-rbd-demo-pod -oyaml |grep uid uid: cdf7b785-4eb7-44f7-99cc-ef53890f4dfd [$]kubectl get po csi-rbd-demo-pod -oyaml |grep -i containerID - containerID: cri-o://77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9 [$]kubectl get po csi-rbd-demo-pod -oyaml |grep -i qosClass qosClass: BestEffort ``` Now check in the `kubepods-besteffort.slice` and identify the right path using pod UID and container UID Before that check `io.max` on the application pod and see if there is any limit ```bash [$]kubectl exec -it csi-rbd-demo-pod -- sh sh-4.4# cat /sys/fs/cgroup/io.max sh-4.4# ``` Come back to the Node and find the right cgroup scope ```bash sh-5.1# cd kubepods-besteffort.slice/kubepods-besteffort-podcdf7b785_4eb7_44f7_99cc_ef53890f4dfd.slice/crio-77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9.scope/ sh-5.1# echo "252:0 wbps=1048576" > io.max sh-5.1# cat io.max 252:0 rbps=max wbps=1048576 riops=max wiops=max ``` Now go back to the application pod and check if we have the right limit set ```bash [$]kubectl exec -it csi-rbd-demo-pod -- sh sh-4.4# cat /sys/fs/cgroup/io.max 252:0 rbps=max wbps=1048576 riops=max wiops=max sh-4.4# ``` Note:- We can only support the QoS that cgroup v2 io controller supports, this means that cumulative read+write QoS limits won't be supported. Below are the configurations that will be supported | Parameter | Description | | --- | --- | | MaxReadIOPS | Max read IO operations per second | | MaxWriteIOPS | Max write IO operations per second | | MaxReadBytesPerSecond | Max read bytes per second | | MaxWriteBytesPerSecond | Max write bytes per second | ## Different approaches The above solution can be implemented using 3 different approaches. ### 1. QoS using new parameters in RBD StorageClass ```yaml --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rbd-sc provisioner: rbd.csi.ceph.com parameters: MaxReadIOPS: "" MaxWriteIOPS: "" MaxReadBytesPerSecond: "" MaxWriteBytesPerSecond: "" ``` #### Implementation for StorageClass QoS 1. Create new storageClass with new parameters for QoS 1. Modify CSIDriver object to pass pod details to the NodePublishVolume CSI procedure 1. During NodePublishVolume CSI procedure * Retrieve the QoS configuration from the volumeContext in NodePublishRequest * Identify the rbd device using the NodeStageVolumePath * Get the pod UUID from the NodeStageVolume * Set io.max file in all the containers in the pod #### Drawbacks of StorageClass QoS 1. No way to update the QoS at runtime 1. Need to take a backup and restore to New QoS StorageClass 1. Delete and Recreate the PV object ### 2. QoS using parameters in VolumeAttributeClass ```yaml apiVersion: storage.k8s.io/v1alpha1 kind: VolumeAttributesClass metadata: name: silver parameters: MaxReadIOPS: "" MaxWriteIOPS: "" MaxReadBytesPerSecond: "" MaxWriteBytesPerSecond: "" ``` VolumeAttributesClassName is a new parameter in the PVC object the user can choose from and this can also be updated or removed later. This new VolumeAttributeClass is designed to keep storage that supports setting QoS at the storage level which means setting some configuration at the storage (like QoS for nbd) #### Implementation of VolumeAttributeClass QoS 1. Modify CSIDriver object to pass pod details to the NodePublishVolume CSI procedure 1. Add support in Ceph-CSI to expose ModifyVolume CSI procedure 1. Ceph-CSI will store QoS in the rbd image metadata 1. During NodeStage operation retrieve the image metadata and store it in stagingPath 1. Whenever a new pod comes in apply the QoS #### Drawbacks of VolumeAttributeClass QoS One problem with above is all application need to be scaled downed and scaled up to get the new QoS value even though its changed in the PVC object, this is sometime impossible as it will have downtime. ### 3. QoS using parameters in VolumeAttributeClass with NodePublish Secret 1. Modify CSIDriver object to pass pod details to the NodePublishVolume CSI procedure 1. Add support in Ceph-CSI to expose ModifyVolume CSI procedure 1. Ceph-CSI will store QoS in the rbd image metadata 1. During NodePublishVolume operation retrieve the QoS from image metadata 1. Whenever a new pod comes in apply the QoS This solution addresses the aforementioned issue, but it requires a secret to communicate with the ceph cluster. Therefore, we must create a new PublishSecret for the storageClass, which may be beneficial when Kubernetes eventually enables Node operations. Both options 2 and 3 are contingent upon changes to the CSI spec and Kubernetes support. Additionally, [VolumeAttributeClass](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/3751-volume-attributes-class/README.md) is currently being developed within the Kubernetes realm and will initially be in the Alpha stage. Consequently, it will be disabled by default. #### Advantages of QoS using VolumeAttributeClass 1. No Restore/Clone operation is required to change the QoS 1. Easily QoS can be changed for existing PVC only with second approach not with third as it needs new secret. ### Hybrid Approach Considering the advantages and drawbacks, we can use StorageClass and VolumeAttributeClass to support QoS, with VolumeAttributeClass taking precedence over StorageClass. This approach offers a flexible solution that accounts for dynamic changes while addressing the challenges of existing approaches. ### References Some of the useful links that helped me to understand cgroup v2 and how to set QoS on the device. * [Kubernetes cgroup v2 Architecture](https://kubernetes.io/docs/concepts/architecture/cgroups/) * [cgroup v2 kernel doc](https://docs.kernel.org/admin-guide/cgroup-v2.html) * [ceph RBD QoS tracker](https://tracker.ceph.com/issues/36191) * [cgroup v2 io controller](https://facebookmicrosites.github.io/cgroup2/docs/io-controller.html) * [Kubernetes IOPS issue](https://github.com/kubernetes/kubernetes/issues/92287)