Commit Graph

4382 Commits

Author SHA1 Message Date
Praveen M
6e1c55c52d rbd: add RegenerateVolumeGroupJournal method for Manager interface
This commit adds `RegenerateVolumeGroupJournal` to Manager
interface. RegenerateVolumeGroupJournal regenerate the omap
data for the volume group.

This performs the following operations:
  - extracts clusterID and Mons from the cluster mapping
  - Retrieves pool and journalPool parameters from the VolumeGroupReplicationClass
  - Reserves omap data
  - Add volumeIDs mapping to the reserved volume group omap object
  - Generate new volume group handle

Returns the generated volume group handler.

Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-11-06 18:09:52 +05:30
Praveen M
1a1630d0c5 journal: pass groupUUID to be used for omap name reserve
This commit adds groupUUID param for `ReserveName` to be used for
OMAP name reserve instead of auto-generating.
This is useful for mirroring and metro-DR ensuring that mirrored
resources have consistent OMAP names across mirrored clusters.

Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-11-06 18:09:45 +05:30
Niels de Vos
cea8bf8110 rbd: set SnapshotGroupID on each Snapshot of a VolumeGroupSnapshot
Without the SnapshotGroupID in the Snapshot object, Kubernetes CSI does
not know that the Snapshot belongs to a group. In that case, it allows
the deletion of the Snapshot, which should be denied.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
ec1e7a4ee0 rbd: expose the GroupControllerService
When the GroupSnapGetInfo go-ceph function is supported by librbd, the
Group Controller Servive and VolumeGroupSnapshot capabilities can be
exposed to the Container Orchestrator.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
e34dceff27 rbd: implement CSI Group Controller Server
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
e011e74b9d rbd: fix snapshot deletion by resolving image names correctly
When creating a Snapshot with the new NewSnapshotByID() function, the
name of the RBD-image that is created is the same as the name of the
Snapshot. The `RbdImageName` points to the name of parent image, which
causes deleting the Snapshot to delete the parent image instead.

Correcting the `RbdImageName` and setting it to the `RbdSnapName` makes
sure that upon deletion, the Snapshot RBD-image is removed, and not the
parent image.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
fdccba1f33 rbd: add Manager.GetVolumeGroupSnapshotByName
The Group Controller Server may need to fetch a VolumeGroupSnapshot that
was statically provisioned. In that case, only the name of the
VolumeGroupSnapshot is known and should be resolved to an object.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
ad381c4ff0 rbd: implement Manager.GetVolumeGroupSnapshotByID
The GetVolumeGroupSnapshotByID function makes it possible to get a
VolumeGroupSnapshot object from the Manager by passing a request-id.
This makes it simple for the Group Controller Server to check if a
VolumeGroupSnapshot already exists, so it is not needed to try and
re-create an existing one.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
7563f4285d rbd: add manager.CreateVolumeGroupSnapshot()
Implement the CreateVolumeGroupSnapshot for the rbd.Manager. A Group
Controller Server can use the rbd.Manager to create VolumeGroupSnapshots
in an easy an idempotent way.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
9bea3feff1 rbd: add manager GetSnapshotByID and SnapshotResolver interface
A (CSI) VolumeGroupSnapshot object contains references to Snapshot IDs
(or CSI Snapshot handles). In order to work with a VolumeGroupSnapshot
struct, the Snapshot IDs need to be resolved into rbdSnapshot structs.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
455a90e9f4 rbd: add VolumeGroupSnapshot type
The VolumeGroupSnapshot type will be used by the rbd.Manager to create,
inspect and delete VolumeGroupSnapshos.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
efb7bccaea rbd: add VolumeGroup.CreateSnapshots() implementation
When the rbd.Manager creates a VolumeGroupSnapshot, each RBD-snapshot
that is created as part of the RBD-group needs to be cloned into its own
RBD-image that will be used as a CSI Snapshot.

The VolumeGroup.CreateSnapshots() creates the RBD-group snapshot and
returns a list of the Snapshot structs.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
20fadf2016 rbd: add rbdVolume.NewSnapshotByID to clone images by RBD snapshot-id
The NewSnapshotByID() function makes it possible to clone a new Snapshot
from an existing RBD-image and the ID of an RBD-snapshot on that image.

This will be used by the VolumeGroupSnapshot feature, where the ID of an
RBD-snapshot is obtained for the RBD-snapshot on the RBD-images.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
9808408340 rbd: pass CSI-drivername to volume group instead of journal instance
Each object is responsible for maintaining a connection to the journal.

By sharing a single journal, cleanup of objects becomes more complex as
the journal is used in deferred functions and only the last should
destroy the journal connection resources.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
29bf5797b0 rbd: add .requestName to the commonVolumeGroup struct
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
4b13e9132b rbd: have GetVolumeGroup() return an empty volume group if it was not found
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
5d5171c7d7 journal: store csi.groupid for snapshots
Commit 95733b3a9 introduced the `StoreGroupID()` function, but that
unfortunately set an empty key in the journal.

Passing the `csiGroupIDKey` key (with value `csi.groupid`) caused
setting `csi.csi.groupid` as a key. Reading the value back with the
right `csi.groupid` key always returned an empty value.

Fixes: 95733b3a9 "journal: add option to store the groupID"
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Niels de Vos
6d88e0a4c7 rbd: close the RBD-image after adding it to a VolumeGroup
When the image is not closed, it keeps a watch open. This prevents the
CSI Controller to delete the Volume, as there is still a user of it.

Fixes: f9ab14e826 "rbd: check if an image is part of a group before adding it"
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-06 11:37:44 +00:00
Madhu Rajanna
b4592a55eb rbd: parse IP address
The address we get from ceph
contains the ip in the format
of 10.244.0.1:0/2686266785 we
need to extract the client IP
from this address, we already
have a helper to extract it,
This makes the helper more generic
can be reused by multiple packages
in the fence controller.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-06 09:48:45 +00:00
Madhu Rajanna
facf805941 rbd: implement GetFenceClients
implemented GetFenceClients which
connects to the ceph cluster and
returns the ceph clusterID and the
clientaddress that is used for rados
connection.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-06 09:48:45 +00:00
Madhu Rajanna
ba8c5a359c util: add GetAddrs method
added GetAddrs to get the client
Adress of the rados connection
which is helpful for NetworkFencing

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-06 09:48:45 +00:00
Madhu Rajanna
1246e2fac7 rebase: update go-ceph to latest commit
updating go-ceph to the latest commit to
pull the new function to get client address

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-06 09:48:45 +00:00
Madhu Rajanna
fdc74973d8 rbd: register GET_CLIENTS_TO_FENCE caps
register Capability_NetworkFence_
GET_CLIENTS_TO_FENCE capability and
start a NetworkFence controllers
as part of rbd nodeplugin.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-06 09:48:45 +00:00
Madhu Rajanna
34dfd0adb8 rebase: update csi-addons spec
updating csi-addons spec to the
latest main which GetFenceClients
API.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-06 09:48:45 +00:00
dependabot[bot]
7b06b0f218 rebase: bump the github-dependencies group with 2 updates
Bumps the github-dependencies group with 2 updates: [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) and [github.com/onsi/gomega](https://github.com/onsi/gomega).


Updates `github.com/onsi/ginkgo/v2` from 2.20.2 to 2.21.0
- [Release notes](https://github.com/onsi/ginkgo/releases)
- [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/ginkgo/compare/v2.20.2...v2.21.0)

Updates `github.com/onsi/gomega` from 1.34.2 to 1.35.1
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.34.2...v1.35.1)

---
updated-dependencies:
- dependency-name: github.com/onsi/ginkgo/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-05 13:06:23 +00:00
Niraj Yadav
1c02e69ba4 rbd: Add timeout for cryptsetup commands
This PR modifies the execCryptSetupCommand so that
the process is killed in an event of lock timeout.

Useful in cases where the volume lock is released but
the command is still running.

Signed-off-by: Niraj Yadav <niryadav@redhat.com>
2024-11-05 11:39:59 +00:00
Niels de Vos
c451997762 e2e: detect support for VolumeGroupSnapshot in librbd
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-04 15:38:18 +00:00
Niels de Vos
b59a701777 e2e: add tests for RBD VolumeGroupSnapshots
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-11-04 15:38:18 +00:00
Madhu Rajanna
f4f03044f2 e2e: enhance volumegroupsnapshot test
update the e2e for volumegroupsnapshot
to create application pods from clone pvc
and delete the pods once we are able to
create the pods.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-11-04 10:28:27 +00:00
dependabot[bot]
a7959d4721 rebase: bump k8s.io/api in /api in the k8s-dependencies group
Bumps the k8s-dependencies group in /api with 1 update: [k8s.io/api](https://github.com/kubernetes/api).

Updates `k8s.io/api` from 0.31.1 to 0.31.2
- [Commits](https://github.com/kubernetes/api/compare/v0.31.1...v0.31.2)

---
updated-dependencies:
- dependency-name: k8s.io/api
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-10-31 10:59:08 +00:00
Madhu Rajanna
00c7fb3136 e2e: log stderror in execCommandInPodAndAllowFail
If the command we try to execute fails, log
the stdError along with error for better debugging.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-10-29 17:30:49 +00:00
Madhu Rajanna
98471b27fc e2e: add a test case of ROX PVC-PVC clone
adding a test case to test below case for RBD

* Create PVC and application
* Scale down application pod
* Create PVC-PVC ROX clone
* Try to create many pods that use the ROX PVC
* Try to write the data and verify only read
access.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-10-29 17:30:49 +00:00
yati1998
a6f1af49c6 doc: resturcture the doc folder
This commit is the restructure the doc folder to
include sub-folders like csi-addons, rbd, cephfs
to contain related docs and other general docs can be
placed under doc folder.
This will enhance the doc structure will make it easier
for the users to search the docs as it get more populated.

Signed-off-by: yati1998 <ypadia@redhat.com>
2024-10-29 15:21:36 +00:00
dependabot[bot]
f9496dbe45 rebase: bump the k8s-dependencies group with 2 updates
Bumps the k8s-dependencies group with 2 updates: [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) and [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime).


Updates `k8s.io/kubernetes` from 1.31.1 to 1.31.2
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.31.1...v1.31.2)

Updates `sigs.k8s.io/controller-runtime` from 0.19.0 to 0.19.1
- [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases)
- [Changelog](https://github.com/kubernetes-sigs/controller-runtime/blob/main/RELEASE.md)
- [Commits](https://github.com/kubernetes-sigs/controller-runtime/compare/v0.19.0...v0.19.1)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
- dependency-name: sigs.k8s.io/controller-runtime
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-29 15:15:41 +00:00
dependabot[bot]
26c237e314 rebase: bump the github-dependencies group with 2 updates
Bumps the github-dependencies group with 2 updates: [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) and [github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets](https://github.com/Azure/azure-sdk-for-go).


Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.32.2 to 1.32.3
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.32.2...v1.32.3)

Updates `github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets` from 1.1.0 to 1.2.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/v1.1...v1.2)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-dependencies
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-29 07:49:34 +00:00
dependabot[bot]
c3c00b0e61 rebase: bump github/codeql-action from 3.26.13 to 3.27.0
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.26.13 to 3.27.0.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](f779452ac5...662472033e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-29 07:10:48 +00:00
dependabot[bot]
2f0af46080 rebase: bump actions/dependency-review-action from 4.3.4 to 4.4.0
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 4.3.4 to 4.4.0.
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](5a2ce3f5b9...4081bf99e2)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-29 06:40:42 +00:00
dependabot[bot]
710e3340ee rebase: bump actions/checkout from 4.2.1 to 4.2.2
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](eef61447b9...11bd71901b)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-29 06:09:06 +00:00
dependabot[bot]
52fcb7b00c rebase: bump github.com/prometheus/client_golang
Bumps the github-dependencies group with 1 update: [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang).


Updates `github.com/prometheus/client_golang` from 1.20.4 to 1.20.5
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.20.4...v1.20.5)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-22 17:10:57 +00:00
Niels de Vos
512bba485b build: remove conflicting protobuf package
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-10-22 13:53:10 +00:00
dependabot[bot]
9152b4c2c7 rebase: bump github.com/ceph/go-ceph
Bumps [github.com/ceph/go-ceph](https://github.com/ceph/go-ceph) from 0.29.1-0.20240925141413-065319c78733 to 0.30.0.
- [Release notes](https://github.com/ceph/go-ceph/releases)
- [Changelog](https://github.com/ceph/go-ceph/blob/master/docs/release-process.md)
- [Commits](https://github.com/ceph/go-ceph/commits/v0.30.0)

---
updated-dependencies:
- dependency-name: github.com/ceph/go-ceph
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-22 13:53:10 +00:00
Praveen M
3bcf6afe30 helm: add example for cephFS radosnamespace
Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-10-21 14:11:27 +00:00
Praveen M
3aa91cfa6f e2e: cephfs rados namespace test
Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-10-21 14:11:27 +00:00
Praveen M
86759d4653 cephfs: support omap store in radosnamespace
This commit adds the support for storing the CephFS omap data
in a namespace specified in the ceph-csi-config ConfigMap under
cephFS.radosNamespace field.

If the radosNamespace is not set, the default radosNamespace will
be used i.e, csi.

Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-10-21 14:11:27 +00:00
Praveen M
c7f41cf84b util: add GetCephFSRadosNamespace method
This commit adds `GetCephFSRadosNamespace` util method that returns
the `RadosNamespace` specified in ceph-csi-config ConfigMap under
cephFS.radosNamespace.

If not specified, the method returns the default RadosNamespace
i.e, csi.

Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-10-21 14:11:27 +00:00
Praveen M
178b642b01 deploy: add RadosNamespace field for CephFS
This change introduces a `RadosNamespace` field to the CephFS struct.
The RadosNamespace field specifies the RADOS namespace for storing
the omap data of CephFS subvolumes.

Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-10-21 14:11:27 +00:00
Niels de Vos
a51a6ae43a rbd: add types.Snapshot interface
The rbdSnapshot/rbdImage object implements all functions for a useful
Snapshot interface. The rbd.Manager will be able to use this for
providing VolumeGroupSnapshot support.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-10-17 16:30:33 +00:00
Niels de Vos
f885c77f4e rbd: use GetCreationTime() to build the CSI-Snapshot object
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-10-17 16:30:33 +00:00
Niels de Vos
8ddb615df2 build: no need to use the CentOS 8 archived repositories on CentOS 9
The Ceph base container-image moved to CentOS Stream 9, so there is no
need to adapt the repositories anymore.

Closes: #4659
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-10-16 16:17:04 +00:00
dependabot[bot]
e75fec74f9 rebase: bump the github-dependencies group with 2 updates
Bumps the github-dependencies group with 2 updates: [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) and [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go).


Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.32.1 to 1.32.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.32.1...v1.32.2)

Updates `github.com/Azure/azure-sdk-for-go/sdk/azidentity` from 1.7.0 to 1.8.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.7.0...sdk/azcore/v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-dependencies
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-16 15:47:04 +00:00