Commit Graph

3319 Commits

Author SHA1 Message Date
Humble Chirammal
3c9d7e3cd5 rbd: detect migration volID in DeleteVolume() and delete rbd image
This commit adds the logic to detect a passed in volumeID
is a migrated volume ID and if yes, the driver connect to the
backend cluster and clean/delete the image. The logic
only applied if its a migration volume ID. The migration volume ID
carry the information like mons, pool and image name which is
good enough for the driver to identify and connect to the backend
cluster for its operations.

migration volID format:
<mig>_mons-<monsHash>_image-<imageUID>_<poolHash>

Details on the hash values:

* MonsHash: this carry a hash value (md5sum) which will be acted as the
`clusterID` for the operations in this context.

* ImageUID: this is the unique UUID generated by kubernetes for the created
volume.

* PoolHash: this is an encoded string of pool name.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Humble Chirammal
b778fe51a4 e2e: add test for migration volID detection and delete of image
This commit add test for migration delete volID detection scenario
by passing a custom volID and with the entries in configmap changed
to simulate the situation. The staticPV function also changed its
accept the annotation map which make it more general usage.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Humble Chirammal
1171111a94 e2e: deletePodWithLabel fails on unparam linter
this commit address the unparam linter error on deletePodWithLabel
function.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Humble Chirammal
a4a2dc93c1 e2e: change createCustomConfigmap to be more general
createCustomConfigmap helps to create a custom cluster entry in
the configmap, however this was coupled with subvolumegroup filling
in the cluster configuration. This commit helps to make it more
general and the subvolumegroup filling is controlled now with a flag

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Rakshith R
90d246fc55 doc: update dev standup meeting link
Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-30 13:27:11 +00:00
Niels de Vos
dfc8f64bdd ci: require passing of k8s-e2e-external-storage jobs for merge
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-29 12:45:02 +05:30
Niels de Vos
bb68cc9bee rebase: update vault/api to v1.1.1
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-29 05:55:26 +00:00
dependabot[bot]
b85076365c rebase: bump google.golang.org/grpc from 1.40.0 to 1.41.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.40.0 to 1.41.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.40.0...v1.41.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-28 11:36:21 +00:00
Niels de Vos
3480cb2c25 rebase: update minikube to v1.23.2
See-also: https://github.com/kubernetes/minikube/releases/tag/v1.23.2
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-28 09:28:28 +00:00
dependabot[bot]
27a2718f0b rebase: bump sigs.k8s.io/controller-runtime from 0.10.0 to 0.10.1
Bumps [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases)
- [Commits](https://github.com/kubernetes-sigs/controller-runtime/compare/v0.10.0...v0.10.1)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/controller-runtime
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-28 06:38:08 +00:00
dependabot[bot]
f0e2f84a40 rebase: bump github.com/aws/aws-sdk-go from 1.40.46 to 1.40.50
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.46 to 1.40.50.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.46...v1.40.50)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-28 04:29:46 +00:00
Yati Padia
1cf14cd83c cleanup: rework on naming conventions
This commits replaces cephfs -> cephFS
to maintain consistency throughout the
codebase

Updates: #1465

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-09-24 06:17:17 +00:00
Madhu Rajanna
34a21cdbe3 cleanup: move mount functions to new pkg
moved fuse and kernel mount functions
to a new package.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-23 06:39:37 +00:00
Madhu Rajanna
b1ef842640 cleanup: move core functions to core pkg
as we are refractoring the cephfs code,
Moving all the core functions to a new folder
/pkg called core. This will make things easier
to implement. For now onwards all the core
functionalities will be added to the core
package.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-23 06:39:37 +00:00
dependabot[bot]
64ade1d4c3 rebase: bump github.com/aws/aws-sdk-go from 1.40.34 to 1.40.46
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.34 to 1.40.46.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.34...v1.40.46)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-23 04:47:17 +00:00
Madhu Rajanna
2a8eb95966 ci: disable fsGroupChangePolicy change test for cephfs
disabling fsGroupChangePolicy test suite for
cephfs.

updates: #2017

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 10:34:41 +00:00
Madhu Rajanna
981835a073 deploy: remove extra permission for snapshot controller
removed extra PSP permissions added for the
snapshot controller deployment

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
e45bf03bd8 helm: remove securityContext from cephfs deployment
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations like mount,
selinux operations etc .

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
f267d77801 helm: remove extra volumes from rbd plugin PSP
removed extra volume permissions from the rbd
nodeplugin PSP

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
6f1066fd36 helm: reduce the PSP permission for rbd deployment
rbd deployment doesnot need extra permission like
privileged and extra volumes etc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
c154886926 helm: remove extra volumes from cephfs plugin PSP
removed extra volume permissions from the cephfs
nodeplugin PSP.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
8374fa929a helm: reduce the PSP permission for cephfs deployment
cephfs deployment doesnot need extra permission like
privileged,Capabilities and reduce unwanted volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
9bff7b0ac9 deploy: remove extra volumes from cephfs plugin PSP
removed extra volume permissions from the cephfs
nodeplugin PSP.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
051af3b257 deploy: reduce the PSP permission for cephfs deployment
cephfs deployment doesnot need extra permission like
privileged,Capabilities and remove unwanted volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
7fc1bf1321 deploy: remove extra volumes from rbd plugin PSP
removed extra volume permissions from the rbd
nodeplugin PSP.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
9e88fd1eb7 deploy: reduce the PSP permission for rbd deployment
rbd deployment doesnot need extra permission like
privileged,Capabilities and remove unwanted volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
e5569f0547 deploy: remove securityContext from rbd provisioner
we dont need securityContext for the rbd provisioner
pod as its not doing any special operations like map
,unmap selinux etc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
62a9ab6d9f helm: remove securityContext from cephfs provisioner
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
f1c64a2a6b deploy: remove securityContext from cephfs provisioner
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations like mounts,
selinux etc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Rakshith R
8f432e8bfa rebase: update k8s.io packages from v0.22.1 to v0.22.2
This commit also fixes k8s.io/cloud-providers v0.22.2,
instead of v1.22.1 which does not exist and was overrided
in replace.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-21 08:09:12 +00:00
dependabot[bot]
c1931c8192 rebase: bump k8s.io/kubernetes from 1.22.1 to 1.22.2
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.22.1 to 1.22.2.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.22.1...v1.22.2)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-21 08:09:12 +00:00
dependabot[bot]
566cef2719 rebase: bump github.com/csi-addons/spec from 0.1.0 to 0.1.1
Bumps [github.com/csi-addons/spec](https://github.com/csi-addons/spec) from 0.1.0 to 0.1.1.
- [Release notes](https://github.com/csi-addons/spec/releases)
- [Commits](https://github.com/csi-addons/spec/compare/v0.1.0...v0.1.1)

---
updated-dependencies:
- dependency-name: github.com/csi-addons/spec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-21 05:06:08 +00:00
Humble Chirammal
4804f47b18 e2e: Add e2e for rbd migration static pvc
This commit adds e2e for rbd migration static PVCs

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-20 09:54:54 +00:00
Humble Chirammal
2e8e8f5e64 rbd: fill clusterID if its a migration nodestage request
the migration nodestage request does not carry the 'clusterID' in it
and only monitors are available with the volumeContext. The volume
context flag 'migration=true' and 'static=true' flags allow us to
fill 'clusterID' from the passed in monitors to the volume Context,so
that rest of the static operations on nodestage can be proceeded as we
do treat static volumes today.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-20 09:54:54 +00:00
Humble Chirammal
1f5963919f util: get clusterID for the passed in mon string
as part of migration support, the clusterID has to be fetched
from passed in mon. Because the intree RBD storage class only
got monitor and not `clusterID` parameter support. However, in
CSI, SC has the `clusterID` parameter support but not mon. Due
to that we have to fetch the clusterID from config file for the
passed in mon and use it in our operations. This adds a helper
function to retrieve clusterID from passed in mon string.

Updates https://github.com/ceph/ceph-csi/issues/2509

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-20 09:54:54 +00:00
Prasanna Kumar Kalever
22bb31df19 doc: update the rbd-nbd doc with log strategies options
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Prasanna Kumar Kalever
c9cc36d8db rbd: provide alternatives to preserve the ceph log files
Currently, we delete the ceph client log file on unmap/detach.

This patch provides additional alternatives for users who would like to
persist the log files.

Strategies:
-----------
`remove`: delete log file on unmap/detach
`compress`: compress the log file to gzip on unmap/detach
`preserve`: preserve the log file in text format

Note that the default strategy will be remove on unmap, and these options
can be tweaked from the storage class

Compression size details example:

On Map: (with debug-rbd=20)
---------
$ ls -lh
-rw-r--r-- 1 root root 526K Sep  1 18:15
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-d2e89c87-0b4d-11ec-8ea6-160f128e682d.log

On unmap:
---------
$ ls -lh
-rw-r--r-- 1 root root  33K Sep  1 18:15
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-d2e89c87-0b4d-11ec-8ea6-160f128e682d.gz

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Prasanna Kumar Kalever
10bbb049f7 cleanup: passing pointers to larger type
Log:
internal/rbd/rbd_attach.go:424:2: hugeParam: dArgs is heavy (88 bytes);
consider passing it by pointer (gocritic)

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Prasanna Kumar Kalever
ad2c6d2851 util: add gzip helper function
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Prasanna Kumar Kalever
314516cedd deploy: fix cephLogDir passing to storageclass via helm
cephLogDir: is a storage class option that is passed to rbd-nbd daemon.
cephLogDirHostPath: is a nodeplugin daemonset level option that helps in
                   using the right host-path while bind-mounting

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Shyamsundar Ranganathan
47dc9cf28d rbd: Report errors when a resync maybe in progress
Currently we return a !ready status if an image
is not found when a replication resync is issued.

We also return a !ready just post issuing a resync.

The change is to ensure we return errors in these
cases for the caller to retry the operation till
we can determine we are actually resyncing, and then
return !ready with nil errors.

Part of addressing:
  https://github.com/csi-addons/volume-replication-operator/issues/101

Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2021-09-15 15:59:22 +00:00
Rakshith R
6bb4ec6715 ci: set NUM_DISKS default value to 1 in minikube.sh
Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-15 07:20:50 +00:00
Madhu Rajanna
3aab8dad00 ci: start minikube with psp addon
we need to start minikube with the
PodSecurityPolicy admission controller
and the pod-security-policy addon enabled
for psp.

Ref:- https://minikube.sigs.k8s.io/docs/
tutorials/using_psp/#tutorial

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-14 11:27:29 +00:00
Madhu Rajanna
1e037a56c1 ci: remove minikube workaround for psp
as we are no longer using older (<1.11.1)
version of minikube, removing the
workaround to support the psp with
older minikube versions.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-14 11:27:29 +00:00
Rakshith R
82d09d81cf util: modify GetMonsAndClusterID() to take clusterID instead of options
This commit:
- modifies GetMonsAndClusterID() to take clusterID instead of options.
- moves out validation of clusterID is set or not out of GetMonsAndClusterID().
- defines ErrClusterIDNotSet new error for reusability.
- add GetClusterID() to obtain clusterID from options.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-14 08:39:57 +00:00
Rakshith R
9d1e98ca60 rbd: check for clusterid mapping in genVolFromVolumeOptions()
This commit adds capability to genVolFromVolumeOptions() to fetch
mapped clusted-id & mon ips for mirrored PVC on secondary cluster
which may have different cluster-id.

This is required for NodeStageVolume().

We also don't need to check for mapping during volume create requests,
so it can be disabled by passing a bool checkClusterIDMapping as false.

GetMonsAndClusterID() is modified to accept bool checkClusterIDMapping
based on which clustermapping is checked to fetch mapped cluster-id and
mon-ips.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-14 08:39:57 +00:00
Humble Chirammal
4be53a27d3 cleanup: replace parentName to snapParentName in checkReservation
at present, eventhough the checkReservation works for both volume
and snapshot, the arg parentName make sense only for snapshot cases
renaming that arg to more approprite

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-14 05:32:54 +00:00
Humble Chirammal
1fee3ec460 cleanup: correct checkReservation return description
it wrongly mention that the return is imageUUID string where actually
it is the imageData struct

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-14 05:32:54 +00:00
Rakshith R
8f75a24cfd ci: use 0 as default NUM_DISKS in minikube.sh
This is done to prevent conflicts with current ci setup externally
attaching disks.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-09 05:18:36 +00:00
Rakshith R
42a6c3c006 ci: pass $DISK_CONFIG properly to minikube start
Having double quotes around $DISK_CONFIG led to these args
not being properly passed to minikube start. This commit fixes it.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-09 05:18:36 +00:00