Commit Graph

2736 Commits

Author SHA1 Message Date
Humble Chirammal
7c2cbf473c doc: update readme for 3.4.0 release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-17 06:57:09 +00:00
Madhu Rajanna
5562e46d0f rbd: Cleanup OMAP data for secondary image
If the image is in a secondary state and its
up+replaying means its an healthy secondary
and the image is primary somewhere in the remote cluster
and the local image is getting replayed. Delete the
OMAP data generated as we cannot delete the
secondary image. When the image on the primary
cluster gets deleted/mirroring disabled, the image on
all the remote (secondary) clusters will get
auto-deleted. This helps in garbage collecting
the OMAP, PVC and PV objects after failback operation.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-16 17:38:25 +00:00
Madhu Rajanna
fc0d6f6b8b rbd: return succuss if image is healthy secondary
If the image is in secondary state and its
up+replaying means its an healthy secondary
and the image is primary somewhere in the remote
cluster and the local image is getting replayed.
Return success for the Disabling mirroring as
we cannot disable the mirroring on the secondary
state, when the image on the remote site gets
disabled the image on all the remote (secondary)
will get auto deleted. This helps in garbage
collecting the volume replication kuberentes
artifacts

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-16 17:38:25 +00:00
Madhu Rajanna
35324b2e17 rbd: add helper function to get local state
added helper function to check the local image
state is up+replaying.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-16 17:38:25 +00:00
Humble Chirammal
3462cd9bbd helm: correct the groupVersion of CSIDriver in the chart
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-16 15:21:27 +00:00
Humble Chirammal
8e00c2c810 helm: correct watch verb in topology RBAC
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-16 15:21:27 +00:00
Rakshith R
2bd6b669fa ci: add csi sidecar version info to build.env
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-16 11:17:09 +00:00
Rakshith R
dc8479f5ad ci: make csi sidecar image version configurable in minikube.sh
This commit makes csi sidecar image version configurable in
minikube.sh.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-16 11:17:09 +00:00
Yug Gupta
bc18732cb7 ci: require job for k8s v1.22 in place of v1.19
As kubernetes v1.19 is heading towards its EOL
on 2021-09-30, run tests on kubernetes v1.22
and require it to pass for merging.

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2021-08-12 18:12:27 +05:30
Humble Chirammal
56ac143450 rebase: update go-ceph version to v0.11.0
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-12 12:42:20 +00:00
Humble Chirammal
87beaac25b rbd: add ReadWriteOncePod in accessModeStrToInt() conversion function
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-12 09:55:50 +00:00
Humble Chirammal
aa698bc3e1 rebase: update kubernetes and libraries to v1.22.0 version
Kubernetes v1.22 version has been released and this update
ceph csi dependencies to use the same version.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-12 09:55:50 +00:00
Yati Padia
e077c1fdf5 cleanup: run codespell on containerized testing
This commit adds a new target codespell to the
make containerized-test.

Fixes: #2229

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-08-12 09:42:54 +05:30
Rakshith R
7fba62dd47 ci: internally create & delete cephcsi namespace in install-helm.sh
This ensures the kubectl call is retried with kubectl_retry function.

Updates: #2309

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Rakshith R
eb8c1cd5ab ci: use kubectl_retry in install_helm.sh script
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Rakshith R
2b19197e2f ci: modify kubectl_retry() to handle NotFound on delete cmd
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Rakshith R
a15892a87a ci: move kubectl_retry() to utils.sh to be able to import it
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Prasanna Kumar Kalever
2723353f8d e2e: add testcase for encrypted volume with rbd-nbd mounter
Fixes: #2235

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-11 04:10:34 +00:00
Prasanna Kumar Kalever
396ab1b4d7 doc: update rbd-nbd documentation with encryption volume support details
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-11 04:10:34 +00:00
Prasanna Kumar Kalever
9669394b23 e2e: add testcase for volume expansion with rbd-nbd mounter
The rbd-nbd resize volume support with its netlink interface needs linux
kernel version >= v5.3.0

Hence define a defence check for the supported kernel version

Fixes: #2234

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-10 13:21:16 +00:00
Prasanna Kumar Kalever
85a1fba0f4 doc: update rbd-nbd documentation with volume expansion support details
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-10 13:21:16 +00:00
Rakshith R
1d49b6a288 e2e: create reusable variable vaultUserSecretPath = "user-secret.yaml"
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
2f995eada2 e2e: add modification to test encrypted PVC with rbd controller
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
8ca7a35820 e2e: use retryKubectlFile() for creating & deleting secrets
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
0744ad502b e2e: add prefixname to rbd controller test
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
f05ac2b25d rbd: extract kmsID from volumeAttributes in RegenerateJournal()
This commit adds functionality of extracting encryption kmsID,
owner from volumeAttributes in RegenerateJournal() and adds utility
functions ParseEncryptionOpts and FetchEncryptionKMSID.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
b960e3633a rbd: extract volumeNamePrefix in RegenerateJournal()
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
b9b4b1e34e rbd: refractor RegenerateJournal() to take in volumeAttributes
This commit refractors RegenerateJournal() to take in
volumeAttributes map[string]string as argument so it
can extract required attributes internally.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
39d6752fc1 rbd: use CSIInstanceID var instead of "default" in RegenerateJournal()
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Ben Ye
9cd8326bb2 cleanup: allocate slice with known size
As the input capabilities size is known, it is better to allocate slice with a specified size.

Signed-off-by: Ben Ye <ben.ye@bytedance.com>
2021-08-10 05:39:44 +00:00
Madhu Rajanna
6cc37f0a17 cleanup: use different file name for testing
For clusterMappingConfigFile using different
file name so that multiple unit test cases can
work without any data race.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 13:37:25 +00:00
Madhu Rajanna
3c85219962 rbd: consider empty mirroring mode
consider the empty mirroring mode when
validating the snapshot interval and
the scheduling time.
Even if the mirroring Mode is not set
validate the snapshot scheduling details
as cephcsi sets the mirroring mode to default
snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 11:05:05 +00:00
Rakshith R
825211730c rbd: fix snapshot id idempotency issue
This commit fixes snapshot id idempotency issue by
always returning an error when flattening is in progress
and not using `readyToUse:false` response.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
7f6b73e71f e2e: log imageList in validateRBDImageCount for better debugging
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
9d57717222 e2e: add test cases for pvc-pvcClone chain with depth 2
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
9321b4bce4 e2e: add test cases for snapshot-restore chain with depth 2
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
859d696279 cleanup: refractor checkCloneImage to reducing nesting if
This commit refractors checkCloneImage function to
address nestif linter issue.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Madhu Rajanna
a5a8952716 rbd: fix clone problem
This commit fixes a bug in checkCloneImage() which was caused
by checking cloned image before checking on temp-clone image snap
in a subsequent request which lead to stale images. This was solved
by checking temp-clone image snap and flattening temp-clone if
needed.
This commit also fixes comparison bug in flattenCloneImage().

Signed-off-by: Rakshith R <rar@redhat.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 07:28:43 +00:00
Madhu Rajanna
916c97b4a8 rbd: copy creds when copying the connection
rbd flatten functions is a CLI call and it expects
the creds as the input and copying of creds is
required when we generate the temp clone image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 07:28:43 +00:00
Rakshith R
08728b631b rbd: fix vol.VolID in cloneFromSnapshot()
Volume generated from snap using genrateVolFromSnap
already copies volume ID correctly, therefore removing
`vol.VolID = rbdVol.VolID` which wrongly copies parent
Volume ID instead leading to error from copyEncryption()
on parent and clone volume ID being equal.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Niels de Vos
bb60173a98 e2e: add verifyKeyDestroyed() for validating vaultDestroyKeys
The kmsConfig type in the e2e suite has been enhanced with two functions
that make it possible to validate the destruction of deleted keys.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Niels de Vos
b5d2321d57 cleanup: use vaultDefaultCAVerify to set default value
Golang-ci complains about the following:

    internal/util/vault_tokens.go:99:20: string `true` has 4 occurrences, but such constant `vaultDefaultDestroyKeys` already exists (goconst)
    	v.VaultCAVerify = "true"
    	                  ^

This occurence of "true" can be replaced by vaultDefaultCAVerify so
address the warning.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Niels de Vos
f584db41e6 util: add vaultDestroyKeys option to destroy Vault kv-v2 secrets
Hashicorp Vault does not completely remove the secrets in a kv-v2
backend when the keys are deleted. The metadata of the keys will be
kept, and it is possible to recover the contents of the keys afterwards.

With the new `vaultDestroyKeys` configuration parameter, this behaviour
can now be selected. By default the parameter will be set to `true`,
indicating that the keys and contents should completely be destroyed.
Setting it to any other value will make it possible to recover the
deleted keys.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Niels de Vos
d7bcb42481 rebase: update libopenstorage/secrets
libopenstorage has added a new feature that makes it possible to destroy
the contents of a key/value in the Hashicorp Vault kv-v2 secrets backend.

See-also: https://github.com/libopenstorage/secrets/pull/55
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Madhu Rajanna
2782878ea2 rbd: log LastUpdate in UTC format
This Commit converts the LastUpdate
from int to the UTC format and logs
it for better debugging.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 10:18:51 +00:00
Madhu Rajanna
2c66dfc3e4 e2e: retry running kubectl on known errors
By using retryKubectl helper function,
a retry will be done, and the known error
messages will be skipped.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
2071c535fa e2e: pass variadic argument to kubectl helper function
this provides caller ability to pass the arguments
like ignore-not-found=true etc when executing
the kubectl commands.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
9f0af30735 e2e: add retryKubectlArgs helper for kubectl retry
added helper function retryKubectlArgs to perform
action if its a known error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
dd9fabf747 e2e: add isAlreadyExistsCLIError to check known error
added isAlreadyExistsCLIError to check for known error.
if error is already exists we are considering it
as a success.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
d321663872 deploy: add template changes for mapping
added template changes for the clusterID and
poolID,fsID mapping details for the pod templates.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 16:07:51 +00:00