Commit Graph

3376 Commits

Author SHA1 Message Date
Humble Chirammal
87beaac25b rbd: add ReadWriteOncePod in accessModeStrToInt() conversion function
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-12 09:55:50 +00:00
Humble Chirammal
aa698bc3e1 rebase: update kubernetes and libraries to v1.22.0 version
Kubernetes v1.22 version has been released and this update
ceph csi dependencies to use the same version.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-12 09:55:50 +00:00
Yati Padia
e077c1fdf5 cleanup: run codespell on containerized testing
This commit adds a new target codespell to the
make containerized-test.

Fixes: #2229

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-08-12 09:42:54 +05:30
Rakshith R
7fba62dd47 ci: internally create & delete cephcsi namespace in install-helm.sh
This ensures the kubectl call is retried with kubectl_retry function.

Updates: #2309

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Rakshith R
eb8c1cd5ab ci: use kubectl_retry in install_helm.sh script
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Rakshith R
2b19197e2f ci: modify kubectl_retry() to handle NotFound on delete cmd
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Rakshith R
a15892a87a ci: move kubectl_retry() to utils.sh to be able to import it
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 08:42:21 +00:00
Prasanna Kumar Kalever
2723353f8d e2e: add testcase for encrypted volume with rbd-nbd mounter
Fixes: #2235

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-11 04:10:34 +00:00
Prasanna Kumar Kalever
396ab1b4d7 doc: update rbd-nbd documentation with encryption volume support details
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-11 04:10:34 +00:00
Prasanna Kumar Kalever
9669394b23 e2e: add testcase for volume expansion with rbd-nbd mounter
The rbd-nbd resize volume support with its netlink interface needs linux
kernel version >= v5.3.0

Hence define a defence check for the supported kernel version

Fixes: #2234

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-10 13:21:16 +00:00
Prasanna Kumar Kalever
85a1fba0f4 doc: update rbd-nbd documentation with volume expansion support details
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-10 13:21:16 +00:00
Rakshith R
1d49b6a288 e2e: create reusable variable vaultUserSecretPath = "user-secret.yaml"
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
2f995eada2 e2e: add modification to test encrypted PVC with rbd controller
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
8ca7a35820 e2e: use retryKubectlFile() for creating & deleting secrets
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
0744ad502b e2e: add prefixname to rbd controller test
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
f05ac2b25d rbd: extract kmsID from volumeAttributes in RegenerateJournal()
This commit adds functionality of extracting encryption kmsID,
owner from volumeAttributes in RegenerateJournal() and adds utility
functions ParseEncryptionOpts and FetchEncryptionKMSID.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
b960e3633a rbd: extract volumeNamePrefix in RegenerateJournal()
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
b9b4b1e34e rbd: refractor RegenerateJournal() to take in volumeAttributes
This commit refractors RegenerateJournal() to take in
volumeAttributes map[string]string as argument so it
can extract required attributes internally.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Rakshith R
39d6752fc1 rbd: use CSIInstanceID var instead of "default" in RegenerateJournal()
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-10 09:17:59 +00:00
Ben Ye
9cd8326bb2 cleanup: allocate slice with known size
As the input capabilities size is known, it is better to allocate slice with a specified size.

Signed-off-by: Ben Ye <ben.ye@bytedance.com>
2021-08-10 05:39:44 +00:00
Madhu Rajanna
6cc37f0a17 cleanup: use different file name for testing
For clusterMappingConfigFile using different
file name so that multiple unit test cases can
work without any data race.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 13:37:25 +00:00
Madhu Rajanna
3c85219962 rbd: consider empty mirroring mode
consider the empty mirroring mode when
validating the snapshot interval and
the scheduling time.
Even if the mirroring Mode is not set
validate the snapshot scheduling details
as cephcsi sets the mirroring mode to default
snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 11:05:05 +00:00
Rakshith R
825211730c rbd: fix snapshot id idempotency issue
This commit fixes snapshot id idempotency issue by
always returning an error when flattening is in progress
and not using `readyToUse:false` response.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
7f6b73e71f e2e: log imageList in validateRBDImageCount for better debugging
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
9d57717222 e2e: add test cases for pvc-pvcClone chain with depth 2
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
9321b4bce4 e2e: add test cases for snapshot-restore chain with depth 2
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Rakshith R
859d696279 cleanup: refractor checkCloneImage to reducing nesting if
This commit refractors checkCloneImage function to
address nestif linter issue.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Madhu Rajanna
a5a8952716 rbd: fix clone problem
This commit fixes a bug in checkCloneImage() which was caused
by checking cloned image before checking on temp-clone image snap
in a subsequent request which lead to stale images. This was solved
by checking temp-clone image snap and flattening temp-clone if
needed.
This commit also fixes comparison bug in flattenCloneImage().

Signed-off-by: Rakshith R <rar@redhat.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 07:28:43 +00:00
Madhu Rajanna
916c97b4a8 rbd: copy creds when copying the connection
rbd flatten functions is a CLI call and it expects
the creds as the input and copying of creds is
required when we generate the temp clone image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-09 07:28:43 +00:00
Rakshith R
08728b631b rbd: fix vol.VolID in cloneFromSnapshot()
Volume generated from snap using genrateVolFromSnap
already copies volume ID correctly, therefore removing
`vol.VolID = rbdVol.VolID` which wrongly copies parent
Volume ID instead leading to error from copyEncryption()
on parent and clone volume ID being equal.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 07:28:43 +00:00
Niels de Vos
bb60173a98 e2e: add verifyKeyDestroyed() for validating vaultDestroyKeys
The kmsConfig type in the e2e suite has been enhanced with two functions
that make it possible to validate the destruction of deleted keys.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Niels de Vos
b5d2321d57 cleanup: use vaultDefaultCAVerify to set default value
Golang-ci complains about the following:

    internal/util/vault_tokens.go:99:20: string `true` has 4 occurrences, but such constant `vaultDefaultDestroyKeys` already exists (goconst)
    	v.VaultCAVerify = "true"
    	                  ^

This occurence of "true" can be replaced by vaultDefaultCAVerify so
address the warning.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Niels de Vos
f584db41e6 util: add vaultDestroyKeys option to destroy Vault kv-v2 secrets
Hashicorp Vault does not completely remove the secrets in a kv-v2
backend when the keys are deleted. The metadata of the keys will be
kept, and it is possible to recover the contents of the keys afterwards.

With the new `vaultDestroyKeys` configuration parameter, this behaviour
can now be selected. By default the parameter will be set to `true`,
indicating that the keys and contents should completely be destroyed.
Setting it to any other value will make it possible to recover the
deleted keys.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Niels de Vos
d7bcb42481 rebase: update libopenstorage/secrets
libopenstorage has added a new feature that makes it possible to destroy
the contents of a key/value in the Hashicorp Vault kv-v2 secrets backend.

See-also: https://github.com/libopenstorage/secrets/pull/55
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-06 12:19:18 +00:00
Madhu Rajanna
2782878ea2 rbd: log LastUpdate in UTC format
This Commit converts the LastUpdate
from int to the UTC format and logs
it for better debugging.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 10:18:51 +00:00
Madhu Rajanna
2c66dfc3e4 e2e: retry running kubectl on known errors
By using retryKubectl helper function,
a retry will be done, and the known error
messages will be skipped.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
2071c535fa e2e: pass variadic argument to kubectl helper function
this provides caller ability to pass the arguments
like ignore-not-found=true etc when executing
the kubectl commands.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
9f0af30735 e2e: add retryKubectlArgs helper for kubectl retry
added helper function retryKubectlArgs to perform
action if its a known error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
dd9fabf747 e2e: add isAlreadyExistsCLIError to check known error
added isAlreadyExistsCLIError to check for known error.
if error is already exists we are considering it
as a success.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Madhu Rajanna
d321663872 deploy: add template changes for mapping
added template changes for the clusterID and
poolID,fsID mapping details for the pod templates.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 16:07:51 +00:00
Madhu Rajanna
92ad2ceec9 rbd: read clusterID and PoolID from mapping
Whenever Ceph-CSI receives a CSI/Replication
request it will first decode the
volumeHandle and try to get the required
OMAP details if it is not able to
retrieve, receives a `Not Found` error
message and Ceph-CSI will check for the
clusterID mapping. If the old volumeID
`0001-00013-site1-storage-0000000000000001
-b0285c97-a0ce-11eb-8c66-0242ac110002`
contains the `site1-storage` as the clusterID,
now Ceph-CSI will look for the corresponding
clusterID `site2-storage` from the above configmap.
If the clusterID mapping is found now Ceph-CSI
will look for the poolID mapping ie mapping between
`1` and `2`. Example:- pool with name exists on
both the clusters with different ID's Replicapool
with ID `1` on site1 and Replicapool with ID `2`
on site2. After getting the required mapping Ceph-CSI
has the required information to get more details
from the rados OMAP. If we have multiple clusterID mapping
it will loop through all the mapping and checks the
corresponding pool to get the OMAP data. If the clusterID
mapping does not exist Ceph-CSI will return an `Not Found`
error message to the caller.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 16:07:51 +00:00
Madhu Rajanna
ac11d71e19 util: add helper function to read clusterID mapping
added helper function to read the clusterID mapping
from the mounted file.

The clusterID mapping contains below mappings
* ClusterID mappings (to cluster to which we are failingover
and from which cluster failover happened)
* RBD PoolID mapping of between the clusters.
* CephFS FscID mapping between the clusters.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 16:07:51 +00:00
Madhu Rajanna
fce5a181d0 doc: change FsID to FscID for cephfs
updated the filesystem identifier from
FsId to FscID.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 16:07:51 +00:00
Yug Gupta
080f7538c0 helm: update cephfs provisioner updateStrategy
Update ceph-csi-cephfs.provisioner updatestrategy
to allow maxUnavailable pods at a time to be 50%

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2021-08-05 14:04:16 +00:00
Yug Gupta
ea088d40be helm: update rbd provisioner updateStrategy
Update ceph-csi-rbd.provisioner updatestrategy
to allow maxUnavailable pods at a time to be 50%

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2021-08-05 14:04:16 +00:00
Madhu Rajanna
3805c29f36 build: update commitlint to use latest tag
updaing the commitlint to the latest, so
each time latest release can be installed by
default.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 14:50:12 +05:30
Madhu Rajanna
0b6322afda ci: update mergify for commitlint
updated commitlint mergify rules to
consider the commitlint status to
merge the PR.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-05 14:50:12 +05:30
Madhu Rajanna
38ef32a496 ci: trailer-exists to verify sign-off
This commit uses trailer-exists instead
of signed-off-by to verify the sign-off-by
message.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Suggested-by: Ade Attwood
2021-08-05 14:50:12 +05:30
Yug Gupta
1dc032e554 doc: update comments in voljournal
Update spell errors and comments in
voljournal.go

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2021-08-05 08:11:15 +00:00
Niels de Vos
4859f2dfdb util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS
The VAULT_AUTH_MOUNT_PATH is a Vault configuration parameter that allows
a user to set a non default path for the Kubernetes ServiceAccount
integration. This can already be configured for the Vault KMS, and is
now added to the Vault Tenant SA KMS as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-05 06:02:57 +00:00