Commit Graph

2349 Commits

Author SHA1 Message Date
Madhu Rajanna
fbc1e5f3d5 e2e: retry running kubectl on known errors
By using retryKubectl helper function,
a retry will be done, and the known error
messages will be skipped.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2c66dfc3e4)
2021-08-11 07:03:05 +00:00
Madhu Rajanna
f7e150b84f e2e: pass variadic argument to kubectl helper function
this provides caller ability to pass the arguments
like ignore-not-found=true etc when executing
the kubectl commands.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2071c535fa)
2021-08-11 07:03:05 +00:00
Madhu Rajanna
64937f1f68 e2e: add retryKubectlArgs helper for kubectl retry
added helper function retryKubectlArgs to perform
action if its a known error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 9f0af30735)
2021-08-11 07:03:05 +00:00
Madhu Rajanna
9e84583063 e2e: add isAlreadyExistsCLIError to check known error
added isAlreadyExistsCLIError to check for known error.
if error is already exists we are considering it
as a success.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit dd9fabf747)
2021-08-11 07:03:05 +00:00
Madhu Rajanna
72a2b97be2 rbd: consider empty mirroring mode
consider the empty mirroring mode when
validating the snapshot interval and
the scheduling time.
Even if the mirroring Mode is not set
validate the snapshot scheduling details
as cephcsi sets the mirroring mode to default
snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 3c85219962)
2021-08-10 12:55:41 +00:00
Madhu Rajanna
75ff33785b rbd: log LastUpdate in UTC format
This Commit converts the LastUpdate
from int to the UTC format and logs
it for better debugging.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2782878ea2)
2021-08-10 08:56:08 +00:00
Rakshith R
0b43e91c77 rbd: fix snapshot id idempotency issue
This commit fixes snapshot id idempotency issue by
always returning an error when flattening is in progress
and not using `readyToUse:false` response.

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 825211730c)
2021-08-09 12:10:42 +00:00
Rakshith R
05622b87c0 e2e: log imageList in validateRBDImageCount for better debugging
Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 7f6b73e71f)
2021-08-09 12:10:42 +00:00
Rakshith R
f5e73009e1 e2e: add test cases for pvc-pvcClone chain with depth 2
Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 9d57717222)
2021-08-09 12:10:42 +00:00
Rakshith R
dc046d0204 e2e: add test cases for snapshot-restore chain with depth 2
Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 9321b4bce4)
2021-08-09 12:10:42 +00:00
Rakshith R
33234c1b51 cleanup: refractor checkCloneImage to reducing nesting if
This commit refractors checkCloneImage function to
address nestif linter issue.

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 859d696279)
2021-08-09 12:10:42 +00:00
Madhu Rajanna
32faed322a rbd: fix clone problem
This commit fixes a bug in checkCloneImage() which was caused
by checking cloned image before checking on temp-clone image snap
in a subsequent request which lead to stale images. This was solved
by checking temp-clone image snap and flattening temp-clone if
needed.
This commit also fixes comparison bug in flattenCloneImage().

Signed-off-by: Rakshith R <rar@redhat.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit a5a8952716)
2021-08-09 12:10:42 +00:00
Madhu Rajanna
a7a5a527c2 rbd: copy creds when copying the connection
rbd flatten functions is a CLI call and it expects
the creds as the input and copying of creds is
required when we generate the temp clone image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 916c97b4a8)
2021-08-09 12:10:42 +00:00
Rakshith R
33509ca90a rbd: fix vol.VolID in cloneFromSnapshot()
Volume generated from snap using genrateVolFromSnap
already copies volume ID correctly, therefore removing
`vol.VolID = rbdVol.VolID` which wrongly copies parent
Volume ID instead leading to error from copyEncryption()
on parent and clone volume ID being equal.

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 08728b631b)
2021-08-09 12:10:42 +00:00
Madhu Rajanna
1470af8316 doc: change FsID to FscID for cephfs
updated the filesystem identifier from
FsId to FscID.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit fce5a181d0)
2021-08-09 09:24:16 +00:00
Madhu Rajanna
f65961d01e doc: add design doc for clusterid poolid mapping
added design doc to handle volumeID mapping in case
of the failover in the Disaster Recovery.

update #2118

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 5fc9c3a046)
2021-08-09 09:24:16 +00:00
Madhu Rajanna
cbe3ac71f3 deploy: add template changes for mapping
added template changes for the clusterID and
poolID,fsID mapping details for the pod templates.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit d321663872)
2021-08-09 09:24:16 +00:00
Madhu Rajanna
829fc5ed95 rbd: read clusterID and PoolID from mapping
Whenever Ceph-CSI receives a CSI/Replication
request it will first decode the
volumeHandle and try to get the required
OMAP details if it is not able to
retrieve, receives a `Not Found` error
message and Ceph-CSI will check for the
clusterID mapping. If the old volumeID
`0001-00013-site1-storage-0000000000000001
-b0285c97-a0ce-11eb-8c66-0242ac110002`
contains the `site1-storage` as the clusterID,
now Ceph-CSI will look for the corresponding
clusterID `site2-storage` from the above configmap.
If the clusterID mapping is found now Ceph-CSI
will look for the poolID mapping ie mapping between
`1` and `2`. Example:- pool with name exists on
both the clusters with different ID's Replicapool
with ID `1` on site1 and Replicapool with ID `2`
on site2. After getting the required mapping Ceph-CSI
has the required information to get more details
from the rados OMAP. If we have multiple clusterID mapping
it will loop through all the mapping and checks the
corresponding pool to get the OMAP data. If the clusterID
mapping does not exist Ceph-CSI will return an `Not Found`
error message to the caller.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 92ad2ceec9)
2021-08-09 09:24:16 +00:00
Madhu Rajanna
daea5177e5 util: add helper function to read clusterID mapping
added helper function to read the clusterID mapping
from the mounted file.

The clusterID mapping contains below mappings
* ClusterID mappings (to cluster to which we are failingover
and from which cluster failover happened)
* RBD PoolID mapping of between the clusters.
* CephFS FscID mapping between the clusters.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit ac11d71e19)
2021-08-09 09:24:16 +00:00
Yug Gupta
459f6eca5a helm: update cephfs provisioner updateStrategy
Update ceph-csi-cephfs.provisioner updatestrategy
to allow maxUnavailable pods at a time to be 50%

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
(cherry picked from commit 080f7538c0)
2021-08-06 12:37:33 +00:00
Yug Gupta
45e80f8952 helm: update rbd provisioner updateStrategy
Update ceph-csi-rbd.provisioner updatestrategy
to allow maxUnavailable pods at a time to be 50%

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
(cherry picked from commit ea088d40be)
2021-08-06 12:37:33 +00:00
Niels de Vos
bc24a8c8ac util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS
The VAULT_AUTH_MOUNT_PATH is a Vault configuration parameter that allows
a user to set a non default path for the Kubernetes ServiceAccount
integration. This can already be configured for the Vault KMS, and is
now added to the Vault Tenant SA KMS as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 4859f2dfdb)
2021-08-06 09:30:32 +00:00
Madhu Rajanna
05c9b3b245 build: update commitlint to use latest tag
updaing the commitlint to the latest, so
each time latest release can be installed by
default.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 3805c29f36)
2021-08-05 14:51:03 +00:00
Madhu Rajanna
1a83027a4d ci: update mergify for commitlint
updated commitlint mergify rules to
consider the commitlint status to
merge the PR.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 0b6322afda)
2021-08-05 14:51:03 +00:00
Madhu Rajanna
e7ea1fd2d9 ci: trailer-exists to verify sign-off
This commit uses trailer-exists instead
of signed-off-by to verify the sign-off-by
message.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Suggested-by: Ade Attwood
(cherry picked from commit 38ef32a496)
2021-08-05 14:51:03 +00:00
Alexandre Lossent
7688bc3a7a cephfs: support selinux mount options
- mount host's /etc/selinux in node plugins
- process mount options in all code paths for cephfs volume options

Signed-off-by: Alexandre Lossent <alexandre.lossent@cern.ch>
(cherry picked from commit 5cba04c470)
2021-08-05 08:37:52 +00:00
Niels de Vos
b866bd491c util: add vaultAuthNamespace option for Vault KMS
The new `vaultAuthNamespace` configuration parameter can be set to the
Vault Namespace where the authentication is setup in the service. Some
Hashicorp Vault deployments use sub-namespaces for their users/tenants,
with a 'root' namespace where the authentication is configured. This
requires passing of different Vault namespaces for different operations.

Example:
 - the Kubernetes Auth mechanism is configured for in the Vault
   Namespace called 'devops'
 - a user/tenant has a sub-namespace called 'devops/website' where the
   encryption passphrases can be placed in the key-value store

The configuration for this, then looks like:

    vaultAuthNamespace: devops
    vaultNamespace: devops/homepage

Note that Vault Namespaces are a feature of the Hashicorp Vault
Enterprise product, and not part of the Open Source version. This
prevents adding e2e tests that validate the Vault Namespace
configuration.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit f2d5c2e0df)
2021-08-05 06:44:23 +00:00
Niels de Vos
a962cccd0a util: correct error message when connecting to Vault fails
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 83167e2ac5)
2021-08-05 06:44:23 +00:00
rtsp
cb40ae5bca deploy: rbd kubernetes manifests
add ability to deploy ceph-csi-rbd on non-default namespace

Signed-off-by: rtsp <git@rtsp.us>
(cherry picked from commit af1f50ba04)
2021-08-04 16:24:58 +00:00
Niels de Vos
3bbcda6174 e2e: use official CentOS container location
registry.centos.org is not officially maintained by the CentOS
infrastructure team. The container images on quay.io are the official
once and we should use those instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit e0ac70f8fb)
2021-08-04 10:50:30 +00:00
Artur Troian
82fd1e5248 util: getCgroupPidsFile produces striped path when extra : present
This commit uses `string.SplitN` instead of `string.Split`.
The path for pids.max has extra `:` symbols in it due to which
getCgroupPidsFile() splits the string into 5 tokens instead of
3 leading to loss of part of the path.
As a result, the below error is reported:
`Failed to get the PID limit, can not reconfigure: open
/sys/fs/cgroup/pids/system.slice/containerd.service/
kubepods-besteffort-pod183b9d14_aed1_4b66_a696_da0c738bc012.slice/pids.max:
no such file or directory`
SplitN takes an argument n and splits the string
accordingly which helps us to get the desired
file path.

Fixes: #2337

Co-authored-by: Yati Padia <ypadia@redhat.com>
Signed-off-by: Yati Padia <ypadia@redhat.com>
(cherry picked from commit 16ec97d8f7)
2021-08-04 07:11:26 +00:00
Humble Chirammal
966841cafc deploy: revert changes made for 3.4.0 release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-07-30 06:45:38 +00:00
Humble Chirammal
94ef181bc8 build: update build.env for 3.4.0 release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-07-29 10:03:20 +00:00
Humble Chirammal
1f515404e7 deploy: change minikube image for 3.4.0 release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-07-29 10:03:20 +00:00
Humble Chirammal
61aab6ddb5 helm: replace image tag to v3.4.0 from canary for the release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-07-29 10:03:20 +00:00
Humble Chirammal
03ab0738a4 deploy: changes the image to v3.4.0 instead of canary for release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-07-29 10:03:20 +00:00
Prasanna Kumar Kalever
0a02343f2d doc: update the upgrade documentation to reflect 3.4.0 changes
Mainly removed rbd-nbd mounter specified at the pre-upgrade
considerations affecting the restarts.

Also updated the 3.3 tags to 3.4

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit d2def71944)
2021-07-28 21:14:35 +05:30
Niels de Vos
ce9e54e5bd ci: add Mergify backport rules for release-v3.4
The new `backport-to-release-v3.4` label can be added to PRs and Mergify
will create a backport once the PR for the devel branch has been merged.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-28 12:53:58 +05:30
Prasanna Kumar Kalever
52799da09d doc: add design doc for volume healer
Closes: #667

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-07-28 11:54:59 +05:30
Prasanna Kumar Kalever
ebe4e1f944 ci: ignore spell check for design proposal images
To avoid failures triggered by checking SVG image formats.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-07-28 11:54:59 +05:30
Prasanna Kumar Kalever
068e44bdb1 cleanup: move rbd-mirror image to a new directory
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-07-28 11:54:59 +05:30
Madhu Rajanna
080b251850 e2e: validate images in trash for rados namespace
added validation check to verify stale images in trash
for the rados namespace testing.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-28 03:48:33 +00:00
Madhu Rajanna
8f185bf7b2 rbd: use rados namespace for manager command
Currently we have a bug that we are not using rados
namespace when adding ceph manager command to
remove the image from the trash. This commit
adds the missing rados namespace when adding
ceph manager task.

without fix the image will be moved to trash
and no task will be added to remove from the
trash. it will become ceph responsibility to
remove the image from trash when it will cleanup
the trash.

workaroud: manually purge the trash

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-28 03:48:33 +00:00
Yug Gupta
d14c0afe28 doc: Add documentation for DR
Add documenation for Disaster Recovery
which steps to Failover and Failback in case
of a planned migration or a Disaster.

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2021-07-27 11:43:01 +00:00
Niels de Vos
ec6703ed58 rbd: rename encryption metadata keys to enable mirroring
RBD image metadata keys that start with '.rbd' are expected to be
internal to RBD itself and are not mirrored to remote sites. Renaming
the keys (dropping the '.' prefix) and using the new MigrateMetadata()
function now makes the keys available on remote sites too.

Closes: #2219
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-26 11:49:56 +00:00
Niels de Vos
607129171d rbd: move image metadata key migration to its own function
The new MigrateMetadata() function can be used to get the metadata of an
image with a deprecated and new key. Renaming metadata keys can be done
easily this way.

A default value will be set in the image metadata when it is missing
completely. But if the deprecated key was set, the data is stored under
the new key and the deprecated key is removed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-26 11:49:56 +00:00
Yati Padia
6691951453 rbd: use go-ceph for getImageMirroringStatus
Currently, getImageMirroringStatus() is using RBD CLI.
This commit converts RBD CLI to go-ceph API.

Fixes: #2120

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-26 06:37:40 +00:00
Niels de Vos
4e6d9be826 ci: fix yamllint error in generated golangci.yml file
When running 'make containerized-test' the following error gets
reported:

    yamllint -s -d '{extends: default, rules: {line-length: {allow-non-breakable-inline-mappings: true}},ignore: charts/*/templates/*.yaml}' ./scripts/golangci.yml
    ./scripts/golangci.yml
      179:81    error    line too long (84 > 80 characters)  (line-length)

The golangci.yml.in is used to generate golangci.yml, addressing the
line-length there resolves the issue.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-26 04:05:50 +00:00
Niels de Vos
e75d308b9c e2e: isRetryableAPIError() should match any etcdserver timeout
framework.RunKubectl() returns an error that does not end with
"etcdserver: request timed out", but contains the text somewhere in the
middle:

    error running /usr/bin/kubectl --server=https://192.168.39.57:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-a44ec4b4 create -f -:
    Command stdout:

    stderr:
    Error from server: error when creating "STDIN": etcdserver: request timed out

    error:
    exit status 1

isRetryableAPIError() should  return `true` for this case as well, so
instead of using HasSuffix(), we'll use Contains().

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-23 12:20:16 +00:00
Prasanna Kumar Kalever
75dda7ac0d e2e: add test for expansion of encrypted volumes
Also adds a test case to validate the default encryption type

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-07-23 10:00:23 +00:00