Commit Graph

2750 Commits

Author SHA1 Message Date
Madhu Rajanna
3338a1013b cephfs: fix omap deletion in DeleteSnapshot
the omap is stored with the requested
snapshot name not with the subvolume
snapshotname. This fix uses the correct
snapshot request name to cleanup the omap
once the subvolume snapshot is deleted.

fixes: #2832

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2943555904)
2022-02-09 07:44:23 +00:00
Humble Chirammal
bb94d537b0 deploy: correct typos in aws encryption yamls
The field name was wrong in example yaml and this correct the same

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit 13f1611396)
2022-02-08 18:16:35 +00:00
Niels de Vos
a05f63d842 util: use vaultNamespace if vaultAuthNamespace is not set
When a tenant configures `vaultNamespace` in their own ConfigMap, it is
not applied to the Vault configuration, unless `vaultAuthNamespace` is
set as well. This is unexpected, as the `vaultAuthNamespace` usually is
something configured globally, and not per tenant.

The `vaultAuthNamespace` is an advanced option, that is often not needed
to be configured. Only when tenants have to configure their own
`vaultNamespace`, it is possible that they need to use a different
`vaultAuthNamespace`. The default for the `vaultAuthNamespace` is now
the `vaultNamespace` value from the global configuration. Tenants can
still set it to something else in their own ConfigMap if needed.

Note that Hashicorp Vault Namespaces are only functional in the
Enterprise version of the product. Therefor this can not be tested in
the Ceph-CSI e2e with the Open Source version of Vault.

Fixes: https://bugzilla.redhat.com/2050056
Reported-by: Rachael George <rgeorge@redhat.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit f6894909d7)
2022-02-08 10:19:57 +00:00
Humble Chirammal
043a71aad1 rbd: add AAD(additionalAuthData) while unwrapping the DEK
As we are using optional additional auth data while wrapping
the DEK, we have to send the same additionally while unwrapping.

Error:
```
 failed to unwrap the DEK: kp.Error: ..(INVALID_FIELD_ERR)',
 reasons='[INVALID_FIELD_ERR: The field `ciphertext` must be: the
 original base64 encoded ciphertext from the wrap operation
```

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit 1c3baa0722)
2022-02-08 07:41:55 +00:00
Niels de Vos
893ad40d15 build: remove container-id file before building
In case building a new container-image fails, the old image has already
been removed by the same make target. The container-id file that is used
to prevent unneeded rebuilds, causes build problems in case the
container-image in the container-id file does not exist (anymore).

By removing the container-id file before rebuilding the image, there
should not be any issues on subsequent (attempted fixed) builds of the
container-images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 07e4084080)
2022-02-07 20:25:40 +00:00
Niels de Vos
dfe91bd00f build: disable removed Apache Arrow repository
The CentOS 8 repository for Apache Arrow has been removed. This causes
container-image builds fail with the following error:

    Errors during downloading metadata for repository 'apache-arrow-centos':
      - Status code: 404 for https://apache.jfrog.io/artifactory/arrow/centos/8/x86_64/repodata/repomd.xml (IP: 54.190.66.70)
    Error: Failed to download metadata for repo 'apache-arrow-centos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

The Ceph base image has `arrow/centos/8` configured, maybe Apache Arrow
offers a CentOS Stream 8 repository now? Once the Ceph container-image
has been updated, the repository can be enabled again.

Ceph-CSI does not depend on Apache Arrow, so there is no functional
change by disabling the repository.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit cbec296543)
2022-02-07 20:25:40 +00:00
Niels de Vos
f0db85daa9 e2e: use pstree instead of ps for checking rbd-nbd process
The CentOS Stream 8 base container image does not have `ps` installed.
This causes CI jobs to fail, when checking for a restarted rbd-nbd
process.

Instead of using `ps`, the `pstree` command can be used. This will add
some ASCII-tree symbols in front of the command that is logged by the
e2e tests, but that is only used for manual reviewing and does not harm
the running test.

Fixes: #2850
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 693aabbe1e)
2022-02-03 17:55:30 +00:00
Humble Chirammal
014ab47a57 rbd: change the keyprotect metadata name to ibmkeyprotect
To be consistent with other components and also to explictly
state it belong to `ibm keyprotect` service introducing this
change

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit f822600689)
2022-01-26 07:32:21 +00:00
Humble Chirammal
bf5553971d e2e: add podsecuritycontext fsgroup for normal user validation
considering the pod has run as normal user, the fsgroup has also
set to the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit 7ff048bf1e)
2022-01-26 04:27:48 +00:00
Humble Chirammal
825649185e rbd: dont attempt explicit permission mod change from the RBD driver
currently we are overriding the permission to `0o777` at time of node
stage which is not the correct action. That said, this permission
change causes an extra permission correction at time of nodestaging
by the CO while the FSGROUP change policy has been set to
`OnRootMismatch`.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit bf4ba0ec84)
2022-01-26 04:27:48 +00:00
Madhu Rajanna
bbc8ab2bd3 revert: update templates to point to 3.5-canary
updating deployment templates to point to
3.5-canary image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-24 12:15:03 +00:00
Madhu Rajanna
00f88e58a0 helm: changes the image to v3.5.1 instead of canary for release
This commit changes the provisioner and plugin images
for 3.5.1 image in helm charts.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-21 17:23:03 +00:00
Madhu Rajanna
3c4177a4c2 deploy: changes the image to v3.5.1 instead of canary for release
This commit changes the provisioner and plugin images for 3.5.1 image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-21 17:23:03 +00:00
Madhu Rajanna
d4df72c8af deploy: change minikube image for 3.5.1 release
This update the minikube cluster image to 3.5.1 release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-21 17:23:03 +00:00
Madhu Rajanna
0f1b41992c build: update build.env for 3.5.1 release
This update the build value specific to 3.5.1 release

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-21 17:23:03 +00:00
Madhu Rajanna
9afb3c078e build: pull ceph base image from quay.io
pull the ceph image from quay.io instead
of dockerhub.
From ceph doc, the images are available
in both quay and dockerhub
https://docs.ceph.com/en/latest/install/
containers/#official-releases but latest
images are not updated in dockerhub.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 6bdeffda59)
2022-01-21 04:37:37 +00:00
Madhu Rajanna
dcc0fdc736 cephfs: log error message if clone fails
During CreateVolume from snapshot/volume,
its difficult to identify if the clone is
failed and a new clone is created. In case
of clone failure logging the error message
for better debugging.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2daf2f9f0c)
2022-01-19 16:32:55 +00:00
Humble Chirammal
cc0babffa9 deploy: revert deploy template changes for release-3.5
This commit revert the template changes brought in for release-3.5
and making it refer to canary.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-18 15:20:16 +00:00
Humble Chirammal
c1100dec15 helm: add release-3.5 tag for the release
This commit tag the release 3.5 helm charts

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-17 08:40:27 +00:00
Humble Chirammal
43182b39fe deploy: changes the image to v3.5.0 instead of canary for release
This commit changes the provisioner and plugin images for 3.5.0 image.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-14 16:26:28 +00:00
Humble Chirammal
bdf5fc302a deploy: change minikube image for 3.5.0 release
This update the minikube cluster image to 3.5.0 release.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-14 16:26:28 +00:00
Humble Chirammal
065c0b6c53 build: update build.env for 3.5.0 release
This update the build value specific to 3.5.0 release

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-14 16:26:28 +00:00
Humble Chirammal
1efaa63bcf deploy: update node driver registrar container to v2.4.0
This commit updates the node driver registrar container to latest
version.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit 0078e5c8e7)
2022-01-14 19:16:02 +05:30
Humble Chirammal
814e22c4ab deploy: update sidecar versions to deploy values.
This commit adds latest versions of the sidecars to the build.env
to pass the latest versions on the deployment

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit c0c2d72933)
2022-01-14 19:16:02 +05:30
Humble Chirammal
42f38d878b deploy: update csi-attacher to v3.4.0
This commit update the csi-attacher sidecar version to v3.4.0

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit 0ab717f06f)
2022-01-14 19:16:02 +05:30
Humble Chirammal
1d8a5c9e0f deploy: update sidecars to latest versions.
This commit updates sidecars to the latest available version
which is compatible with kubernetes 1.23 and csi spec 1.5

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit ea8e360888)
2022-01-14 19:16:02 +05:30
Prasanna Kumar Kalever
02010955d8 doc: add kernel version recommendation for rbd-nbd
Without commit [1] Kernel doesn't handle io-timeout=0 correctly
Hence we recommend Kernel version 5.4 or higher that has commit [1]

[1] https://bit.ly/34CFh06

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 1c153b120c)
2022-01-13 14:10:51 +00:00
Humble Chirammal
9e7c3c9a7c doc: add udgrade from v3.4 to v3.5
This commit adds the upgrade documentation from v3.4 to v3.5

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit b151325871)
2022-01-13 10:42:32 +00:00
Humble Chirammal
dbb523c150 doc: add in-tree volume to csi migration usage doc
This commit adds in-tree to csi volume migration usage documentation
to the examples.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-12 14:54:40 +00:00
Madhu Rajanna
42d6130cbc e2e: cephfs e2e for bigger size clone,restore pvc
adding E2E to verify the bigger size clone
and restore pvc from a smaller size pvc/snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-12 10:44:11 +00:00
Madhu Rajanna
ef14ea7723 cephfs: resize cloned, restored volume if required
Currently, as a workaround, we are calling
the resize volume on the cloned, restore volumes
to adjust the cloned, restored volumes.
With this fix, we are calling the resize volume
only if there is a size mismatch with requested
and the volume from which the new volume needs
to be created.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-12 10:44:11 +00:00
Humble Chirammal
0bd1d44dc0 e2e: add rwop validation for cephfs volumes
This commit adds the rwop validation tests for cephfs.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
8737d7e0a5 e2e: add rwop validation tests for rbd filemode and block mode
This commit adds the validation tests for rbd block and filemode
PVCs.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
375e9c8a51 e2e: add rwop validation helper function in pod.go
This commit adds the rwop validation helper for rbd and cephfs tests.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
8f7016229a e2e: add example yamls for RWOP POD and PVC
This commit add example yamls for RBD and CephFS Pod and PVCs

RBD:
 Raw Block Volume and File Mode PVCs with RWOP accessmode
 Raw Block Volume POD and FileMode POD yamls referring RWOP PVC

CephFS:
 RWOP PVC and POD yaml referring RWOP PVC

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
233ac8f936 deploy: adjust minikube deployment for RWOP support
The ReadWriteOncePod feature gate need to be enabled only when we
are operating on kube 1.22 or above cluster. This commit adds the
logic to parse the kubernetes cluster at time of minikube deployment
and if it is above v1.22, enable the RWOP feature gate

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
4a69378698 rbd: introduce a helper function to detect multi writer,block & rwofile
SINGLE_NODE_WRITER capability ambiguity has been fixed in csi spec v1.5
which allows the SP drivers to declare more granular WRITE capability in form
of SINGLE_NODE_SINGLE_WRITER or SINGLE_NODE_MULTI_WRITER.

These are not really new capabilities rather capabilities introduced to
get the desired functionality from CO side based on the capabilities SP
driver support for various CSI operations, this new capabilities also help
to address new access mode RWOP (readwriteoncepod).

This commit adds a helper function which identity the request is of
multiwriter mode and also validates whether it is filesystem mode or
block mode. Based on the inspection it fails to allow multi write
requests for filesystem mode and only allow multi write request against
block mode.

This commit also adds unit tests for isMultiWriterBlock function which
validates various accesstypes and accessmodes.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
68350e8815 cephfs: add SINGLE_NODE_{SINGLE/MULTI}_WRITER capability
SINGLE_NODE_WRITER capability ambiguity has been fixed in csi spec v1.5
which allows the SP drivers to declare more granular WRITE capability.
These are not really new capabilities rather capabilities introduced to
get the desired functionality from CO side based on the capabilities SP
driver support for various CSI operations.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
3730a462f4 rbd: add SINGLE_NODE{SINGLE_MULTI}_WRITER capabilities
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
d0b846772a doc: add RWOP support matrix to README.md
This commit add RWOP support matrix to README

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 19:40:22 +00:00
Humble Chirammal
bc354b6fb5 rbd: add BaseURL and tokenURL configuration
This commit adds optional BaseURL and TokenURL configuration to
key protect/hpcs configuration and client connections, if not
provided default values are used.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 21:12:56 +05:30
Niels de Vos
0c109c5dec ci: remove queue actions from non-merge rules in Mergify config
By the addition of the queue rules in the Mrgify configuration, all PRs
that require changes, or have been updated and review should be dropped,
are now added to the queue for merging. This is obviously not what we
want.

Fixes: 43fc945 ("ci: move from merge action to queue action")
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-01-11 13:57:17 +05:30
Humble Chirammal
43fc945be6 ci: move from merge action to queue action
as mentioned in the below blog the support for strict mode
and merge action will be done soon in mergify. This brings
the change requested for the same.

Ref# https://blog.mergify.com/strict-mode-deprecation/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-11 09:11:46 +01:00
Yug Gupta
9d34809425 rbd: add NetworkFence operation
Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00
Yug Gupta
fa5866deec ci: add unit test for NetworkFence grpc calls
Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00
Yug Gupta
29782bf377 rbd: implement UnfenceClusterNetwork
implement UnfenceClusterNetwork grpc call
which allows to unblock the access to a
CIDR block by removing it from network fence.

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00
Yug Gupta
ebd8a762f0 rbd: implement FenceClusterNetwork
implement FenceClusterNetwork grpc call which
allows to blocks access to a CIDR block by
creating a network fence.

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00
Yug Gupta
ab15053fef ci: add unit test for networkfencing util
Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00
Yug Gupta
7d5879ad81 rbd: add network fencing utils
Convert the CIDR block into a range of IPs,
and then add network fencing via "ceph osd blocklist"
for each IP in that range.

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00
Yug Gupta
96f8f2ac5a cleanup: re-run "go mod vendor" with Go 1.17
Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2022-01-07 14:48:12 +00:00