Commit Graph

3858 Commits

Author SHA1 Message Date
Niels de Vos
26696b4979 e2e: retry deleting Rook artifacts in case of error
On occasion deploying CephFS components fail due to errors like these:

    failed to delete provisioner rbac .../csi-provisioner-rbac.yaml

By using the deleteResource() helper, an retry is done in case of a
failure.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-15 16:21:07 +00:00
Niels de Vos
8a0377ef02 e2e: retry creating StorageClass in case of error
There have been errors while CephFS tests were running, like:

    failed to create storageclass: etcdserver: request timed out

When retrying to create the StorageClass, the e2e tests are expected to
continue and (hopefully) succeed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-15 16:21:07 +00:00
Deividas Burškaitis
91c22f521b helm: add port sections to helm templates
to show what ports containers are exposing add port sections to nodeplugin
and provisioner helm templates

Signed-off-by: Deividas Burškaitis <deividas.burskaitis@oxylabs.io>
2022-02-15 10:06:26 +00:00
dependabot[bot]
7064642dee rebase: bump github.com/aws/aws-sdk-go from 1.42.48 to 1.42.53
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.48 to 1.42.53.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.48...v1.42.53)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-15 06:57:12 +00:00
Madhu Rajanna
46378f3bfc rbd: log stderror when running modprobe
logging the error is not user-friendly and
it contains system error message. Log the
stderr which is user-friendly error message
for identifying the problem.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-14 15:03:31 +00:00
Madhu Rajanna
0311eb5f44 helm: remove namespace from storageclass yaml
removes namespace from non-namespaced storageclass
object.

fixes: #2714

Replacement for #2715 as we didnt receive any update
and PR is already closed.

Co-authored-by: jhrcz-ls
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-11 12:32:58 +00:00
Sébastien BERNARD
ee8fb3f05f rbd: Fix dataPool in createVolumeResponse
Return the dataPool used to create the image instead of the default one
provided by the createVolumeRequest.
In case of topologyConstrainedDataPools, they may differ.
Don't add datapool if it's not present

Signed-off-by: Sébastien Bernard <sebastien.bernard@sfr.com>
2022-02-10 11:44:22 +00:00
Humble Chirammal
8f6a7da538 cephfs: dont set explicit permissions on the volume
At present we are node staging with worldwide permissions which is
not correct. We should allow the CO to take care of it and make
the decision. This commit also remove `fuseMountOptions` and
`KernelMountOptions` as they are no longer needed

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-09 17:30:29 +00:00
dependabot[bot]
c0e70ca8cb rebase: bump k8s.io/kubernetes from 1.23.2 to 1.23.3
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.23.2 to 1.23.3.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.23.2...v1.23.3)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 15:16:21 +00:00
dependabot[bot]
266e5b3b80 rebase: bump github.com/onsi/gomega from 1.18.0 to 1.18.1
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.18.0...v1.18.1)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 07:48:51 +00:00
dependabot[bot]
d1d4d92131 rebase: bump github.com/aws/aws-sdk-go from 1.42.40 to 1.42.48
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.40 to 1.42.48.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.40...v1.42.48)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 05:33:25 +00:00
Madhu Rajanna
2943555904 cephfs: fix omap deletion in DeleteSnapshot
the omap is stored with the requested
snapshot name not with the subvolume
snapshotname. This fix uses the correct
snapshot request name to cleanup the omap
once the subvolume snapshot is deleted.

fixes: #2832

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-08 20:37:53 +00:00
Humble Chirammal
ad6a3d7575 rbd: remove kp-metadata register functions of HPCS/Key Protect
This commit removes `kp-metadata` registration from existing HPCS
or Key Protect code as per the plan.

Fix #2816

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-08 18:27:03 +00:00
dependabot[bot]
7654a56367 rebase: bump google.golang.org/grpc from 1.43.0 to 1.44.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.43.0 to 1.44.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.43.0...v1.44.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-08 14:54:05 +00:00
Humble Chirammal
1c3baa0722 rbd: add AAD(additionalAuthData) while unwrapping the DEK
As we are using optional additional auth data while wrapping
the DEK, we have to send the same additionally while unwrapping.

Error:
```
 failed to unwrap the DEK: kp.Error: ..(INVALID_FIELD_ERR)',
 reasons='[INVALID_FIELD_ERR: The field `ciphertext` must be: the
 original base64 encoded ciphertext from the wrap operation
```

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-08 03:06:30 +00:00
dependabot[bot]
3014b722ad rebase: bump github.com/prometheus/client_golang from 1.12.0 to 1.12.1
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.0 to 1.12.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 14:00:43 +00:00
Niels de Vos
f6894909d7 util: use vaultNamespace if vaultAuthNamespace is not set
When a tenant configures `vaultNamespace` in their own ConfigMap, it is
not applied to the Vault configuration, unless `vaultAuthNamespace` is
set as well. This is unexpected, as the `vaultAuthNamespace` usually is
something configured globally, and not per tenant.

The `vaultAuthNamespace` is an advanced option, that is often not needed
to be configured. Only when tenants have to configure their own
`vaultNamespace`, it is possible that they need to use a different
`vaultAuthNamespace`. The default for the `vaultAuthNamespace` is now
the `vaultNamespace` value from the global configuration. Tenants can
still set it to something else in their own ConfigMap if needed.

Note that Hashicorp Vault Namespaces are only functional in the
Enterprise version of the product. Therefor this can not be tested in
the Ceph-CSI e2e with the Open Source version of Vault.

Fixes: https://bugzilla.redhat.com/2050056
Reported-by: Rachael George <rgeorge@redhat.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-07 08:20:48 +00:00
Rakshith R
3203673d17 cleanup: remove ceph.conf WA options which are already fixed
This commit removes ceph.conf WA options:
```
     # Workaround for http://tracker.ceph.com/issues/23446
     fuse_set_user_groups = false

     # ceph-fuse which uses libfuse2 by default has write buffer size of 2KiB
     # adding 'fuse_big_writes = true' option by default to override this limit
     # see https://github.com/ceph/ceph-csi/issues/1928
     fuse_big_writes = true
```
Since they are already fixed.

Refer: https://tracker.ceph.com/issues/44885
Refer: https://tracker.ceph.com/issues/23446
Closes: #2825

Signed-off-by: Rakshith R <rar@redhat.com>
2022-02-04 15:42:32 +00:00
Niels de Vos
07e4084080 build: remove container-id file before building
In case building a new container-image fails, the old image has already
been removed by the same make target. The container-id file that is used
to prevent unneeded rebuilds, causes build problems in case the
container-image in the container-id file does not exist (anymore).

By removing the container-id file before rebuilding the image, there
should not be any issues on subsequent (attempted fixed) builds of the
container-images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-04 10:23:58 +00:00
Niels de Vos
cbec296543 build: disable removed Apache Arrow repository
The CentOS 8 repository for Apache Arrow has been removed. This causes
container-image builds fail with the following error:

    Errors during downloading metadata for repository 'apache-arrow-centos':
      - Status code: 404 for https://apache.jfrog.io/artifactory/arrow/centos/8/x86_64/repodata/repomd.xml (IP: 54.190.66.70)
    Error: Failed to download metadata for repo 'apache-arrow-centos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

The Ceph base image has `arrow/centos/8` configured, maybe Apache Arrow
offers a CentOS Stream 8 repository now? Once the Ceph container-image
has been updated, the repository can be enabled again.

Ceph-CSI does not depend on Apache Arrow, so there is no functional
change by disabling the repository.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-04 10:23:58 +00:00
Humble Chirammal
de2489ed7d deploy: update csi-snapshotter sidecar to v5.0.1
This release of snapshotter has a breaking change as mentioned
in the release note:

Refer#
[1]: https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v5.0.0

RBAC rules are also updated with this commit.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-03 19:01:57 +00:00
Niels de Vos
693aabbe1e e2e: use pstree instead of ps for checking rbd-nbd process
The CentOS Stream 8 base container image does not have `ps` installed.
This causes CI jobs to fail, when checking for a restarted rbd-nbd
process.

Instead of using `ps`, the `pstree` command can be used. This will add
some ASCII-tree symbols in front of the command that is logged by the
e2e tests, but that is only used for manual reviewing and does not harm
the running test.

Fixes: #2850
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-03 13:25:49 +00:00
Humble Chirammal
13f1611396 deploy: correct typos in aws encryption yamls
The field name was wrong in example yaml and this correct the same

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-02 14:26:27 +00:00
Humble Chirammal
66f8a51c93 doc: clarified subvol in shallow-ro-vol.md
instead of subvol, subvolume has been used for consistency across
the doc

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-02 13:46:52 +00:00
Humble Chirammal
e1cbd90c0b doc: correct typo in hpcs or key protect design
File name is corrected and also a typo in the same

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-02 13:46:52 +00:00
Madhu Rajanna
28fef9b379 cleanup: remove thick provisioning code
This commit removes the thick provisioning
code as thick provisioning is deprecated in
cephcsi 3.5.0.

fixes: #2795

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-28 11:17:15 +00:00
Humble Chirammal
4ee4fdfebd rbd: unexport SecretsKMS from KMS implementation
This commit unexport SecretsKMS from KMS implementation.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
4058246637 rbd: unexport vaultTokenSA struct from KMS implementation
This commit unexport the vaultTokenSA from the vault KMS
implementation

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
b75c562217 rbd: Unexport VaultTenantSA struct from KMS implementation
This commit unexport VaultTenantSA struct from KMS implemenation
of Vault KMS.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
c8a3b9352e rbd: Unexport SecretsMetadataKMS struct
This commit unexport SecretsMetadataKMS struct from KMS
implementation

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
3f18d6e4b4 rbd: Unexport IntegratedDEK struct from kms
This commit unexport IntegratedDEK struct from KMS
implementation

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
6141aabcd2 rbd: unexport KeyProtect kms struct
At present the KMS structs are exported and ideally we should be
able to work without exporting the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
a86121f756 rbd: unexport aws kms structs
At present the KMS structs are exported and ideally we should be
able to work without exporting the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Niels de Vos
a553157b15 ci: don't dismiss review if PR has not merged
Pull requests are not going to be queued if at the same time the pull
request review are dismiss

Original-author: Mehdi ABAAKOUK <sileht@sileht.net>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-01-27 20:57:15 +05:30
Humble Chirammal
177cc078e1 ci: remove duplicate validation DCO in default rule
removing duplicated validation of DCO from the default
rule.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-27 16:07:45 +01:00
Madhu Rajanna
992d257530 cephfs: fix error logging in filesystem.go
fix error message logging in filesystem.go

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 14:31:12 +00:00
Madhu Rajanna
14c008c419 cleanup: use interface in filesystem.go
Currently, we are using methods and all the methods
makes a network call to fetch details from the ceph
clusters, its difficult to write test cases for
these functions, if we move to the interfaces
we can make use of mock to write unit testing
for the caller functions.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 14:31:12 +00:00
dependabot[bot]
ab20ab728f rebase: bump github.com/onsi/gomega from 1.17.0 to 1.18.0
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.17.0...v1.18.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-27 09:39:21 +00:00
Madhu Rajanna
e08084fbb6 e2e: update cephfs data pool name
cephfs data pool name is changed from filesystem-data0
to filesystem-replicated in Rook 1.8. updating
the cephcsi helper functions also to use new
pool names.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
8a479d3865 deploy: use osd as failureDomain for ec pool
in Rook 1.8 templates the failureDomain is
set to host for ec pools, as we are using
single node minikube cluster, setting the
failureDomain for osd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
0d6db19252 deploy: disable startupProbe
As startup probe is failing on the ceph pods,
disabling startupProbe in ceph cluster CRD.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
e5100eacd4 deploy: add check for device-health-metrics pool
Rook creates a detault pool with name
device_health_metrics for
device-health-metrics CephBlockPool CR.

device-health-metrics is added to cluster-test.yaml
in Rook 1.8

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
493db5c183 deploy: log complete ouput of rook operator
kubectl log with labels will log only
last 10 lines by default adding tail=-1
to log complete output.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
25796aa4ea deploy: update rook version to 1.8.2
update rook version to 1.8.2 in build.env
to pick up the latest rook release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
04f2261e60 deploy: support rook deployment for v1.8+
In Rook v1.8+ the path for the deployment articafts
are changed from `"cluster/examples/kubernetes/ceph`
to `deploy/examples`.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
Madhu Rajanna
c32bf71846 deploy: re-alignment of code in rook.sh
currently, rook.sh code is not aligned properly
unside the functions, this commit does
the code alignment.

PS: this is done by vscode for me :)

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 04:18:01 +00:00
dependabot[bot]
901a1368ab rebase: bump k8s.io/kubernetes from 1.23.1 to 1.23.2
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.23.1 to 1.23.2.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.23.1...v1.23.2)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-26 20:05:06 +00:00
Madhu Rajanna
3a445cfc36 deploy: update resizer to 1.4.0
updating external resizer image version
from 1.3.0 to latest available release i.e
1.4.0

1.4.0 changelog link
https://github.com/kubernetes-csi/
external-resizer/blob/master/CHANGELOG/CHANGELOG-1.4.md

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-26 15:22:24 +00:00
Humble Chirammal
f822600689 rbd: change the keyprotect metadata name to ibmkeyprotect
To be consistent with other components and also to explictly
state it belong to `ibm keyprotect` service introducing this
change

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-26 02:28:05 +00:00
Humble Chirammal
7ff048bf1e e2e: add podsecuritycontext fsgroup for normal user validation
considering the pod has run as normal user, the fsgroup has also
set to the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-25 16:25:11 +00:00