Commit Graph

2970 Commits

Author SHA1 Message Date
Madhu Rajanna
fb3835691f rbd: add support for deep-flatten image feature
as deep-flatten is long supported in ceph and its
enabled by default in the librbd, providing an option
to enable it in cephcsi for the rbd images we are
creating.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-28 13:10:03 +00:00
Humble Chirammal
eb40fbcb18 doc: remove deprecated alpha1 reference of snapshot objects
alpha1 version is deprecated and this commit correct the
example doc and make it GAd version

Additional note: we have already moved the artifacts to v1 version.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-25 08:42:46 +00:00
Robert Vasek
e9c6fab287 e2e: loadAppDeployment now sets ImagePullPolicy to PullIfNotPresent
Fixes: #2889

Signed-off-by: Robert Vasek <robert.vasek@cern.ch>
2022-02-24 15:38:24 +00:00
dependabot[bot]
ac6bfeb6ef rebase: bump k8s.io/kubernetes from 1.23.3 to 1.23.4
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.23.3 to 1.23.4.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.23.3...v1.23.4)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-23 14:47:13 +00:00
Madhu Rajanna
e9802c4940 cephfs: refactor cephfs core functions
This commits refactors the cephfs core
functions with interfaces. This helps in
better code structuring and writing the
unit test cases.

update #852

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-22 20:39:23 +00:00
dependabot[bot]
f19ca4a473 rebase: bump github.com/aws/aws-sdk-go from 1.42.53 to 1.43.3
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.53 to 1.43.3.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.53...v1.43.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-22 07:09:48 +00:00
dependabot[bot]
cd83c7be48 rebase: bump github.com/ceph/go-ceph from 0.13.0 to 0.14.0
Bumps [github.com/ceph/go-ceph](https://github.com/ceph/go-ceph) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/ceph/go-ceph/releases)
- [Changelog](https://github.com/ceph/go-ceph/blob/master/docs/release-process.md)
- [Commits](https://github.com/ceph/go-ceph/compare/v0.13.0...v0.14.0)

---
updated-dependencies:
- dependency-name: github.com/ceph/go-ceph
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-22 04:54:50 +00:00
Madhu Rajanna
3922cfc860 e2e: consider not found error in deployment check
it might need sometime for the deployment to
get created, consider the NotFound as a valid
error and retry again.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-21 12:18:39 +00:00
Silvio Gissi
9c50e255fb helm: make ceph.conf ConfigMap name configurable
ConfigMap name was hardcoded and led to conflicts. Fixes #2858.

Signed-off-by: Silvio Gissi <silvio@gissilabs.com>
2022-02-21 07:25:22 +00:00
Francesco Astegiano
4235178f7c helm: Add selinuxMount flag to enable/disable /etc/selinux host mount
Add selinuxMount flag to enable/disable /etc/selinux host mount inside pods
to support selinux-enabled filesystems

Signed-off-by: Francesco Astegiano <francesco.astegiano@gmail.com>
2022-02-16 12:48:00 +00:00
Niels de Vos
ea89b26f65 e2e: pass --ignore-not-found=true when deleting resources
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-15 16:21:07 +00:00
Niels de Vos
a669258688 cleanup: remove unused global variable in e2e
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-15 16:21:07 +00:00
Niels de Vos
26696b4979 e2e: retry deleting Rook artifacts in case of error
On occasion deploying CephFS components fail due to errors like these:

    failed to delete provisioner rbac .../csi-provisioner-rbac.yaml

By using the deleteResource() helper, an retry is done in case of a
failure.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-15 16:21:07 +00:00
Niels de Vos
8a0377ef02 e2e: retry creating StorageClass in case of error
There have been errors while CephFS tests were running, like:

    failed to create storageclass: etcdserver: request timed out

When retrying to create the StorageClass, the e2e tests are expected to
continue and (hopefully) succeed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-15 16:21:07 +00:00
Deividas Burškaitis
91c22f521b helm: add port sections to helm templates
to show what ports containers are exposing add port sections to nodeplugin
and provisioner helm templates

Signed-off-by: Deividas Burškaitis <deividas.burskaitis@oxylabs.io>
2022-02-15 10:06:26 +00:00
dependabot[bot]
7064642dee rebase: bump github.com/aws/aws-sdk-go from 1.42.48 to 1.42.53
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.48 to 1.42.53.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.48...v1.42.53)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-15 06:57:12 +00:00
Madhu Rajanna
46378f3bfc rbd: log stderror when running modprobe
logging the error is not user-friendly and
it contains system error message. Log the
stderr which is user-friendly error message
for identifying the problem.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-14 15:03:31 +00:00
Madhu Rajanna
0311eb5f44 helm: remove namespace from storageclass yaml
removes namespace from non-namespaced storageclass
object.

fixes: #2714

Replacement for #2715 as we didnt receive any update
and PR is already closed.

Co-authored-by: jhrcz-ls
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-11 12:32:58 +00:00
Sébastien BERNARD
ee8fb3f05f rbd: Fix dataPool in createVolumeResponse
Return the dataPool used to create the image instead of the default one
provided by the createVolumeRequest.
In case of topologyConstrainedDataPools, they may differ.
Don't add datapool if it's not present

Signed-off-by: Sébastien Bernard <sebastien.bernard@sfr.com>
2022-02-10 11:44:22 +00:00
Humble Chirammal
8f6a7da538 cephfs: dont set explicit permissions on the volume
At present we are node staging with worldwide permissions which is
not correct. We should allow the CO to take care of it and make
the decision. This commit also remove `fuseMountOptions` and
`KernelMountOptions` as they are no longer needed

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-09 17:30:29 +00:00
dependabot[bot]
c0e70ca8cb rebase: bump k8s.io/kubernetes from 1.23.2 to 1.23.3
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.23.2 to 1.23.3.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.23.2...v1.23.3)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 15:16:21 +00:00
dependabot[bot]
266e5b3b80 rebase: bump github.com/onsi/gomega from 1.18.0 to 1.18.1
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.18.0...v1.18.1)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 07:48:51 +00:00
dependabot[bot]
d1d4d92131 rebase: bump github.com/aws/aws-sdk-go from 1.42.40 to 1.42.48
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.40 to 1.42.48.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.40...v1.42.48)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-09 05:33:25 +00:00
Madhu Rajanna
2943555904 cephfs: fix omap deletion in DeleteSnapshot
the omap is stored with the requested
snapshot name not with the subvolume
snapshotname. This fix uses the correct
snapshot request name to cleanup the omap
once the subvolume snapshot is deleted.

fixes: #2832

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-02-08 20:37:53 +00:00
Humble Chirammal
ad6a3d7575 rbd: remove kp-metadata register functions of HPCS/Key Protect
This commit removes `kp-metadata` registration from existing HPCS
or Key Protect code as per the plan.

Fix #2816

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-08 18:27:03 +00:00
dependabot[bot]
7654a56367 rebase: bump google.golang.org/grpc from 1.43.0 to 1.44.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.43.0 to 1.44.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.43.0...v1.44.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-08 14:54:05 +00:00
Humble Chirammal
1c3baa0722 rbd: add AAD(additionalAuthData) while unwrapping the DEK
As we are using optional additional auth data while wrapping
the DEK, we have to send the same additionally while unwrapping.

Error:
```
 failed to unwrap the DEK: kp.Error: ..(INVALID_FIELD_ERR)',
 reasons='[INVALID_FIELD_ERR: The field `ciphertext` must be: the
 original base64 encoded ciphertext from the wrap operation
```

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-08 03:06:30 +00:00
dependabot[bot]
3014b722ad rebase: bump github.com/prometheus/client_golang from 1.12.0 to 1.12.1
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.0 to 1.12.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 14:00:43 +00:00
Niels de Vos
f6894909d7 util: use vaultNamespace if vaultAuthNamespace is not set
When a tenant configures `vaultNamespace` in their own ConfigMap, it is
not applied to the Vault configuration, unless `vaultAuthNamespace` is
set as well. This is unexpected, as the `vaultAuthNamespace` usually is
something configured globally, and not per tenant.

The `vaultAuthNamespace` is an advanced option, that is often not needed
to be configured. Only when tenants have to configure their own
`vaultNamespace`, it is possible that they need to use a different
`vaultAuthNamespace`. The default for the `vaultAuthNamespace` is now
the `vaultNamespace` value from the global configuration. Tenants can
still set it to something else in their own ConfigMap if needed.

Note that Hashicorp Vault Namespaces are only functional in the
Enterprise version of the product. Therefor this can not be tested in
the Ceph-CSI e2e with the Open Source version of Vault.

Fixes: https://bugzilla.redhat.com/2050056
Reported-by: Rachael George <rgeorge@redhat.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-07 08:20:48 +00:00
Rakshith R
3203673d17 cleanup: remove ceph.conf WA options which are already fixed
This commit removes ceph.conf WA options:
```
     # Workaround for http://tracker.ceph.com/issues/23446
     fuse_set_user_groups = false

     # ceph-fuse which uses libfuse2 by default has write buffer size of 2KiB
     # adding 'fuse_big_writes = true' option by default to override this limit
     # see https://github.com/ceph/ceph-csi/issues/1928
     fuse_big_writes = true
```
Since they are already fixed.

Refer: https://tracker.ceph.com/issues/44885
Refer: https://tracker.ceph.com/issues/23446
Closes: #2825

Signed-off-by: Rakshith R <rar@redhat.com>
2022-02-04 15:42:32 +00:00
Niels de Vos
07e4084080 build: remove container-id file before building
In case building a new container-image fails, the old image has already
been removed by the same make target. The container-id file that is used
to prevent unneeded rebuilds, causes build problems in case the
container-image in the container-id file does not exist (anymore).

By removing the container-id file before rebuilding the image, there
should not be any issues on subsequent (attempted fixed) builds of the
container-images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-04 10:23:58 +00:00
Niels de Vos
cbec296543 build: disable removed Apache Arrow repository
The CentOS 8 repository for Apache Arrow has been removed. This causes
container-image builds fail with the following error:

    Errors during downloading metadata for repository 'apache-arrow-centos':
      - Status code: 404 for https://apache.jfrog.io/artifactory/arrow/centos/8/x86_64/repodata/repomd.xml (IP: 54.190.66.70)
    Error: Failed to download metadata for repo 'apache-arrow-centos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

The Ceph base image has `arrow/centos/8` configured, maybe Apache Arrow
offers a CentOS Stream 8 repository now? Once the Ceph container-image
has been updated, the repository can be enabled again.

Ceph-CSI does not depend on Apache Arrow, so there is no functional
change by disabling the repository.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-04 10:23:58 +00:00
Humble Chirammal
de2489ed7d deploy: update csi-snapshotter sidecar to v5.0.1
This release of snapshotter has a breaking change as mentioned
in the release note:

Refer#
[1]: https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v5.0.0

RBAC rules are also updated with this commit.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-03 19:01:57 +00:00
Niels de Vos
693aabbe1e e2e: use pstree instead of ps for checking rbd-nbd process
The CentOS Stream 8 base container image does not have `ps` installed.
This causes CI jobs to fail, when checking for a restarted rbd-nbd
process.

Instead of using `ps`, the `pstree` command can be used. This will add
some ASCII-tree symbols in front of the command that is logged by the
e2e tests, but that is only used for manual reviewing and does not harm
the running test.

Fixes: #2850
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-03 13:25:49 +00:00
Humble Chirammal
13f1611396 deploy: correct typos in aws encryption yamls
The field name was wrong in example yaml and this correct the same

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-02 14:26:27 +00:00
Humble Chirammal
66f8a51c93 doc: clarified subvol in shallow-ro-vol.md
instead of subvol, subvolume has been used for consistency across
the doc

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-02 13:46:52 +00:00
Humble Chirammal
e1cbd90c0b doc: correct typo in hpcs or key protect design
File name is corrected and also a typo in the same

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-02-02 13:46:52 +00:00
Madhu Rajanna
28fef9b379 cleanup: remove thick provisioning code
This commit removes the thick provisioning
code as thick provisioning is deprecated in
cephcsi 3.5.0.

fixes: #2795

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-28 11:17:15 +00:00
Humble Chirammal
4ee4fdfebd rbd: unexport SecretsKMS from KMS implementation
This commit unexport SecretsKMS from KMS implementation.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
4058246637 rbd: unexport vaultTokenSA struct from KMS implementation
This commit unexport the vaultTokenSA from the vault KMS
implementation

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
b75c562217 rbd: Unexport VaultTenantSA struct from KMS implementation
This commit unexport VaultTenantSA struct from KMS implemenation
of Vault KMS.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
c8a3b9352e rbd: Unexport SecretsMetadataKMS struct
This commit unexport SecretsMetadataKMS struct from KMS
implementation

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
3f18d6e4b4 rbd: Unexport IntegratedDEK struct from kms
This commit unexport IntegratedDEK struct from KMS
implementation

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
6141aabcd2 rbd: unexport KeyProtect kms struct
At present the KMS structs are exported and ideally we should be
able to work without exporting the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Humble Chirammal
a86121f756 rbd: unexport aws kms structs
At present the KMS structs are exported and ideally we should be
able to work without exporting the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-28 06:55:12 +00:00
Niels de Vos
a553157b15 ci: don't dismiss review if PR has not merged
Pull requests are not going to be queued if at the same time the pull
request review are dismiss

Original-author: Mehdi ABAAKOUK <sileht@sileht.net>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-01-27 20:57:15 +05:30
Humble Chirammal
177cc078e1 ci: remove duplicate validation DCO in default rule
removing duplicated validation of DCO from the default
rule.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-01-27 16:07:45 +01:00
Madhu Rajanna
992d257530 cephfs: fix error logging in filesystem.go
fix error message logging in filesystem.go

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 14:31:12 +00:00
Madhu Rajanna
14c008c419 cleanup: use interface in filesystem.go
Currently, we are using methods and all the methods
makes a network call to fetch details from the ceph
clusters, its difficult to write test cases for
these functions, if we move to the interfaces
we can make use of mock to write unit testing
for the caller functions.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-01-27 14:31:12 +00:00
dependabot[bot]
ab20ab728f rebase: bump github.com/onsi/gomega from 1.17.0 to 1.18.0
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.17.0...v1.18.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-27 09:39:21 +00:00