Commit Graph

3463 Commits

Author SHA1 Message Date
Niels de Vos
92866f46fd e2e: allow passing CephFS filesystem name on CLI
A new -filesystem=... option has been added so that the e2e tests can
run against environments that do not have a "myfs" CephFS filesystem.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-04-12 22:43:15 +00:00
Niels de Vos
ab5ca13586 e2e: return useful error message when ConfigMap creation fails
In case the toolbox pod is not available, the error message lists that
no Pods are found, but there is no hint about the toolbox. By mentioning
the toolbox in the error message, it suggests a good place to start
troubleshooting the environment.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-04-12 22:43:15 +00:00
Humble Chirammal
6043a30877 e2e: change the default release version of upgrade tests
This commit correct the release version of upgrade tests from
unsupported 3.3.1 to supported version.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-12 08:54:55 +00:00
Humble Chirammal
a6a1a7775e doc: mention csi string in flag arguments correctly
This commit correct the csi string mention in the documentation.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-12 08:54:55 +00:00
Madhu Rajanna
dffb6e72c2 rbd: check nbd tool features only for rbd driver
calling setRbdNbdToolFeatures inside an init
gets called in main.go for both cephfs and rbd
driver. instead of calling it in init function
calling this in rbd driver.go as this is specific
to rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-04-11 21:18:27 +00:00
dependabot[bot]
539050a857 rebase: bump github.com/aws/aws-sdk-go-v2/service/sts
Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.16.0 to 1.16.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.16.0...service/efs/v1.16.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 17:17:05 +00:00
Humble Chirammal
a44a97f3c5 deploy: snapshot deployment is no longer beta
Considering snapshot controllers have been moved to GA since
kube version 1.20, we no longer need to have a mention of beta
version of the same in our deployment.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-11 12:15:48 +00:00
Humble Chirammal
9b8bcbd874 e2e: correct the cephfs fuse verification podnames
The podnames were wrongly mention in the e2e tests and this correct
the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-11 12:15:48 +00:00
Humble Chirammal
959df4dbac doc: correct typos in struct field comments and release.md
corrected strings in the release guide and util server.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-11 06:23:25 +00:00
Prasanna Kumar Kalever
4100ad2406 e2e: add test case for snapshot metadata validation
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
d870cb152a deploy: add --extra-create-metadata arg to csi-snapshotter sidecar
This argument in csi-snapshotter sidecar allows us to receive
snapshot-name/snapshot-namespace/snapshotcontent-name metadata in the
CreateSnapshot() request.

For ex:

csi.storage.k8s.io/volumesnapshot/name
csi.storage.k8s.io/volumesnapshot/namespace
csi.storage.k8s.io/volumesnapshotcontent/name

This is a useful information which can be used depend on the use case we
have at our driver. The features like adding metadata to snapshot image
can consume this based on the need.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
41fe2c7dda rbd: set metadata on the snapshot
Set snapshot-name/snapshot-namespace/snapshotcontent-name details
on RBD backend snapshot image as metadata on snapshot

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
463990c999 e2e: add test cases for image metadata validation
* create a PVC and check PVC/PV metadata on RBD image
* create and delete a PVC, attach the old PV to a new PVC and check if
  PVC metadata is updated on  RBD image

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
0ef79c6fc0 rbd: set metadata on restart of provisioner pod
Make sure to set metadata when image exist, i.e. if the provisioner pod
is restarted while createVolume is in progress, say it created the image
but didn't yet set the metadata.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
ae5925f04c rbd: update PV/PVC metadata on a reattach of PV
Example if a PVC was delete by setting `persistentVolumeReclaimPolicy` as
`Retain` on PV, and PV is reattached to a new PVC, we make sure to update
PV/PVC image metadata on a PV reattach.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
0119d69ab2 rbd: set PV/PVC details on the image as metadata on create
This helps Monitoring solutions without access to Kubernetes clusters to
display the details of the PV/PVC/NameSpace in their dashboard.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Prasanna Kumar Kalever
4d750ed0e5 rbd: add set/Get VolumeMetadata() utility function
Define and use PV and PVC metadata keys used by external provisioner.
The CSI external-provisioner (v1.6.0+) introduces the
--extra-create-metadata flag, which automatically sets map<string, string>
parameters in the CSI CreateVolumeRequest.

Add utility functions to set/Get PV/PVC/PVCNamespace metadata on image

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-08 15:43:14 +00:00
Madhu Rajanna
7b2aef0d81 util: add support for the nsenter
add support to run rbd map and mount -t
commands with the nsenter.

complete design of pod/multus network
is added here https://github.com/rook/rook/
blob/master/design/ceph/multus-network.md#csi-pods

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-04-08 10:23:21 +00:00
Madhu Rajanna
6333c4b1e6 ci: remove duplicate release-3.5 merge rules
we already have generic rules to merge the PR's
in devel and release branches with `automatic merge`
rules. Removing the duplicate release-3.5 rule.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-04-06 20:59:07 +05:30
Madhu Rajanna
519202629e ci: remove release-v3.4 mergify rules
as we have deprecated the 3.4 release
removing the mergify rules for the same.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-04-06 20:59:07 +05:30
Madhu Rajanna
1b52256a22 ci: add mergify rule for 3.6 backport
added mergify rules to create the backport
PR for release-3.6 branch.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-04-06 20:59:07 +05:30
Humble Chirammal
2bb5ca9dbc doc: move release 3.4.0 to unsupported version
As we have 3.6.0 release available now, moving 3.4.0
to unsupported version.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-06 09:49:00 +00:00
dependabot[bot]
6ba740edc3 rebase: bump github.com/aws/aws-sdk-go from 1.43.22 to 1.43.32
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.43.22 to 1.43.32.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.43.22...v1.43.32)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-06 06:38:20 +00:00
Humble Chirammal
bd3db134b9 build: consume quincy release of Ceph
This promotes the ceph release to Quincy

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-05 02:37:28 +00:00
Humble Chirammal
32ecbdeb71 doc: update documentation for release 3.6.0
This commit add upgrade documentation for release 3.6.0
and also update support matrix for v3.6.0.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-04-04 13:29:08 +00:00
Niels de Vos
682840476f build: ignore generated go-tags file
The `scripts/golangci.yml.buildtags.in` file is generated from the
`Makefile`, there is no need to include it in the repository. By adding
the file to the `.gitignore` list, the output of `git status` will not
show the file anymore.

Fixes: 8fb5739f2
    "build: more flexible handling of go build tags; added ceph_preview"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-04-04 12:59:12 +00:00
Prasanna Kumar Kalever
d760d0ab6d rbd: check for cookie support from kernel
Currently we only check if the rbd-nbd tool supports cookie feature.
This change will also defend cookie addition based on kernel version

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-04 09:51:13 +00:00
Niels de Vos
804e2715d8 deploy: add deployment artifacts for NFS support
These deployment files are heavily based on the CephFS deployment.

Deploying an environment with these files work for me in minikube. This
should make it possible to add e2e testing as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-04-01 10:37:41 +00:00
Niels de Vos
591cd694ab doc: mark NFS support Alpha state
There is currently no e2e testing, unit-tests or Helm Chart for NFS
support. Until the functionality is confirmed to be working on a regular
basis, support for NFS provisioner volume will be Alpha.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-31 14:16:44 +00:00
Madhu Rajanna
f8bbd2f60f cephfs: fix omap deletion in DeleteSnapshot
The omap is stored with the requested
snapshot name not with the subvolume
snapshotname. This fix uses the correct
snapshot request name to cleanup the omap
once the subvolume snapshot is deleted.

fixes: #2974

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-31 13:46:03 +00:00
Niels de Vos
1da19680b4 nfs: support new and old NFS-management commands
The `ceph nfs export ...` commands have changed in recent Ceph releases.
Use the most recent command as a default, fall back to the older command
when an error is reported.

This shoud make the NFS-provisioner work on any current Ceph version.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-31 11:28:40 +00:00
Madhu Rajanna
f90408be4d rbd: increase force promote timeout to 2 minutes
Increase the timeout to 2 minutes to give enough time
for rollback to complete.
As rollback is performed by the force-promote command it,
at times, may take more than a minute
(based on dirty blocks that need to be rolled
back approximately) to rollback.

The added extra 1 minute is useful though to avoid
multiple calls to complete the rollback and in
extremely corner cases to avoid failures in the
first instance of the call when the mirror watcher
is not yet removed (post scaling down the
RBD mirror instance)

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-30 13:46:27 +00:00
dependabot[bot]
e45c70b84f rebase: bump google.golang.org/protobuf from 1.27.1 to 1.28.0
Bumps [google.golang.org/protobuf](https://github.com/protocolbuffers/protobuf-go) from 1.27.1 to 1.28.0.
- [Release notes](https://github.com/protocolbuffers/protobuf-go/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf-go/blob/master/release.bash)
- [Commits](https://github.com/protocolbuffers/protobuf-go/compare/v1.27.1...v1.28.0)

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-30 11:17:14 +00:00
Thibaut Blanchard
e874c9c11b rbd: fix topology snapshot pool
Restoring a snapshot with a new PVC results with a wrong
dataPoolName in case of initial volume linked
to a storageClass with topology constraints and erasure coding.

Signed-off-by: Thibaut Blanchard <thibaut.blanchard@gmail.com>
2022-03-30 04:40:30 +00:00
dependabot[bot]
134603540b rebase: bump github.com/onsi/gomega from 1.18.1 to 1.19.0
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.18.1 to 1.19.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.18.1...v1.19.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-29 12:07:17 +00:00
dependabot[bot]
4652b8facf rebase: bump github.com/hashicorp/vault/api from 1.4.1 to 1.5.0
Bumps [github.com/hashicorp/vault/api](https://github.com/hashicorp/vault) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/hashicorp/vault/releases)
- [Changelog](https://github.com/hashicorp/vault/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/vault/compare/v1.4.1...v1.5.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/vault/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-29 04:06:00 +00:00
Niels de Vos
190504713a doc: initial/partial instructions for using NFS examples
The README explains some of the requirements and basic configuration for
using the NFS-provisioner. When more deployment artifacts are added, the
README will get extended.

The Rook CephNFS example is included, as it is the easiest to get
started with dynamic provisioning of NFS-volumes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:58:42 +00:00
Niels de Vos
b72774f9e9 doc: example for PVC and Pod using a NFS-volume
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:58:42 +00:00
Niels de Vos
2743510009 doc: example of StorageClass for NFS-provisioning
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:58:42 +00:00
Niels de Vos
885295fcc9 nfs: store the NFS-cluster name in the journal
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
3b4d193ca8 journal: add StoreAttribute/FetchAttribute
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
010fd816dd nfs: store the calling Context in NFSVolume
NFSVolume instances are short lived, they only extist for a certain gRPC
procedure. It is easier to store the calling Context in the NFSVolume
struct, than to pass it to some of the functions that require it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
3d0c4e0659 nfs: enable NFS-provisioner with --type=nfs
Deployments can use --type=nfs to deploy the NFS Controller Server
(provisioner).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
6d83df9cc9 nfs: add basic provisioner with create/delete procedures
These NFS Controller and Identity servers are the base for the new
provisioner. The functionality is currently extremely limited, follow-up
PRs will implement various CSI procedures.

CreateVolume is implemented with the bare minimum. This makes it
possible to create a volume, and mount it with the
kubernetes-csi/csi-driver-nfs NodePlugin.

DeleteVolume unexports the volume from the Ceph managed NFS-Ganesha
service. In case the Ceph cluster provides multiple NFS-Ganesha
deployments, things might not work as expected. This is going to be
addressed in follow-up improvements.

Lots of TODO comments need to be resolved before this can be declared
"production ready". Unit- and e2e-tests are missing as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
87f87141be deploy: add CSIDriver for NFS
The API is extended for generation of the NFS CSIDriver object. The
YAML file under deploy/ was created by `yamlgen`.

The contents of the csidriver.yaml file is heavily based on the upstream
CSIDriver from the Kubernetes csi-driver-nfs project.

Because ./tools/yamlgen uses the API, it gets copied under vendor/ .
This causes two copies of the API to be included in the repository, but
that can not be prevented, it seems.

See-also: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/deploy/csi-nfs-driverinfo.yaml
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
984119b30d ci: add "nfs" as allowed commit prefix
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
4dc1d36218 cleanup: reduce complexity of main()
Move the printing of the version and other information to its own
function. This reduces the complexity enough so that golang-ci does not
complain about it anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Humble Chirammal
16abbbc846 build: remove cache while building container image
Reduce size of the container image by removing the cache in deploy
and devel container.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-03-28 06:09:27 +00:00
Robert Vasek
f6ae612003 util: added reference tracker
RT, reference tracker, is key-based implementation of a reference counter.
Unlike an integer-based counter, RT counts references by tracking unique
keys. This allows accounting in situations where idempotency must be
preserved. It guarantees there will be no duplicit increments or decrements
of the counter.

Signed-off-by: Robert Vasek <robert.vasek@cern.ch>
2022-03-27 19:24:26 +00:00
Robert Vasek
8fb5739f21 build: more flexible handling of go build tags; added ceph_preview
ceph_preview tag is needed to make new go-ceph's RADOS read/write ops available

Signed-off-by: Robert Vasek <robert.vasek@cern.ch>
2022-03-27 19:24:26 +00:00