Commit Graph

3138 Commits

Author SHA1 Message Date
Niels de Vos
682840476f build: ignore generated go-tags file
The `scripts/golangci.yml.buildtags.in` file is generated from the
`Makefile`, there is no need to include it in the repository. By adding
the file to the `.gitignore` list, the output of `git status` will not
show the file anymore.

Fixes: 8fb5739f2
    "build: more flexible handling of go build tags; added ceph_preview"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-04-04 12:59:12 +00:00
Prasanna Kumar Kalever
d760d0ab6d rbd: check for cookie support from kernel
Currently we only check if the rbd-nbd tool supports cookie feature.
This change will also defend cookie addition based on kernel version

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2022-04-04 09:51:13 +00:00
Niels de Vos
804e2715d8 deploy: add deployment artifacts for NFS support
These deployment files are heavily based on the CephFS deployment.

Deploying an environment with these files work for me in minikube. This
should make it possible to add e2e testing as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-04-01 10:37:41 +00:00
Niels de Vos
591cd694ab doc: mark NFS support Alpha state
There is currently no e2e testing, unit-tests or Helm Chart for NFS
support. Until the functionality is confirmed to be working on a regular
basis, support for NFS provisioner volume will be Alpha.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-31 14:16:44 +00:00
Madhu Rajanna
f8bbd2f60f cephfs: fix omap deletion in DeleteSnapshot
The omap is stored with the requested
snapshot name not with the subvolume
snapshotname. This fix uses the correct
snapshot request name to cleanup the omap
once the subvolume snapshot is deleted.

fixes: #2974

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-31 13:46:03 +00:00
Niels de Vos
1da19680b4 nfs: support new and old NFS-management commands
The `ceph nfs export ...` commands have changed in recent Ceph releases.
Use the most recent command as a default, fall back to the older command
when an error is reported.

This shoud make the NFS-provisioner work on any current Ceph version.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-31 11:28:40 +00:00
Madhu Rajanna
f90408be4d rbd: increase force promote timeout to 2 minutes
Increase the timeout to 2 minutes to give enough time
for rollback to complete.
As rollback is performed by the force-promote command it,
at times, may take more than a minute
(based on dirty blocks that need to be rolled
back approximately) to rollback.

The added extra 1 minute is useful though to avoid
multiple calls to complete the rollback and in
extremely corner cases to avoid failures in the
first instance of the call when the mirror watcher
is not yet removed (post scaling down the
RBD mirror instance)

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-30 13:46:27 +00:00
dependabot[bot]
e45c70b84f rebase: bump google.golang.org/protobuf from 1.27.1 to 1.28.0
Bumps [google.golang.org/protobuf](https://github.com/protocolbuffers/protobuf-go) from 1.27.1 to 1.28.0.
- [Release notes](https://github.com/protocolbuffers/protobuf-go/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf-go/blob/master/release.bash)
- [Commits](https://github.com/protocolbuffers/protobuf-go/compare/v1.27.1...v1.28.0)

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-30 11:17:14 +00:00
Thibaut Blanchard
e874c9c11b rbd: fix topology snapshot pool
Restoring a snapshot with a new PVC results with a wrong
dataPoolName in case of initial volume linked
to a storageClass with topology constraints and erasure coding.

Signed-off-by: Thibaut Blanchard <thibaut.blanchard@gmail.com>
2022-03-30 04:40:30 +00:00
dependabot[bot]
134603540b rebase: bump github.com/onsi/gomega from 1.18.1 to 1.19.0
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.18.1 to 1.19.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.18.1...v1.19.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-29 12:07:17 +00:00
dependabot[bot]
4652b8facf rebase: bump github.com/hashicorp/vault/api from 1.4.1 to 1.5.0
Bumps [github.com/hashicorp/vault/api](https://github.com/hashicorp/vault) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/hashicorp/vault/releases)
- [Changelog](https://github.com/hashicorp/vault/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/vault/compare/v1.4.1...v1.5.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/vault/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-29 04:06:00 +00:00
Niels de Vos
190504713a doc: initial/partial instructions for using NFS examples
The README explains some of the requirements and basic configuration for
using the NFS-provisioner. When more deployment artifacts are added, the
README will get extended.

The Rook CephNFS example is included, as it is the easiest to get
started with dynamic provisioning of NFS-volumes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:58:42 +00:00
Niels de Vos
b72774f9e9 doc: example for PVC and Pod using a NFS-volume
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:58:42 +00:00
Niels de Vos
2743510009 doc: example of StorageClass for NFS-provisioning
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:58:42 +00:00
Niels de Vos
885295fcc9 nfs: store the NFS-cluster name in the journal
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
3b4d193ca8 journal: add StoreAttribute/FetchAttribute
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
010fd816dd nfs: store the calling Context in NFSVolume
NFSVolume instances are short lived, they only extist for a certain gRPC
procedure. It is easier to store the calling Context in the NFSVolume
struct, than to pass it to some of the functions that require it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
3d0c4e0659 nfs: enable NFS-provisioner with --type=nfs
Deployments can use --type=nfs to deploy the NFS Controller Server
(provisioner).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
6d83df9cc9 nfs: add basic provisioner with create/delete procedures
These NFS Controller and Identity servers are the base for the new
provisioner. The functionality is currently extremely limited, follow-up
PRs will implement various CSI procedures.

CreateVolume is implemented with the bare minimum. This makes it
possible to create a volume, and mount it with the
kubernetes-csi/csi-driver-nfs NodePlugin.

DeleteVolume unexports the volume from the Ceph managed NFS-Ganesha
service. In case the Ceph cluster provides multiple NFS-Ganesha
deployments, things might not work as expected. This is going to be
addressed in follow-up improvements.

Lots of TODO comments need to be resolved before this can be declared
"production ready". Unit- and e2e-tests are missing as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
87f87141be deploy: add CSIDriver for NFS
The API is extended for generation of the NFS CSIDriver object. The
YAML file under deploy/ was created by `yamlgen`.

The contents of the csidriver.yaml file is heavily based on the upstream
CSIDriver from the Kubernetes csi-driver-nfs project.

Because ./tools/yamlgen uses the API, it gets copied under vendor/ .
This causes two copies of the API to be included in the repository, but
that can not be prevented, it seems.

See-also: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/deploy/csi-nfs-driverinfo.yaml
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
984119b30d ci: add "nfs" as allowed commit prefix
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Niels de Vos
4dc1d36218 cleanup: reduce complexity of main()
Move the printing of the version and other information to its own
function. This reduces the complexity enough so that golang-ci does not
complain about it anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-28 11:23:17 +00:00
Humble Chirammal
16abbbc846 build: remove cache while building container image
Reduce size of the container image by removing the cache in deploy
and devel container.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-03-28 06:09:27 +00:00
Robert Vasek
f6ae612003 util: added reference tracker
RT, reference tracker, is key-based implementation of a reference counter.
Unlike an integer-based counter, RT counts references by tracking unique
keys. This allows accounting in situations where idempotency must be
preserved. It guarantees there will be no duplicit increments or decrements
of the counter.

Signed-off-by: Robert Vasek <robert.vasek@cern.ch>
2022-03-27 19:24:26 +00:00
Robert Vasek
8fb5739f21 build: more flexible handling of go build tags; added ceph_preview
ceph_preview tag is needed to make new go-ceph's RADOS read/write ops available

Signed-off-by: Robert Vasek <robert.vasek@cern.ch>
2022-03-27 19:24:26 +00:00
Humble Chirammal
afeae03069 ci: fetch sidecar versions from build.env and use it
The sidecar images in minikube deployment will be fetched from
build.env and used/validated accordingly.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-03-24 14:09:13 +00:00
dependabot[bot]
bb8bed8ac7 rebase: bump github.com/aws/aws-sdk-go-v2/service/sts
Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.15.0...service/s3/v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-24 10:32:12 +00:00
dependabot[bot]
4ebfe5ded2 rebase: bump k8s.io/klog/v2 from 2.40.1 to 2.60.1
Bumps [k8s.io/klog/v2](https://github.com/kubernetes/klog) from 2.40.1 to 2.60.1.
- [Release notes](https://github.com/kubernetes/klog/releases)
- [Changelog](https://github.com/kubernetes/klog/blob/main/RELEASE.md)
- [Commits](https://github.com/kubernetes/klog/compare/v2.40.1...v2.60.1)

---
updated-dependencies:
- dependency-name: k8s.io/klog/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-23 16:29:15 +00:00
Rakshith R
40de75e0db rbd: modify oidc token file path according to FHS 3.0
OIDC token file path has been modified from
`/var/run/secrets/token` to `/run/secrets/tokens`.
This has been done to ensure compliance with
FHS 3.0.

refer:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s13.html

Signed-off-by: Rakshith R <rar@redhat.com>
2022-03-23 13:29:35 +00:00
dependabot[bot]
0ab8e6a2f5 rebase: bump github.com/stretchr/testify from 1.7.0 to 1.7.1
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.7.0...v1.7.1)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-23 11:34:47 +00:00
dependabot[bot]
6f1c276dc8 rebase: bump github.com/aws/aws-sdk-go from 1.43.18 to 1.43.22
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.43.18 to 1.43.22.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.43.18...v1.43.22)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-22 08:48:15 +00:00
dependabot[bot]
fe4b5fe337 rebase: bump k8s.io/kubernetes from 1.23.4 to 1.23.5
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.23.4 to 1.23.5.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.23.4...v1.23.5)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-22 06:03:02 +00:00
Madhu Rajanna
8c5e414d53 rbd: do not read pvc namespace from volume attributes
Below are the 3 different cases where we need
the PVC namespace for encryption

* CreateVolume:- Read the namespace from the
createVolume parameters and store it in the omap
* NodeStage:- Read the namespace from the omap
not from the volumeContext
* Regenerate:- Read the pvc namespace from the claimRef
not from the volumeAttributes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-21 08:54:43 +00:00
Madhu Rajanna
77011fbc61 cephfs: remove kubernetes csi prefixed parameters
remove kubernetes csi prefixed parameters
from the volumeContext as we dont want
to store it in the PV VolumeAttributes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-21 08:54:43 +00:00
Madhu Rajanna
a7315a04c1 rbd: remove kubernetes csi prefixed parameters
remove kubernetes csi prefixed parameters
from the volumeContext as we dont want
to store it in the PV VolumeAttributes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-21 08:54:43 +00:00
Madhu Rajanna
366c2ace31 util: add helper to get pvcnamespace from input
added helper function to return the pvc namespace
name from the input parameters.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-21 08:54:43 +00:00
Madhu Rajanna
772fe8d6c8 util: add helper function to strip kube parameters
added helper function to strip the kubernetes
specific parameters from the volumeContext as
volumeContext is storaged in the PV volumeAttributes

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-21 08:54:43 +00:00
Niels de Vos
f57b7f88b0 ci: update minikube to latest v1.25.2 release
This version addresses several bugs and includes other improvements.

See-also: https://github.com/kubernetes/minikube/releases/tag/v1.25.2
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-18 12:52:22 +00:00
Rakshith R
a56f9a0c05 rbd: flatten datasource image before creating volume
This commit ensures that parent image is flattened before
creating volume.
- If the data source is a PVC, the underlying image's parent
  is flattened(which would be a temp clone or snapshot).
  hard & soft limit is reduced by 2 to account for depth that
  will be added by temp & final clone.

- If the data source is a Snapshot, the underlying image is
  itself flattened.
  hard & soft limit is reduced by 1 to account for depth that
  will be added by the clone which will be restored from the
  snapshot.

Flattening step for resulting PVC image restored from snapshot is removed.
Flattening step for temp clone & final image is removed when pvc clone is
being created.

Fixes: #2190

Signed-off-by: Rakshith R <rar@redhat.com>
2022-03-18 10:27:27 +00:00
Rakshith R
6dd5fe9360 ci: use CEPH_CSI_BOT token for retest action
For retest action's comment `@Mergifyio refresh`
to be accepted by mergifyio,
the bot should have write permissions to the repo.
Therefore, use Ceph-csi-bot instead of github actions
bot.

Signed-off-by: Rakshith R <rar@redhat.com>
2022-03-18 05:27:49 +00:00
Niels de Vos
9fe3870874 ci: add "component/nfs" label with Mergify
"component/nfs" has been added to the labels in the repository, and
Mergify should be able to add that to PRs.

Updates: #2913
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-17 14:10:57 +01:00
Rakshith R
4e8033b762 ci: modify retest action to comment @Mergifyio refresh
When a test fails, mergifyio kicks that pr out of queue.
Retest action will now comment `@Mergifyio refresh` to
add the pr back into queue, if any test has been restarted.
This makes sure mergifyio rebases the pr automatically
according to the queue.

Signed-off-by: Rakshith R <rar@redhat.com>
2022-03-17 07:52:26 +00:00
Madhu Rajanna
d357bebbc2 cephfs: disallow creating small volumes from snapshot/volume
as per the CSI standard the size is optional parameter,
as we are allowing the clone to a bigger size
today we need to block the clone to a smaller size
as its a have side effects like data corruption etc.

Note:- Even though this check is present in kubernetes
sidecar as CSI is CO independent adding the check
here.

fixes: #2718

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2022-03-17 05:07:26 +00:00
Yati Padia
732ab13096 cleanup: adds short description to disable goerr113 linter
This commit adds a short description with reason to
disable the goerr113 linter.

issue: #1227

Signed-off-by: Yati Padia <ypadia@redhat.com>
2022-03-16 16:18:52 +00:00
Humble Chirammal
525ff5d97f rbd: remove unimplemented responses for node operations
These RPCs( nodestage,unstage,volumestats) are
implemented RPCs for our drivers atm. This commit removes
the `unimplemented` responses from the common/default
server initialization routins.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-03-16 15:27:48 +00:00
Humble Chirammal
66e7f3525f cleanup: remove unimplemented controller expand,snapshot RPCs
These RPCs ( controller expand, create and delete snapshots) are
no longer unimplmented and we dont have to declare these as with
`unimplemented` states. This commit remove the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-03-16 15:27:48 +00:00
Niels de Vos
1b16db4ff6 doc: proposal for providing NFS-exports
The design for NFS-provising of volumes explains how the existing NFS
CSI Node-Plugin can be used, and what the steps are to create the
NFS-export with Ceph Mgr.

Updates: #2913
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-03-16 11:36:18 +00:00
Rakshith R
23145bc1f7 cleanup: fix csi-kms-connection-details.yaml ibmkeyprotect-test ex
The last option in `ibmkeyprotect-test` example had a extra `,`
at the end, which caused parsing to fail. This commit removes
the same.

Signed-off-by: Rakshith R <rar@redhat.com>
2022-03-16 07:29:56 +00:00
Rakshith R
4f0bb2315b rbd: add aws-sts-metdata encryption type
With Amazon STS and kubernetes cluster is configured with
OIDC identity provider, credentials to access Amazon KMS
can be fetched using oidc-token(serviceaccount token).
Each tenant/namespace needs to create a secret with aws region,
role and CMK ARN.
Ceph-CSI will assume the given role with oidc token and access
aws KMS, with given CMK to encrypt/decrypt DEK which will stored
in the image metdata.

Refer: https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html
Resolves: #2879

Signed-off-by: Rakshith R <rar@redhat.com>
2022-03-16 07:29:56 +00:00
dependabot[bot]
13dcc89ac8 rebase: bump google.golang.org/grpc from 1.44.0 to 1.45.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.44.0 to 1.45.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.44.0...v1.45.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-15 12:19:19 +00:00