Commit Graph

3721 Commits

Author SHA1 Message Date
Rakshith R
8f75a24cfd ci: use 0 as default NUM_DISKS in minikube.sh
This is done to prevent conflicts with current ci setup externally
attaching disks.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-09 05:18:36 +00:00
Rakshith R
42a6c3c006 ci: pass $DISK_CONFIG properly to minikube start
Having double quotes around $DISK_CONFIG led to these args
not being properly passed to minikube start. This commit fixes it.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-09 05:18:36 +00:00
Rakshith R
0a7a7f4866 util: call WriteCephConfig() in cephcsi.go
This commit calls WriteCephConfig() in cephcsi.go to
create ceph.conf and keyring if it is not mounted to
be used by all cli calls and conn cmds.

Before this change, rbd-controller/omap-generator did not create
ceph.conf on startup.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-08 16:05:27 +00:00
Madhu Rajanna
8c8f34cf7a rbd: set vaultAuthNamespace to vaultNamespace if empty
When we read the csi-kms-connection-details configmap
vaultAuthNamespace might not be set when we do the
conversion the vaultAuthNamespace might be set to empty
key and this commits check for the empty value of
vaultAuthNamespace and set the vaultAuthNamespace
to vaultNamespace.

setting empty value for vaultAuthNamespace happened due
to Marshalling at https://github.com/ceph/ceph-csi/blob/devel/
internal/kms/vault_tokens.go#L136-L139.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-08 11:18:03 +00:00
Rakshith R
e99dd3dea4 util: read ceph.conf by calling conn.ReadConfigFile(CephConfigPath)
The configurations in cpeh.conf is not picked up by rados connection
automatically, hence we need to call conn.ReadConfigFile before calling
Connect().

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-07 16:50:12 +00:00
Madhu Rajanna
76f1b42498 cephfs: correct comment for validateExpandVolumeRequest
corrected the function comment for
validateExpandVolumeRequest.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Madhu Rajanna
9fd51d9bec cephfs: add comment for validateCreateVolumeRequest
added function comment for
validateCreateVolumeRequest

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Madhu Rajanna
8caeb409bb cephfs: add comment for validateDeleteVolumeRequest
added function comment for the
validateDeleteVolumeRequest function.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Madhu Rajanna
be7749c90e cleanup: move volumeID to the volumeoptions
volumeID can be moved to the volumeOptions
as most of the volume related helper functions
are available on the volumeoptions.go

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Madhu Rajanna
da70ed50dc cleanup: move execCommandErr to volumemounter
Moved execCommandErr to the volumemounter.go
which is the only caller of this function and
moving the execCommandErr helps in reducing the
util file.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Madhu Rajanna
31696a6ce0 cleanup: move genSnapFromOptions to volumeoptions
moved genSnapFromOptions function to volumeoptions.go
which is more appropriated than util.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Madhu Rajanna
73e2ffe8b8 cleanup: move cephfs csi spec validation to validator
moved the cephfs related validation like
validating the input parameters sent in the
GRPC request to a new file.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-07 14:33:02 +00:00
Humble Chirammal
4efcc5bf97 cleanup: simplify checkStaticVolume function and remove unwanted vars
checkStaticVolume() in the reconcilePV function has been unwantedly
introducing variables to confirm the pv spec is static or not. This
patch simplify it and make a smaller footprint of the functions.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-07 12:51:30 +00:00
Rakshith R
1b64a0a505 ci: add support to create extra disks through minikube
Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-07 07:10:59 +00:00
Rakshith R
08c10c9f94 rebase: update minikube to v1.23.0
See-also: https://github.com/kubernetes/minikube/releases/tag/v1.23.0

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-07 07:10:59 +00:00
Humble Chirammal
dda811c195 doc: add 'known to work' CO matrix with versions tested
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-07 04:49:24 +00:00
Humble Chirammal
df2d9548ae cephfs: no need to check for zero volume size
At present there is a 'todo' to check for zero volume size
in the createVolume request which in unwanted, ie the pvc
creation with size 0 fail from the kubernetes api validation itself:

For ex:

```
..spec.resources[storage]: Invalid value: "0": must be greater than zero```
```
so we dont need any extra check in the controller server

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-07 04:49:24 +00:00
Prasanna Kumar Kalever
9e55f015de rbd: avoid supplying map options on unmap
Thanks to the random unmap failure on my local machine:

I0901 17:08:37.841890 2617035 cephcmds.go:55] ID: 11 Req-ID:
0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-024983f3-0b47-11ec-8fcb-e671f0b9f58e
an error (exit status 22) occurred while running rbd args: [unmap
rbd-pool/csi-vol-024983f3-0b47-11ec-8fcb-e671f0b9f58e --device-type nbd
--options try-netlink --options reattach-timeout=300 --options
io-timeout=0]

Noticed the map args are also getting passed to/as unmap args, which is not
correct. We have separate things for mapOptions and unmapOptions. This PR
makes sure that the map args are not passed at the time of unmap.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-06 15:59:30 +00:00
Humble Chirammal
3f31ca8a3a cleanup: introduce populateVolOptions(), to fill rbdVol from stage req
At present the nodeStageVolume() handle many logic of filling rbdvol
struct based on the request received and this method is complex to
follow. with this patch, filling or populating volOptions has been
segregrated and handled hence make the stage functions' job easy.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-06 07:49:03 +00:00
Humble Chirammal
f0b8a3f626 rbd: use String() method of MirrorImageState in return error
MirrorImageState (type C.rbd_mirror_image_state_t) has a string
method which can be used while returning error in the replication
controller. Previously, we were using int return in the error which
is not the proper usage.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-03 16:02:53 +00:00
Madhu Rajanna
5a53a9ad76 e2e: update e2e deployment for ceph conf
updated e2e deployment code to create and
delete the ceph conf configmap.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Madhu Rajanna
cfc070b169 deploy: remove ceph-conf cm in helm installation
removed the ceph-conf configmap as a workaround
to avoid helm installation issue.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Madhu Rajanna
d22e7a1bdb helm: update templates for ceph.conf
updated cephfs and rbd templates to
configure the ceph.conf content.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Madhu Rajanna
4865061ab9 util: create ceph configuration files if not present
create ceph.conf and keyring files if its not
present in the /et/ceph/ path.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Madhu Rajanna
cc6c51395e deploy: update templates for ceph.conf
updated cephfs and rbd templates to mount
the ceph.conf configmap.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Madhu Rajanna
43925477a5 doc: update deployment doc for ceph.conf
updated cephfs and rbd deployment doc
to deploy the configmap that contains the
ceph.conf

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Madhu Rajanna
4dba4e3f6f deploy: add configmap for ceph.conf
added an example configmap for the
ceph.conf file. This allows the users to
specify different ceph configuration when
deploying cephcsi.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-03 14:14:43 +00:00
Humble Chirammal
1d94c12cd6 cleanup: add checkErrAndUndoReserve() for error check,unreserve omap
all the error check scenarios of genVolFromVolID() and unreserving
omap entries based on the error made deleteVolume method complex,
this patch create a new function which handle the error check and
unrerving omap entries accordingly and finally return the response
to deletevolume/caller.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-03 12:20:04 +00:00
dependabot[bot]
8fc7db8d80 rebase: bump github.com/aws/aws-sdk-go from 1.38.63 to 1.40.34
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.38.63 to 1.40.34.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.38.63...v1.40.34)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-02 16:57:22 +00:00
dependabot[bot]
fb000647b5 rebase: bump sigs.k8s.io/controller-runtime from 0.9.2 to 0.10.0
Bumps [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) from 0.9.2 to 0.10.0.
- [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases)
- [Commits](https://github.com/kubernetes-sigs/controller-runtime/compare/v0.9.2...v0.10.0)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/controller-runtime
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-02 13:49:51 +00:00
Niels de Vos
60c2afbcca util: NewK8sClient() should not panic on non-Kubernetes clusters
When NewK8sClient() detects and error, it used to call FatalLogMsg()
which causes a panic. There are additional features that can be used on
Kubernetes clusters, but these are not a requirement for most
functionalities of the driver.

Instead of causing a panic, returning an error should suffice. This
allows using the driver on non-Kubernetes clusters again.

Fixes: #2452
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-02 11:22:14 +00:00
dependabot[bot]
e8efa272a6 rebase: bump github.com/onsi/gomega from 1.13.0 to 1.16.0
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.13.0 to 1.16.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.13.0...v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-02 09:32:38 +00:00
Niels de Vos
96ba1d4799 ci: prevent mergify conflict message for dependabot PRs
There is no use in having Mergify posting messages to Dependabot and
asking for resolving of merge conflicts. Dependabot will try to do that
automatically, and posts a message when some other action is needed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-02 12:32:30 +05:30
Niels de Vos
c12b155940 rebase: use latest google.golang.org/grpc version
google.golang.org/grpc was listed in go.mod:replace and updating the
dependency was not possible. There should not be a reason to prevent
updating, commit 5aca04d540 added it but without explanation.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-01 15:19:26 +00:00
dependabot[bot]
bf77ab57dc rebase: bump google.golang.org/grpc from 1.38.0 to 1.40.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.38.0 to 1.40.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.38.0...v1.40.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-01 15:19:26 +00:00
Niels de Vos
28899a36c8 ci: do not run commitlint GitHub Action for dependabot
The commit messages that Dependabot (@app/dependabot) creates are not
always accepted by the commitlint check. The configuration for
dependabot does not give a lot of options to customize the message, so
instead of adjusting the message to pass commitlint, just skip the check
(Mergify does not require commitlint status either).

See-also: #2460
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-01 15:02:18 +02:00
Niels de Vos
3f6db2f851 rebase: update k8s.io/kubernetes dependencies to v0.22.1
Dependabot can not update the dependencies for k8s.io/kubernetes
correctly. Helping the bot out with this additional commit.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-01 13:01:01 +00:00
dependabot[bot]
b9b5ae5387 rebase: bump k8s.io/kubernetes from 1.22.0 to 1.22.1
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.22.0 to 1.22.1.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.22.0...v1.22.1)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-01 13:01:01 +00:00
Humble Chirammal
247795517f cephfs: remove explicit size setting of cloned volume
CephFS csi driver explictly set the size of the cloned volume
to the size of parent volume as cephfs mgr was lacking this
functionality previously. However it has been addressed in cephfs
so we dont need explicit size setting.

Ref#https://tracker.ceph.com/issues/46163

Supported Ceph releases:

Ceph versions equal or above - v16.0.0, v15.2.9, v14.2.12

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-01 09:32:29 +00:00
Madhu Rajanna
b383af20b4 cleanup: move cephfs errors to new util package
As part of the refactoring, moving the cephfs errors file to a new
package.

Updates: #852
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-01 06:50:16 +00:00
Humble Chirammal
aeebd5d03b doc: remove upgrade instructions for earlier releases
As we have deprecated earlier versions than v3.3.0, it is not required
to keep the upgrade docs for the same. The upgrade doc for v3.2.0 to
v3.3.0 has been kept intact.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-01 03:37:23 +00:00
dependabot[bot]
ed38157f9f rebase: bump k8s.io/klog/v2 from 2.9.0 to 2.10.0
Bumps [k8s.io/klog/v2](https://github.com/kubernetes/klog) from 2.9.0 to 2.10.0.
- [Release notes](https://github.com/kubernetes/klog/releases)
- [Changelog](https://github.com/kubernetes/klog/blob/main/RELEASE.md)
- [Commits](https://github.com/kubernetes/klog/compare/v2.9.0...v2.10.0)

---
updated-dependencies:
- dependency-name: k8s.io/klog/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-31 19:31:20 +00:00
Rakshith R
99168dc822 rbd: check for clusterid mapping in RegenerateJournal()
This commit adds fetchMappedClusterIDAndMons() which returns
monitors and clusterID info after checking cluster mapping info.
This is required for regenerating omap entries in mirrored cluster
with different clusterID.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-31 14:30:06 +00:00
Rakshith R
496bcba85c rbd: move GetMappedID() to util package
This commit moves getMappedID() from rbd to util
package since it is not rbd specific and exports
it from there.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-31 14:30:06 +00:00
Niels de Vos
e08d184984 ci: ignore k8s.io/kubernetes dependencies
These dependencies are pulled in by k8s.io/kubernetes with version
v0.0.0. It is therefore required to use 'replace' in go.mod to select a
compatible version of the additional k8s.io packages.

Dependabot does not seem to update packages listed in 'replace', only
under 'require'. That means, the version updates done by Dependabot do
not have any effect, as the contents is replaced with a different
version anyway. Ignoring these packages prevents the creation of
non-functional PRs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-31 09:03:12 +00:00
Niels de Vos
1b30a58e53 ci: skip commitlint for dependabot PRs
Dependabot creates commit messages that commitlint fails to parse
correctly. The messages contain simple references to which dependencies
are updated, and do not need full validation like manual written
messages do.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-31 09:02:11 +02:00
Humble Chirammal
92e7f16654 ci: remove the rules for backport on unsupported versions
CSI version v3.3.0 and above are the supported versions now and this
patch also adjust the mergify rules according to that, ie the backport
rules are removed for all unsupported versions here.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-31 10:56:28 +05:30
Niels de Vos
4a3b1181ce cleanup: move KMS functionality into its own package
A new "internal/kms" package is introduced, it holds the API that can be
consumed by the RBD components.

The KMS providers are currently in the same package as the API. With
later follow-up changes the providers will be placed in their own
sub-package.

Because of the name of the package "kms", the types, functions and
structs inside the package should not be prefixed with KMS anymore:

    internal/kms/kms.go:213:6: type name will be used as kms.KMSInitializerArgs by other packages, and that stutters; consider calling this InitializerArgs (golint)

Updates: #852
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-30 16:31:40 +00:00
Niels de Vos
778b5e86de cleanup: move k8s functions to the util/k8s package
By placing the NewK8sClient() function in its own package, the KMS API
can be split from the "internal/util" package. Some of the KMS providers
use the NewK8sClient() function, and this causes circular dependencies
between "internal/utils" -> "internal/kms" -> "internal/utils", which
are not alowed in Go.

Updates: #852
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-30 16:31:40 +00:00
Niels de Vos
2cc96dc539 build: vendor code.cloudfoundry.org/gofileutils from GitHub
There is a problem accessing the code.cloudfoundry.org web service iver
TLS. It seems to redirect to GitHub, so use the package from there:

    running: go mod verify
    go: github.com/libopenstorage/secrets@v0.0.0-20210709082113-dde442ea20ec requires
    	github.com/hashicorp/vault@v1.4.2 requires
    	github.com/hashicorp/vault-plugin-auth-cf@v0.5.4 requires
    	github.com/cloudfoundry-community/go-cfclient@v0.0.0-20190201205600-f136f9222381 requires
    	code.cloudfoundry.org/gofileutils@v0.0.0-20170111115228-4d0c80011a0f: unrecognized import path "code.cloudfoundry.org/gofileutils": https fetch: Get "https://code.cloudfoundry.org/gofileutils?go-get=1": x509: certificate signed by unknown authority

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-30 13:51:49 +00:00