Commit Graph

4231 Commits

Author SHA1 Message Date
Madhu Rajanna
8ebc0659ab rbd: perform resize of file system for static volume
For static volume, the user will manually mounts
already existing image as a volume to the application
pods. As its a rbd Image, if the PVC is of type
fileSystem the image will be mapped, formatted
and mounted on the node,
If the user resizes the image on the ceph cluster.
User cannot not automatically resize the filesystem
created on the rbd image. Even if deletes and
recreates the kubernetes objects, the new size
will not be visible on the node.

With this changes During the NodeStageVolumeRequest
the nodeplugin will check the size of the mapped rbd
image on the node using the devicePath. and also
the rbd image size on the ceph cluster.

If the size is not matching it will do the file
system resize on the node as part of the
NodeStageVolumeRequest RPC call.

The user need to do below operation to see new size
* Resize the rbd image in ceph cluster
* Scale down all the application pods using the static
PVC.
* Make sure no application pods which are using the
static PVC is running on a node.
* Scale up all the application pods.

Validate the new size in application pod mounted
volume.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-10-06 13:15:00 +00:00
Madhu Rajanna
fe9020260d rbd: move flattening to helper function
in NodeStage operation we are flattening
the image to support mounting on the older
clients. this commits moves it to a helper
function to reduce code complexity.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-10-06 13:15:00 +00:00
Madhu Rajanna
cda2abca5d rbd: use NewMetricsBlock to get size
instead of lsblk command use NewMetricsBlock
function from the kubernetes package to get
the size.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-10-06 13:15:00 +00:00
Niels de Vos
97525f5e74 ci: add make go-test-api to GitHub Action
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-10-05 11:26:50 +00:00
Niels de Vos
5ea99fdd5b build: add yamlgen to build deployment files
This initial version of yamlgen generates deploy/scc.yaml based on the
deployment artifact that is provided by the new api/deploy/ocp package.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-10-05 11:26:50 +00:00
Niels de Vos
36e099d939 deploy: add API for getting OpenShift SecurityContextConstraints
This new Go module allows other projects (like Rook) to consume
deployment details (the SCC to get started) directly from Ceph-CSI.

Based-on: https://github.com/rook/rook/blob/3c93eb3/cluster/examples/kubernetes/ceph/operator-openshift.yaml#L47-L90
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-10-05 11:26:50 +00:00
Rakshith R
f60b097f5f e2e: add testcase for thick encrypted PVC restore
Signed-off-by: Rakshith R <rar@redhat.com>
2021-10-05 07:46:57 +00:00
Rakshith R
ded75eb099 rbd: copyEncryptionConfig for thickProvisioned snap restore too
This commit adds bugfix to copy encryption passphrase for thick
provisioned PVC restored from snapshot.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-10-05 07:46:57 +00:00
Rakshith R
b471cac6bd e2e: add nolint:param to retryKubectlArgs
Currently only kubectlCreate arg is used with retryKubectlArgs(),
But it maybe used later on.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-10-05 07:46:57 +00:00
Rakshith R
dac4e76ae1 e2e: add testcase for PVC restore from vaultKMS to vaultTenantSAKMS
Signed-off-by: Rakshith R <rar@redhat.com>
2021-10-05 07:46:57 +00:00
Rakshith R
f63ed2ca5a e2e: modify validatePVCSnapshot() to use restoreSCName & restoreKMS
Signed-off-by: Rakshith R <rar@redhat.com>
2021-10-05 07:46:57 +00:00
Rakshith R
59b7a26175 rbd: modify copyEncryptionConfig to accept copyOnlyPassphrase arg
During PVC snapshot/clone both kms config and passphrase needs to copied,
while for PVC restore only passphrase needs to be copied to dest rbdvol
since destination storageclass may have another kms config.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-10-05 07:46:57 +00:00
Humble Chirammal
3c9d7e3cd5 rbd: detect migration volID in DeleteVolume() and delete rbd image
This commit adds the logic to detect a passed in volumeID
is a migrated volume ID and if yes, the driver connect to the
backend cluster and clean/delete the image. The logic
only applied if its a migration volume ID. The migration volume ID
carry the information like mons, pool and image name which is
good enough for the driver to identify and connect to the backend
cluster for its operations.

migration volID format:
<mig>_mons-<monsHash>_image-<imageUID>_<poolHash>

Details on the hash values:

* MonsHash: this carry a hash value (md5sum) which will be acted as the
`clusterID` for the operations in this context.

* ImageUID: this is the unique UUID generated by kubernetes for the created
volume.

* PoolHash: this is an encoded string of pool name.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Humble Chirammal
b778fe51a4 e2e: add test for migration volID detection and delete of image
This commit add test for migration delete volID detection scenario
by passing a custom volID and with the entries in configmap changed
to simulate the situation. The staticPV function also changed its
accept the annotation map which make it more general usage.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Humble Chirammal
1171111a94 e2e: deletePodWithLabel fails on unparam linter
this commit address the unparam linter error on deletePodWithLabel
function.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Humble Chirammal
a4a2dc93c1 e2e: change createCustomConfigmap to be more general
createCustomConfigmap helps to create a custom cluster entry in
the configmap, however this was coupled with subvolumegroup filling
in the cluster configuration. This commit helps to make it more
general and the subvolumegroup filling is controlled now with a flag

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-10-04 16:06:31 +00:00
Rakshith R
90d246fc55 doc: update dev standup meeting link
Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-30 13:27:11 +00:00
Niels de Vos
dfc8f64bdd ci: require passing of k8s-e2e-external-storage jobs for merge
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-29 12:45:02 +05:30
Niels de Vos
bb68cc9bee rebase: update vault/api to v1.1.1
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-29 05:55:26 +00:00
dependabot[bot]
b85076365c rebase: bump google.golang.org/grpc from 1.40.0 to 1.41.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.40.0 to 1.41.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.40.0...v1.41.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-28 11:36:21 +00:00
Niels de Vos
3480cb2c25 rebase: update minikube to v1.23.2
See-also: https://github.com/kubernetes/minikube/releases/tag/v1.23.2
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-09-28 09:28:28 +00:00
dependabot[bot]
27a2718f0b rebase: bump sigs.k8s.io/controller-runtime from 0.10.0 to 0.10.1
Bumps [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases)
- [Commits](https://github.com/kubernetes-sigs/controller-runtime/compare/v0.10.0...v0.10.1)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/controller-runtime
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-28 06:38:08 +00:00
dependabot[bot]
f0e2f84a40 rebase: bump github.com/aws/aws-sdk-go from 1.40.46 to 1.40.50
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.46 to 1.40.50.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.46...v1.40.50)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-28 04:29:46 +00:00
Yati Padia
1cf14cd83c cleanup: rework on naming conventions
This commits replaces cephfs -> cephFS
to maintain consistency throughout the
codebase

Updates: #1465

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-09-24 06:17:17 +00:00
Madhu Rajanna
34a21cdbe3 cleanup: move mount functions to new pkg
moved fuse and kernel mount functions
to a new package.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-23 06:39:37 +00:00
Madhu Rajanna
b1ef842640 cleanup: move core functions to core pkg
as we are refractoring the cephfs code,
Moving all the core functions to a new folder
/pkg called core. This will make things easier
to implement. For now onwards all the core
functionalities will be added to the core
package.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-23 06:39:37 +00:00
dependabot[bot]
64ade1d4c3 rebase: bump github.com/aws/aws-sdk-go from 1.40.34 to 1.40.46
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.34 to 1.40.46.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.34...v1.40.46)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-23 04:47:17 +00:00
Madhu Rajanna
2a8eb95966 ci: disable fsGroupChangePolicy change test for cephfs
disabling fsGroupChangePolicy test suite for
cephfs.

updates: #2017

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 10:34:41 +00:00
Madhu Rajanna
981835a073 deploy: remove extra permission for snapshot controller
removed extra PSP permissions added for the
snapshot controller deployment

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
e45bf03bd8 helm: remove securityContext from cephfs deployment
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations like mount,
selinux operations etc .

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
f267d77801 helm: remove extra volumes from rbd plugin PSP
removed extra volume permissions from the rbd
nodeplugin PSP

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
6f1066fd36 helm: reduce the PSP permission for rbd deployment
rbd deployment doesnot need extra permission like
privileged and extra volumes etc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
c154886926 helm: remove extra volumes from cephfs plugin PSP
removed extra volume permissions from the cephfs
nodeplugin PSP.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
8374fa929a helm: reduce the PSP permission for cephfs deployment
cephfs deployment doesnot need extra permission like
privileged,Capabilities and reduce unwanted volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
9bff7b0ac9 deploy: remove extra volumes from cephfs plugin PSP
removed extra volume permissions from the cephfs
nodeplugin PSP.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
051af3b257 deploy: reduce the PSP permission for cephfs deployment
cephfs deployment doesnot need extra permission like
privileged,Capabilities and remove unwanted volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
7fc1bf1321 deploy: remove extra volumes from rbd plugin PSP
removed extra volume permissions from the rbd
nodeplugin PSP.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
9e88fd1eb7 deploy: reduce the PSP permission for rbd deployment
rbd deployment doesnot need extra permission like
privileged,Capabilities and remove unwanted volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
e5569f0547 deploy: remove securityContext from rbd provisioner
we dont need securityContext for the rbd provisioner
pod as its not doing any special operations like map
,unmap selinux etc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
62a9ab6d9f helm: remove securityContext from cephfs provisioner
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Madhu Rajanna
f1c64a2a6b deploy: remove securityContext from cephfs provisioner
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations like mounts,
selinux etc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 07:12:34 +00:00
Rakshith R
8f432e8bfa rebase: update k8s.io packages from v0.22.1 to v0.22.2
This commit also fixes k8s.io/cloud-providers v0.22.2,
instead of v1.22.1 which does not exist and was overrided
in replace.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-09-21 08:09:12 +00:00
dependabot[bot]
c1931c8192 rebase: bump k8s.io/kubernetes from 1.22.1 to 1.22.2
Bumps [k8s.io/kubernetes](https://github.com/kubernetes/kubernetes) from 1.22.1 to 1.22.2.
- [Release notes](https://github.com/kubernetes/kubernetes/releases)
- [Commits](https://github.com/kubernetes/kubernetes/compare/v1.22.1...v1.22.2)

---
updated-dependencies:
- dependency-name: k8s.io/kubernetes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-21 08:09:12 +00:00
dependabot[bot]
566cef2719 rebase: bump github.com/csi-addons/spec from 0.1.0 to 0.1.1
Bumps [github.com/csi-addons/spec](https://github.com/csi-addons/spec) from 0.1.0 to 0.1.1.
- [Release notes](https://github.com/csi-addons/spec/releases)
- [Commits](https://github.com/csi-addons/spec/compare/v0.1.0...v0.1.1)

---
updated-dependencies:
- dependency-name: github.com/csi-addons/spec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-21 05:06:08 +00:00
Humble Chirammal
4804f47b18 e2e: Add e2e for rbd migration static pvc
This commit adds e2e for rbd migration static PVCs

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-20 09:54:54 +00:00
Humble Chirammal
2e8e8f5e64 rbd: fill clusterID if its a migration nodestage request
the migration nodestage request does not carry the 'clusterID' in it
and only monitors are available with the volumeContext. The volume
context flag 'migration=true' and 'static=true' flags allow us to
fill 'clusterID' from the passed in monitors to the volume Context,so
that rest of the static operations on nodestage can be proceeded as we
do treat static volumes today.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-20 09:54:54 +00:00
Humble Chirammal
1f5963919f util: get clusterID for the passed in mon string
as part of migration support, the clusterID has to be fetched
from passed in mon. Because the intree RBD storage class only
got monitor and not `clusterID` parameter support. However, in
CSI, SC has the `clusterID` parameter support but not mon. Due
to that we have to fetch the clusterID from config file for the
passed in mon and use it in our operations. This adds a helper
function to retrieve clusterID from passed in mon string.

Updates https://github.com/ceph/ceph-csi/issues/2509

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-09-20 09:54:54 +00:00
Prasanna Kumar Kalever
22bb31df19 doc: update the rbd-nbd doc with log strategies options
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Prasanna Kumar Kalever
c9cc36d8db rbd: provide alternatives to preserve the ceph log files
Currently, we delete the ceph client log file on unmap/detach.

This patch provides additional alternatives for users who would like to
persist the log files.

Strategies:
-----------
`remove`: delete log file on unmap/detach
`compress`: compress the log file to gzip on unmap/detach
`preserve`: preserve the log file in text format

Note that the default strategy will be remove on unmap, and these options
can be tweaked from the storage class

Compression size details example:

On Map: (with debug-rbd=20)
---------
$ ls -lh
-rw-r--r-- 1 root root 526K Sep  1 18:15
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-d2e89c87-0b4d-11ec-8ea6-160f128e682d.log

On unmap:
---------
$ ls -lh
-rw-r--r-- 1 root root  33K Sep  1 18:15
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-d2e89c87-0b4d-11ec-8ea6-160f128e682d.gz

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00
Prasanna Kumar Kalever
10bbb049f7 cleanup: passing pointers to larger type
Log:
internal/rbd/rbd_attach.go:424:2: hugeParam: dArgs is heavy (88 bytes);
consider passing it by pointer (gocritic)

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-09-16 13:55:15 +00:00