Commit Graph

75 Commits

Author SHA1 Message Date
ShyamsundarR
1a8f8e3c24 Add support for erasure coded pools
This commit adds support to mention dataPool parameter for the
topology constrained pools in the StorageClass, that can be
leveraged to mention erasure coded pool names to use for RBD
data instead of the replica pools.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
e73921f268 Add e2e tests for topology based provisioning
- This commit adds tests only for RBD, as CephFS still needs
an enhancement in CephFS subvolume commands to effectively use
topology based provisioning

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
5c4abf8347 Add topology support to ceph-csi
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
Humble Chirammal
4c96ad3c85 Changes to e2e to accomodate client-go changes and RunKubectlInput
With client-go v1.18.0 there is a change where Signatures on methods
in generated clientsets, dynamic, metadata, and scale clients have been
modified to accept context.Context as a first argument.
Signatures of Create, Update, and Patch methods have been updated to accept
CreateOptions, UpdateOptions and PatchOptions respectively.
Signatures of Delete and DeleteCollection methods now accept DeleteOptions
by value instead of by reference

The framework.RunkubectlInput now accepts namespace as the first parameter
which is also accommodated with this PR.

Signed-off-by: Humble Chirammal hchiramm@redhat.com
2020-04-14 10:50:12 +00:00
Madhu Rajanna
22f3dc3556 E2E: Add Ability to test with different user #919
Updated E2E to use a normal user other than admin user,
who is having access to create and mount cephfs PVC and
create and map rbd PVC

we will use the user created by rook who is having above
access.

Signed-off-by: Madhu Rajanna madhupr007@gmail.com

This PR adds a test case for #904

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Madhu Rajanna
a1f5425b88 Update cephcsi secret created by rook
as we need to include the encryption
secret key inside the secret created by the
rook, This PR will add the key and value required
for the encryption inside the secrets.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Madhu Rajanna
d09ffbd6de helm: add helm charts E2E
This PR adds the support for helm
installation, and cephcsi helm charts
deployment and teardown and also runs E2E
on for helm charts.

Add socat to provide port forwadring access for helm

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
b2dfcae802 Update configmap if already present
in e2e if the configmap is ready present,
we need to update it to make life simpler
for helm chart e2e.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
e788328750 Refractor E2E to reduce code duplication
Updated E2E to reduce code duplication
and create resouces in different namespaces.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
4690dc6ea7 Fix CI warning related to gosec
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
24405dc178 Fix undeclared golangci-lint warnings
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
c45c426215 Add cephcsi namespace and rook namespace flag
Added namespace flag to cephcsi to deploy cephcsi
resouces in different namespace.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
59fe8c1f2f Provide option to skip ceph-csi plugin deployment
To test helm charts in CI we need to skip the ceph-csi
deployment in E2E, This PR provides an option in E2E
to enable/disable cephcsi deployment.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
cb08a63a02 Fix flag provided but not defined issue
in go 1.13 we are facing above issue,
This commit fixs by calling testing.Init()

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
b4e6504e9b Fix mountoption issue in rbd
use mountoptions when mounting rbd to stagingpath
in stagevolume request, add E2E for mount options

fixes: #846
updates: #757

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-13 14:33:56 +00:00
Reinier Schoof
a4532fafd0 added volumeNamePrefix and snapshotNamePrefix as parameters for storageClass
this allows administrators to override the naming prefix for both volumes and snapshots
created by the rbd plugin.

Signed-off-by: Reinier Schoof <reinier@skoef.nl>
2020-02-25 05:03:51 +00:00
Madhu Rajanna
1ec2bc47e3 Add E2E for rbd static PVC
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
8dcb6a6105 Handle Delete operation if pool not found
If the backend rbd or cephfs pool is already deleted
we need to return success to the  DeleteVolume RPC
call to make it idempotent.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-11 15:58:15 +00:00
Vasyl Purchel
669dc4536f Reduce encryption KMS configuration SC parameters
* moves KMS type from StorageClass into KMS configuration itself
 * updates omapval used to identify KMS to only it's ID without the type

why?

1. when using multiple KMS configurations (not currently supported)
automated parsing of kms configuration will be failing because some
entries in configs won't comply with the requested type
2. less options are needed in the StorageClass and less data used to
identify the KMS

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
2020-02-10 15:21:11 +00:00
Vasyl Purchel
419ad0dd8e Adds per volume encryption with Vault integration
- adds proposal document for PVC encryption from PR448
- adds per-volume encription by generating encryption passphrase
  for each volume and storing it in a KMS
- adds HashiCorp Vault integration as a KMS for encryption passphrases
- avoids encrypting volume second time if it was already encrypted but
  no file system created
- avoids unnecessary checks if volume is a mapped device when encryption
  was not requested
- prevents resizing encrypted volumes (it is not currently supported)
- prevents creating snapshots from encrypted volumes to prevent attack
  on encryption key (security guard until re-encryption of volumes
  implemented)

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com

Fixes #420
Fixes #744
2020-02-05 05:18:56 +00:00
ShyamsundarR
b93ed21fe8 Add e2e test case for DeleteVolume success on backend volume deletion
If a backend volume is deleted, DeleteVolume call for the same should
succeed, detecting the image is missing and delete the related OMaps.

This commit adds a test case to ensure this is occuring correctly.

Updates #474

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-01-29 10:05:13 +00:00
Oguz Kilcan
aadce54b2f Added PodSecurityPolicy support 2020-01-22 08:19:42 +00:00
wilmardo
f04af5742d refact: Remove Kubernetes 1.13.x support
Signed-off-by: wilmardo <info@wilmardenouden.nl>
2020-01-20 10:32:30 +00:00
Humble Chirammal
7df51846da This commit adds "k8s.io/kubernetes/test/e2e/framework/config"
and its functions in E2E.

update vendor packages
log dismounter command output
use kube v1.17.1 in dependency

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-01-17 12:06:02 +00:00
Humble Chirammal
2d0ed298e3 Update framework imports and its functions.
Signed-off-by: Cedric Hauber <hauber.c@gmail.com>
2020-01-17 12:06:02 +00:00
Humble Chirammal
7c96c5c567 Remove unused import from e2e utils
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-01-17 12:06:02 +00:00
Madhu Rajanna
f69ec085cc Improve logging while waiting for deployment to complete
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-31 06:10:40 +00:00
Madhu Rajanna
ad7e5ca301 Add E2E for block PVC resize
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-31 06:10:40 +00:00
Madhu Rajanna
374e47046f Move resize E2E to new resize.go file
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-31 06:10:40 +00:00
Madhu Rajanna
7d15992769 Enable logging in E2E if test fails
source: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/kubectl/kubectl_utils.go
kubectlLogPod function is not exposed in above code so copied it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-18 12:35:17 +00:00
Madhu Rajanna
b4f20e941c Enable Block E2E for rbd
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-18 12:35:17 +00:00
Vasyl Purchel
166eaf700f Adds PVC encryption with LUKS
Adds encryption in StorageClass as a parameter. Encryption passphrase is
stored in kubernetes secrets per StorageClass. Implements rbd volume
encryption relying on dm-crypt and cryptsetup using LUKS extension

The change is related to proposal made earlier. This is a first part of
the full feature that adds encryption with passphrase stored in secrets.

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
Signed-off-by: Ioannis Papaioannou ioannis.papaioannou@workday.com
Signed-off-by: Paul Mc Auley paul.mcauley@workday.com
Signed-off-by: Sergio de Carvalho sergio.carvalho@workday.com
2019-12-16 08:12:44 +00:00
Humble Chirammal
7c8e66e427 Add resize check for XFS formatted FS
Lock out parellel requests against same volumeID
Remove pod after resize and validation in E2E

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-12-13 12:40:12 +00:00
Humble Chirammal
e69f462336 Add e2e tests for RBD resizer
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-12-13 12:40:12 +00:00
Woohyung Han
8a16f740d6 Update golangci-lint version to v1.21.0
Signed-off-by: Woohyung Han <techhanx@gmail.com>
2019-12-12 04:57:14 +00:00
Woohyung Han
5c96ab1d0e Fix to use kubectl create not kubectl apply in the e2e
Signed-off-by: Woohyung Han <techhanx@gmail.com>
2019-12-11 09:31:03 +00:00
Woohyung Han
2c9d711463 Move rook-deploy code from e2e to ./scripts/minikube.sh
We have the e2e test with --deploy-rook=true that makes all test
environment. It works fine, but It does not seem to be the role of
e2e test. In addition, when developing the code we need to run full
test scenario with deploying rook every time, or we need to build
rook environment by hand. Move rook-deploy code to minikube.sh.
2019-12-05 12:32:59 +00:00
Woohyung Han
685e2540a8 Fix typo in deleteConfigMap 2019-12-04 04:31:50 +00:00
Woohyung Han
63589f3314 Fix kubeconfig default working correctly
Kuberentes e2e framework get kubeconfig information from
KUBECONFIG environment variable. set the variable by default.
2019-12-02 10:02:47 +00:00
Humble Chirammal
ac09c5553c Add E2E for cephfs resize functionality
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-11-27 14:00:47 +00:00
Madhu Rajanna
d590434374 Delete provisioner deployment deployed by rook
If kube version is == 1.13.x cephfs
and rbd provisioner are deployed as statefulset
and if kube version is > 1.13.x cephfs and
rbd provisioner are deployed as deployment

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-20 17:45:04 +00:00
techhanx
b1efb36b38 Add skip wait for rook-ceph-agent when version > 1.1.0
after version 1.1.0, flex driver is disabled and doesn't need to
wait for rook-ceph-agent pod anymore. Add skip wait for rook-ceph
-agent.
2019-11-19 11:43:39 +00:00
techhanx
8e117db80d Change e2e rook-ceph default version to 'v1.1.2' not master
When rook-ceph is upgraded and changed some feature, e2e can be
failed. Change rook-ceph default verion to 'v1.1.2' explicitly
which is working fine in current code.
2019-10-21 08:40:58 +00:00
Madhu Rajanna
69b7c4672f Added E2E for rbd nodeplugin restart
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-23 13:22:29 +00:00
Humble Chirammal
0e6617e1ff Format RBD volumes with nodiscard formatting options.
Currently rbd CSI plugin uses formatAndMount of
mount.SafeFormatAndMount. This does not allow to pass or use
specific formatting arguments with it. This patch introduce
RBD specific formatting options with both xfs and ext4,
for example: -E no-discard with ext4 and -k option with
XFS to boost formatting performance of RBD device.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-09-20 13:16:53 +00:00
Niels de Vos
d03a352abf e2e: correct log format in execCommandInPod()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2019-09-04 12:26:55 +00:00
Madhu Rajanna
64ca401a51 Fix mon endpoint issue in E2E
in toolbox mon endpoints are not
updated properly, this is causing an issue in E2E
this PR is a workaround to fix this issue.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-04 11:12:07 +00:00
Madhu Rajanna
ae67534a44 Add wait logic to check fsid from toolbox
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 06:50:32 +00:00
Madhu Rajanna
7d3a18c5b7 Fix e2e failure caused due to mon connection from toolbox
i think now its take time to discover the mon IP
from svc name in tool box, this is a workaround
to fix it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-29 11:17:54 +00:00
Niels de Vos
b812ec26df e2e: do not fail deleting resources when "resource not found"
Sometimes the tests fail cleaning up due unavailable resources that are
listed in the .yaml files. Deleting the missing resources returns
"resource not found". By passing --ignore-not-found to kubectl, this
problem should not happen anymore (and possibly makes it more obvious
where tests do go wrong).
2019-08-13 12:46:07 +00:00