Commit Graph

59 Commits

Author SHA1 Message Date
Yug Gupta
c9ad904331 cephfs: Add support for multiple subvolumegroups
With the current code base, the subvolumegroup will
be created once, and even for a different cluster,
subvolumegroup creation is not allowed again.

Added support multiple subvolumegroups creation by
validating one subvolumegroup creation per cluster.

Fixes: #1123

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-26 09:19:24 +00:00
Humble Chirammal
dcbf71dd5d cleanup: avoid double import of kubernetes library in e2e utils
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-06-19 08:42:08 +00:00
Humble Chirammal
323cc0e3bb cleanup: avoid comparing errors directly
Go 1.13 contains support for error wrapping. To support wrapping,
fmt.Errorf now has a %w verb for creating wrapped errors, and three
new functions in the errors package ( errors.Unwrap, errors.Is and
errors.As) simplify unwrapping and inspecting wrapped errors.

With this change, If we currently compare errors using ==, we have to
use errors.Is instead. Example:

if err == io.ErrUnexpectedEOF

becomes

if errors.Is(err, io.ErrUnexpectedEOF)

https://tip.golang.org/doc/go1.13#error_wrapping

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-06-17 06:26:11 +00:00
Madhu Rajanna
78cdf9d4db e2e: Add option to set mountoptions in sc
Updated E2E code to pass mountoptions when
creating a storageclass.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-01 11:27:25 +00:00
Madhu Rajanna
7a5e064f50 e2e: Add a new function to run commands in toolboxpod
Updated code to use new function execCommandInToolBoxPod
to run commands in toolboxpod.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-20 06:37:36 +00:00
Madhu Rajanna
f0b3cee94a e2e: define replicapool as constant
the static check is failing as the replicapool
is used in 3 or more places, we need to define
a variable and use it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
c5bf95b625 e2e: Enable E2E for rbd snapshot
Updated E2E to use beta client and beta
Snapshot resources.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Yug Gupta
2cdf5c3b9f util: make util.ClusterInfo usable outside util package
functions like getClusterInfo() returns struct
instead of a set of strings.

Fix: #998

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-14 19:43:21 +00:00
Mathias Merscher
0991cdf498 make CephFS SubvolumeGroup configurable
The name of the CephFS SubvolumeGroup for the CSI volumes was hardcoded to "csi". To make permission management in multi tenancy environments easier, this commit makes it possible to configure the CSI SubvolumeGroup.

related to #798 and #931
2020-05-04 05:50:06 +00:00
Niels de Vos
7d642b791c e2e: fix IneffAssign warnings in checkPVSelectorValuesForPVC()
IneffAssign warns about the two following statements:

  Line 1342: warning: ineffectual assignment to rFound (ineffassign)
  Line 1350: warning: ineffectual assignment to zFound (ineffassign)

rFound and zFound should be set before entering the loop, otherwise the
initial value will overwrite the updated value on each iteration.

Reported-by: https://goreportcard.com/report/github.com/ceph/ceph-csi
Updates: #975
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-24 08:35:36 +00:00
ShyamsundarR
1a8f8e3c24 Add support for erasure coded pools
This commit adds support to mention dataPool parameter for the
topology constrained pools in the StorageClass, that can be
leveraged to mention erasure coded pool names to use for RBD
data instead of the replica pools.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
e73921f268 Add e2e tests for topology based provisioning
- This commit adds tests only for RBD, as CephFS still needs
an enhancement in CephFS subvolume commands to effectively use
topology based provisioning

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
Humble Chirammal
4c96ad3c85 Changes to e2e to accomodate client-go changes and RunKubectlInput
With client-go v1.18.0 there is a change where Signatures on methods
in generated clientsets, dynamic, metadata, and scale clients have been
modified to accept context.Context as a first argument.
Signatures of Create, Update, and Patch methods have been updated to accept
CreateOptions, UpdateOptions and PatchOptions respectively.
Signatures of Delete and DeleteCollection methods now accept DeleteOptions
by value instead of by reference

The framework.RunkubectlInput now accepts namespace as the first parameter
which is also accommodated with this PR.

Signed-off-by: Humble Chirammal hchiramm@redhat.com
2020-04-14 10:50:12 +00:00
Madhu Rajanna
22f3dc3556 E2E: Add Ability to test with different user #919
Updated E2E to use a normal user other than admin user,
who is having access to create and mount cephfs PVC and
create and map rbd PVC

we will use the user created by rook who is having above
access.

Signed-off-by: Madhu Rajanna madhupr007@gmail.com

This PR adds a test case for #904

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Madhu Rajanna
a1f5425b88 Update cephcsi secret created by rook
as we need to include the encryption
secret key inside the secret created by the
rook, This PR will add the key and value required
for the encryption inside the secrets.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Madhu Rajanna
d09ffbd6de helm: add helm charts E2E
This PR adds the support for helm
installation, and cephcsi helm charts
deployment and teardown and also runs E2E
on for helm charts.

Add socat to provide port forwadring access for helm

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
b2dfcae802 Update configmap if already present
in e2e if the configmap is ready present,
we need to update it to make life simpler
for helm chart e2e.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
e788328750 Refractor E2E to reduce code duplication
Updated E2E to reduce code duplication
and create resouces in different namespaces.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
4690dc6ea7 Fix CI warning related to gosec
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
24405dc178 Fix undeclared golangci-lint warnings
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
c45c426215 Add cephcsi namespace and rook namespace flag
Added namespace flag to cephcsi to deploy cephcsi
resouces in different namespace.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
b4e6504e9b Fix mountoption issue in rbd
use mountoptions when mounting rbd to stagingpath
in stagevolume request, add E2E for mount options

fixes: #846
updates: #757

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-13 14:33:56 +00:00
Madhu Rajanna
8dcb6a6105 Handle Delete operation if pool not found
If the backend rbd or cephfs pool is already deleted
we need to return success to the  DeleteVolume RPC
call to make it idempotent.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-11 15:58:15 +00:00
Vasyl Purchel
419ad0dd8e Adds per volume encryption with Vault integration
- adds proposal document for PVC encryption from PR448
- adds per-volume encription by generating encryption passphrase
  for each volume and storing it in a KMS
- adds HashiCorp Vault integration as a KMS for encryption passphrases
- avoids encrypting volume second time if it was already encrypted but
  no file system created
- avoids unnecessary checks if volume is a mapped device when encryption
  was not requested
- prevents resizing encrypted volumes (it is not currently supported)
- prevents creating snapshots from encrypted volumes to prevent attack
  on encryption key (security guard until re-encryption of volumes
  implemented)

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com

Fixes #420
Fixes #744
2020-02-05 05:18:56 +00:00
ShyamsundarR
b93ed21fe8 Add e2e test case for DeleteVolume success on backend volume deletion
If a backend volume is deleted, DeleteVolume call for the same should
succeed, detecting the image is missing and delete the related OMaps.

This commit adds a test case to ensure this is occuring correctly.

Updates #474

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-01-29 10:05:13 +00:00
wilmardo
f04af5742d refact: Remove Kubernetes 1.13.x support
Signed-off-by: wilmardo <info@wilmardenouden.nl>
2020-01-20 10:32:30 +00:00
Humble Chirammal
7df51846da This commit adds "k8s.io/kubernetes/test/e2e/framework/config"
and its functions in E2E.

update vendor packages
log dismounter command output
use kube v1.17.1 in dependency

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-01-17 12:06:02 +00:00
Humble Chirammal
2d0ed298e3 Update framework imports and its functions.
Signed-off-by: Cedric Hauber <hauber.c@gmail.com>
2020-01-17 12:06:02 +00:00
Humble Chirammal
7c96c5c567 Remove unused import from e2e utils
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-01-17 12:06:02 +00:00
Madhu Rajanna
f69ec085cc Improve logging while waiting for deployment to complete
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-31 06:10:40 +00:00
Madhu Rajanna
374e47046f Move resize E2E to new resize.go file
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-31 06:10:40 +00:00
Vasyl Purchel
166eaf700f Adds PVC encryption with LUKS
Adds encryption in StorageClass as a parameter. Encryption passphrase is
stored in kubernetes secrets per StorageClass. Implements rbd volume
encryption relying on dm-crypt and cryptsetup using LUKS extension

The change is related to proposal made earlier. This is a first part of
the full feature that adds encryption with passphrase stored in secrets.

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
Signed-off-by: Ioannis Papaioannou ioannis.papaioannou@workday.com
Signed-off-by: Paul Mc Auley paul.mcauley@workday.com
Signed-off-by: Sergio de Carvalho sergio.carvalho@workday.com
2019-12-16 08:12:44 +00:00
Humble Chirammal
7c8e66e427 Add resize check for XFS formatted FS
Lock out parellel requests against same volumeID
Remove pod after resize and validation in E2E

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-12-13 12:40:12 +00:00
Humble Chirammal
e69f462336 Add e2e tests for RBD resizer
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-12-13 12:40:12 +00:00
Woohyung Han
8a16f740d6 Update golangci-lint version to v1.21.0
Signed-off-by: Woohyung Han <techhanx@gmail.com>
2019-12-12 04:57:14 +00:00
Woohyung Han
2c9d711463 Move rook-deploy code from e2e to ./scripts/minikube.sh
We have the e2e test with --deploy-rook=true that makes all test
environment. It works fine, but It does not seem to be the role of
e2e test. In addition, when developing the code we need to run full
test scenario with deploying rook every time, or we need to build
rook environment by hand. Move rook-deploy code to minikube.sh.
2019-12-05 12:32:59 +00:00
Woohyung Han
685e2540a8 Fix typo in deleteConfigMap 2019-12-04 04:31:50 +00:00
Humble Chirammal
ac09c5553c Add E2E for cephfs resize functionality
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-11-27 14:00:47 +00:00
Madhu Rajanna
69b7c4672f Added E2E for rbd nodeplugin restart
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-23 13:22:29 +00:00
Humble Chirammal
0e6617e1ff Format RBD volumes with nodiscard formatting options.
Currently rbd CSI plugin uses formatAndMount of
mount.SafeFormatAndMount. This does not allow to pass or use
specific formatting arguments with it. This patch introduce
RBD specific formatting options with both xfs and ext4,
for example: -E no-discard with ext4 and -k option with
XFS to boost formatting performance of RBD device.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-09-20 13:16:53 +00:00
Niels de Vos
d03a352abf e2e: correct log format in execCommandInPod()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2019-09-04 12:26:55 +00:00
Madhu Rajanna
64ca401a51 Fix mon endpoint issue in E2E
in toolbox mon endpoints are not
updated properly, this is causing an issue in E2E
this PR is a workaround to fix this issue.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-04 11:12:07 +00:00
Madhu Rajanna
ae67534a44 Add wait logic to check fsid from toolbox
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 06:50:32 +00:00
Madhu Rajanna
90fef919d5 Skip snapshot testing in CI
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-13 16:22:55 +05:30
Madhu Rajanna
4ed187eb3f Fix issues in E2E cleanup
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-07 20:26:20 +05:30
Daniel-Pivonka
0063727199 Make parameter pool optional in CephFS storageclass
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
2019-08-07 13:30:38 +00:00
Madhu Rajanna
02bcb5f16a Enable leader election in v1.14+
Use Deployment with leader election instead of StatefulSet

Deployment behaves better when a node gets disconnected
from the rest of the cluster - new provisioner leader
is elected in ~15 seconds, while it may take up to
5 minutes for StatefulSet to start a new replica.

Refer: kubernetes-csi/external-provisioner@52d1fbc

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-05 07:11:44 +00:00
Poornima G
da111de65d Revert "Temperory workaround for the regression to pass"
The rook commit e52efdb8d4ed98129708d49cbbbbbcb083b02d6c fixes the
issue. Hence reverting this patch.
2019-07-25 08:49:43 +00:00
Madhu Rajanna
f4c80dec9a Implement NodeStage and NodeUnstage for rbd
in NodeStage RPC call  we  have to map the
device to the node plugin and make  sure  the
the device will be mounted to  the global path

in  nodeUnstage request unmount the device from
global path and unmap the device

if the volume mode is block  we will be creating
a file inside a stageTargetPath  and it will be
considered  as the global path

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-07-24 12:49:21 +00:00
Poornima G
7d8f465746 Temperory workaround for the regression to pass
Currently the cephfs PVC create/delete and all other operations
related to cephfs are failing. One of the recent commits in rook
900abbc967e108ad622648b740a7c57f1268209f has modified ceph-mgr
to run as ceph user rather than root user. The ceph user currently
has no permission to write to the root of the cephfs filesystem.
The fix will be external to CSI itself, but until that lands, sending
a workaround patch so the CSI CI is unblocked

In this patch, we are setting the permission 777 on root of the cephfs
filesystem. Thus ceph user will be able to modify the cephfs filesystem.

Signed-off-by: Poornima G <pgurusid@redhat.com>
2019-07-24 12:04:46 +00:00