Commit Graph

2736 Commits

Author SHA1 Message Date
Niels de Vos
ad0b8897bf tests: gosec does not support '-mod=vendor'
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 12:38:02 +00:00
Niels de Vos
a262063819 tests: use go modules from ./vendor for running tests
Running tests without `-mod=vendor` causes the tests to download the
dependencies if these are not available in the standard go-module
directories (parent directories of the project). All dependencies are
already included in the ./vendor directory, so passing `-mod=vendor`
prevents downloading the dependencies and speeds up testing a lot.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 12:38:02 +00:00
Niels de Vos
79b0b8cfa6 travis: re-enable running on arm64
Travis CI has fixed their Ubuntu Xenial images for arm64, so the job can
be enabled again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-20 12:01:56 +00:00
Humble Chirammal
8265c431a7 Bring attacher controllers to latest version
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-03-20 11:09:05 +00:00
Madhu Rajanna
e788328750 Refractor E2E to reduce code duplication
Updated E2E to reduce code duplication
and create resouces in different namespaces.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
4690dc6ea7 Fix CI warning related to gosec
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
24405dc178 Fix undeclared golangci-lint warnings
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
c45c426215 Add cephcsi namespace and rook namespace flag
Added namespace flag to cephcsi to deploy cephcsi
resouces in different namespace.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
59fe8c1f2f Provide option to skip ceph-csi plugin deployment
To test helm charts in CI we need to skip the ceph-csi
deployment in E2E, This PR provides an option in E2E
to enable/disable cephcsi deployment.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
0f6bceb662 Update readme for new releases
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 08:55:35 +00:00
Niels de Vos
d15b77d403 dev-guide: add reference to required go-ceph dependencies
Closes: #872
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-19 17:50:49 +00:00
Madhu Rajanna
7f8c535c42 Update upgrade doc for node hang issue
This PR updates the upgrade doc to handle the
node drain issue what we have seen in
https://github.com/ceph/ceph-csi/issues/756

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-19 14:56:04 +00:00
Madhu Rajanna
179084a60a Skip arm64 building in CI
currently we are facing an issue related to
mongodb in arm64 travis CI

updates: #873

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-19 13:47:56 +00:00
Madhu Rajanna
04ea2b6f81 Fix issue in profile.cov
profile.cov need to be created inside
the _output directory.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
cb08a63a02 Fix flag provided but not defined issue
in go 1.13 we are facing above issue,
This commit fixs by calling testing.Init()

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
d5a0606c33 Migrate from dep to go module
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
a9174dd953 Fix logging if the rbd manager command is supported
if there is an error when adding the rbd task
we are logging the output which is empty. This
PR logs the error if the rbd task is supported
and there is an error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-16 11:49:31 +00:00
Madhu Rajanna
b4e6504e9b Fix mountoption issue in rbd
use mountoptions when mounting rbd to stagingpath
in stagevolume request, add E2E for mount options

fixes: #846
updates: #757

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-13 14:33:56 +00:00
Niels de Vos
40d0d5d291 rbd: drop references to ImageFormat
librbd only supports ImageFormat 2. It is not expected that anyone has a
different version of the format in container environments.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
8dc3600899 rbd: use go-ceph API for creating RBD images
This is the initial step for improving performance during provisioning
of CSI volumes backed by RBD.

While creating a volume, an existing connection to the Ceph cluster is
used from the ConnPool. This should speed up the creation of a batch of
volumes significantly.

Updates: #449
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
90f81516ee util/conn_pool: add tests for ConnPool
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
397825c665 util: add ConnPool for connection re-use
By using the ConnPool it is not needed to re-connect every time to the
Ceph cluster when (rbd) operations are executed through the go-ceph/rbd
API.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
ba99275f90 dep: add github.com/ceph/go-ceph for rbd API
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
3226b17d08 enable building with cgo for go-ceph usage
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
66b73e3682 travis: install Ceph development packages for go-ceph
The Ceph community does not guarantee that arm64 packages get build and
published with each release. In order to prevent falling back to ancient
Ceph versions from ubuntu-ports, add an unsigned repository with Ceph
Nautilus 14.2.5.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Reinier Schoof
ca8dd2d8f2 use VolSize from rbdVolume instead of separate parameter 2020-03-10 11:34:53 +00:00
Reinier Schoof
8da7e4bbf9 removed unreachable code path 2020-03-10 11:34:53 +00:00
Reinier Schoof
3af5e0619f created struct for keeping the state of a staging transaction
this way extending transaction rollbacks is easier

Signed-off-by: Reinier Schoof <reinier@skoef.nl>
2020-03-10 11:34:53 +00:00
Jonas Rutishauser
9d7b50dccb Added forcecephkernelclient as startup parameter
Support #664 in the helm chart.

Signed-off-by: Jonas Rutishauser <jonas.rutishauser@alumni.ethz.ch>
2020-03-03 09:50:42 +05:30
Madhu Rajanna
128f3fc2cf check subvolume present in backend
If a CreateVolume call is interrupted,
post creating the required CSI journal entries,
but prior to creating the backing CephFS subvolume,
then a subsequent CreateVolume call will return
a valid response with a VolumeID that has
it's backing image missing. This PR adds a check
for backend image, if image notfound it deletes the
reserved keys in omap.

fixes #839

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-25 11:46:04 +00:00
Reinier Schoof
a4532fafd0 added volumeNamePrefix and snapshotNamePrefix as parameters for storageClass
this allows administrators to override the naming prefix for both volumes and snapshots
created by the rbd plugin.

Signed-off-by: Reinier Schoof <reinier@skoef.nl>
2020-02-25 05:03:51 +00:00
Madhu Rajanna
8163552b81 Add doc for rbd static pvc support
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
1ec2bc47e3 Add E2E for rbd static PVC
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
9f15dded6d Add support for static rbd volumes
currently its not possible to create
and mount static PVC to a application pod
using rbd csi driver. This PR adds the support
for static PVC in ceph-csi driver for rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
0f80ec0664 Return err from nodeserver
getVolumeNameByID function is already
returrning the status.Error return the err
as it is.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
90235c32f8 Fix branch name in notes.txt
currently notes.txt container old branch name.
This PR updated the branch name with release

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-13 21:36:27 +00:00
Madhu Rajanna
d02dfe2dfe Remove unwanted RBAC rules from ceph-csi
There are currently unwanted RBAC permission
is given for ceph-csi, This PR reduces removes
such unwanted RBAC resources.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-13 21:36:27 +00:00
Madhu Rajanna
8dcb6a6105 Handle Delete operation if pool not found
If the backend rbd or cephfs pool is already deleted
we need to return success to the  DeleteVolume RPC
call to make it idempotent.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-11 15:58:15 +00:00
Madhu Rajanna
034b123478 Remove mount cache for cephfs
PR #282 introduces the mount cache to
solve cephfs fuse mount issue when cephfs plugin pod
restarts .This is not working as intended. This PR removes
the code for maintainability.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-11 15:11:21 +00:00
Vasyl Purchel
669dc4536f Reduce encryption KMS configuration SC parameters
* moves KMS type from StorageClass into KMS configuration itself
 * updates omapval used to identify KMS to only it's ID without the type

why?

1. when using multiple KMS configurations (not currently supported)
automated parsing of kms configuration will be failing because some
entries in configs won't comply with the requested type
2. less options are needed in the StorageClass and less data used to
identify the KMS

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
2020-02-10 15:21:11 +00:00
sophal
1695c6965d Add printing failed message when timeout
Signed-off-by: sophal <sophalHong@github.com>
2020-02-07 13:05:30 +00:00
Vasyl Purchel
419ad0dd8e Adds per volume encryption with Vault integration
- adds proposal document for PVC encryption from PR448
- adds per-volume encription by generating encryption passphrase
  for each volume and storing it in a KMS
- adds HashiCorp Vault integration as a KMS for encryption passphrases
- avoids encrypting volume second time if it was already encrypted but
  no file system created
- avoids unnecessary checks if volume is a mapped device when encryption
  was not requested
- prevents resizing encrypted volumes (it is not currently supported)
- prevents creating snapshots from encrypted volumes to prevent attack
  on encryption key (security guard until re-encryption of volumes
  implemented)

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com

Fixes #420
Fixes #744
2020-02-05 05:18:56 +00:00
Jakub Kuzelka
1adef00c86 Repair typo in host-mount and host-dev definition. It was swithced between each other and actually there is not possible to deploy nodeplugin-daemonset via helm chart. 2020-01-29 16:00:10 +00:00
ShyamsundarR
35e8c3b3a5 CephFS: Added ENOENT checks for possible missing volumes
Added checks in DeleteVolume RPC, for image missing errors, and
taking appropriate actions to cleanup the CSI reservations.

Further removed forcing a volume purge, and instead added checks
for missing volume errors in purgeVolume.

This should now fix issues where an continuation of an interrupted
DeleteVolume call, that only deleted the backing volume, will
proceed and not error out.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-01-29 10:05:13 +00:00
ShyamsundarR
b93ed21fe8 Add e2e test case for DeleteVolume success on backend volume deletion
If a backend volume is deleted, DeleteVolume call for the same should
succeed, detecting the image is missing and delete the related OMaps.

This commit adds a test case to ensure this is occuring correctly.

Updates #474

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-01-29 10:05:13 +00:00
Madhu Rajanna
eb2fb9233b Add run hostpath to daemonset pods
`/run/mount` need to be share between host and
csi-plugin containers for `/run/mount/utab`

this is required to ensures that the network
is not stopped prior to unmounting the network devices.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 16:50:18 +00:00
Madhu Rajanna
881f59d142 Add _netdev as default mount options in plugin
This values will be added at both nodestage
and nodepublish for rbd, nbd and ceph kernel client.

As cephfs fuse doesnot support this value,
this is added only during the nodepublish.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 16:50:18 +00:00
Oguz Kilcan
aadce54b2f Added PodSecurityPolicy support 2020-01-22 08:19:42 +00:00
Niels de Vos
7ec8e68b67 examples/rbd-block: use a container image that is supported
Fedora 26 has been End-Of-Life for a long time already, is should not be
used anymore. Instead use the latest CentOS image that gets regular
updates. The Ceph base image used by Ceph-CSI is also based on CentOS,
so uses would be familiar with the available tools.

Also use "sleep infinity" instead of a weird 'tail -f /dev/null' hack.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-01-21 13:30:45 +00:00
Madhu Rajanna
520ceb6dcb Fix allignment issue in shellscript
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-20 17:33:03 +05:30