Commit Graph

1144 Commits

Author SHA1 Message Date
Wong Hoi Sing Edison
d67ad47c45 Upgrade: csi-attacher from v2.1.0 to v2.1.1
This PR follow up the missing files from ceph-csi:v2.0.1, besides:

  - deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml
  - deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml

See https://github.com/kubernetes-csi/external-attacher/releases/tag/v2.1.1
See https://github.com/kubernetes-csi/external-attacher/blob/v2.1.1/CHANGELOG-2.1.md
2020-03-31 05:33:18 +00:00
Madhu Rajanna
84aa1ba392 Use Error instead of Errorf
If the string formatting is not required
use Error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-30 08:53:16 +00:00
Humble Chirammal
b1dfcb4d7e Correct static errors and source code comments.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-03-30 05:13:24 +00:00
xu.chen
399f0b0d89 Audit log and follow klog standard 2020-03-27 09:24:52 +00:00
Niels de Vos
2124f6e775 travis: add job to test 'make containerized-build'
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-26 08:45:53 +00:00
Niels de Vos
7381253ee0 build: add an option to compile in a container
This makes it possible to build on any platform that supports Linux
containers. The container image used for building is created once, or on
updating the `scripts/Dockerfile.build` and is cached afterwards.

To build the executable in a container, use `make containerized-build`
and everything will be done automatically. The executable will also be
available on the usual location.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-26 08:45:53 +00:00
root
ae4d269836 fix typos 2020-03-24 15:43:03 +00:00
Niels de Vos
825448825c examples: rbd/storageclass.yaml causes Python excetion with yamllint
While running the 'make test' target and have 'yamllint' available, the
test fails with the following exception:

yamllint -s -d {extends: default, rules: {line-length: {allow-non-breakable-inline-mappings: true}},ignore: charts/*/templates/*.yaml} ./examples/rbd/storageclass.yaml
Traceback (most recent call last):
  File "/usr/local/bin/yamllint", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.6/site-packages/yamllint/cli.py", line 181, in run
    problems = linter.run(f, conf, filepath)
  File "/usr/local/lib/python3.6/site-packages/yamllint/linter.py", line 237, in run
    content = input.read()
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1947: ordinal not in range(128)

The quotes used in the comments seem to be non-ascii characters.
Replacing these with standard " makes the test pass again.

This problem occurred while running tests in a container based on the
Ceph image (CentOS-7) with Python 3. Travis CI might still use Python 2
for yamllint, and hide the problem.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 14:42:25 +00:00
Niels de Vos
ad0b8897bf tests: gosec does not support '-mod=vendor'
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 12:38:02 +00:00
Niels de Vos
a262063819 tests: use go modules from ./vendor for running tests
Running tests without `-mod=vendor` causes the tests to download the
dependencies if these are not available in the standard go-module
directories (parent directories of the project). All dependencies are
already included in the ./vendor directory, so passing `-mod=vendor`
prevents downloading the dependencies and speeds up testing a lot.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 12:38:02 +00:00
Niels de Vos
79b0b8cfa6 travis: re-enable running on arm64
Travis CI has fixed their Ubuntu Xenial images for arm64, so the job can
be enabled again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-20 12:01:56 +00:00
Humble Chirammal
8265c431a7 Bring attacher controllers to latest version
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-03-20 11:09:05 +00:00
Madhu Rajanna
e788328750 Refractor E2E to reduce code duplication
Updated E2E to reduce code duplication
and create resouces in different namespaces.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
4690dc6ea7 Fix CI warning related to gosec
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
24405dc178 Fix undeclared golangci-lint warnings
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
c45c426215 Add cephcsi namespace and rook namespace flag
Added namespace flag to cephcsi to deploy cephcsi
resouces in different namespace.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
59fe8c1f2f Provide option to skip ceph-csi plugin deployment
To test helm charts in CI we need to skip the ceph-csi
deployment in E2E, This PR provides an option in E2E
to enable/disable cephcsi deployment.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 10:16:23 +00:00
Madhu Rajanna
0f6bceb662 Update readme for new releases
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-20 08:55:35 +00:00
Niels de Vos
d15b77d403 dev-guide: add reference to required go-ceph dependencies
Closes: #872
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-19 17:50:49 +00:00
Madhu Rajanna
7f8c535c42 Update upgrade doc for node hang issue
This PR updates the upgrade doc to handle the
node drain issue what we have seen in
https://github.com/ceph/ceph-csi/issues/756

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-19 14:56:04 +00:00
Madhu Rajanna
179084a60a Skip arm64 building in CI
currently we are facing an issue related to
mongodb in arm64 travis CI

updates: #873

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-19 13:47:56 +00:00
Madhu Rajanna
04ea2b6f81 Fix issue in profile.cov
profile.cov need to be created inside
the _output directory.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
cb08a63a02 Fix flag provided but not defined issue
in go 1.13 we are facing above issue,
This commit fixs by calling testing.Init()

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
d5a0606c33 Migrate from dep to go module
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Madhu Rajanna
a9174dd953 Fix logging if the rbd manager command is supported
if there is an error when adding the rbd task
we are logging the output which is empty. This
PR logs the error if the rbd task is supported
and there is an error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-16 11:49:31 +00:00
Madhu Rajanna
b4e6504e9b Fix mountoption issue in rbd
use mountoptions when mounting rbd to stagingpath
in stagevolume request, add E2E for mount options

fixes: #846
updates: #757

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-13 14:33:56 +00:00
Niels de Vos
40d0d5d291 rbd: drop references to ImageFormat
librbd only supports ImageFormat 2. It is not expected that anyone has a
different version of the format in container environments.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
8dc3600899 rbd: use go-ceph API for creating RBD images
This is the initial step for improving performance during provisioning
of CSI volumes backed by RBD.

While creating a volume, an existing connection to the Ceph cluster is
used from the ConnPool. This should speed up the creation of a batch of
volumes significantly.

Updates: #449
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
90f81516ee util/conn_pool: add tests for ConnPool
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
397825c665 util: add ConnPool for connection re-use
By using the ConnPool it is not needed to re-connect every time to the
Ceph cluster when (rbd) operations are executed through the go-ceph/rbd
API.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
ba99275f90 dep: add github.com/ceph/go-ceph for rbd API
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
3226b17d08 enable building with cgo for go-ceph usage
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Niels de Vos
66b73e3682 travis: install Ceph development packages for go-ceph
The Ceph community does not guarantee that arm64 packages get build and
published with each release. In order to prevent falling back to ancient
Ceph versions from ubuntu-ports, add an unsigned repository with Ceph
Nautilus 14.2.5.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Reinier Schoof
ca8dd2d8f2 use VolSize from rbdVolume instead of separate parameter 2020-03-10 11:34:53 +00:00
Reinier Schoof
8da7e4bbf9 removed unreachable code path 2020-03-10 11:34:53 +00:00
Reinier Schoof
3af5e0619f created struct for keeping the state of a staging transaction
this way extending transaction rollbacks is easier

Signed-off-by: Reinier Schoof <reinier@skoef.nl>
2020-03-10 11:34:53 +00:00
Jonas Rutishauser
9d7b50dccb Added forcecephkernelclient as startup parameter
Support #664 in the helm chart.

Signed-off-by: Jonas Rutishauser <jonas.rutishauser@alumni.ethz.ch>
2020-03-03 09:50:42 +05:30
Madhu Rajanna
128f3fc2cf check subvolume present in backend
If a CreateVolume call is interrupted,
post creating the required CSI journal entries,
but prior to creating the backing CephFS subvolume,
then a subsequent CreateVolume call will return
a valid response with a VolumeID that has
it's backing image missing. This PR adds a check
for backend image, if image notfound it deletes the
reserved keys in omap.

fixes #839

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-25 11:46:04 +00:00
Reinier Schoof
a4532fafd0 added volumeNamePrefix and snapshotNamePrefix as parameters for storageClass
this allows administrators to override the naming prefix for both volumes and snapshots
created by the rbd plugin.

Signed-off-by: Reinier Schoof <reinier@skoef.nl>
2020-02-25 05:03:51 +00:00
Madhu Rajanna
8163552b81 Add doc for rbd static pvc support
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
1ec2bc47e3 Add E2E for rbd static PVC
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
9f15dded6d Add support for static rbd volumes
currently its not possible to create
and mount static PVC to a application pod
using rbd csi driver. This PR adds the support
for static PVC in ceph-csi driver for rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
0f80ec0664 Return err from nodeserver
getVolumeNameByID function is already
returrning the status.Error return the err
as it is.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-18 12:34:46 +00:00
Madhu Rajanna
90235c32f8 Fix branch name in notes.txt
currently notes.txt container old branch name.
This PR updated the branch name with release

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-13 21:36:27 +00:00
Madhu Rajanna
d02dfe2dfe Remove unwanted RBAC rules from ceph-csi
There are currently unwanted RBAC permission
is given for ceph-csi, This PR reduces removes
such unwanted RBAC resources.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-13 21:36:27 +00:00
Madhu Rajanna
8dcb6a6105 Handle Delete operation if pool not found
If the backend rbd or cephfs pool is already deleted
we need to return success to the  DeleteVolume RPC
call to make it idempotent.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-11 15:58:15 +00:00
Madhu Rajanna
034b123478 Remove mount cache for cephfs
PR #282 introduces the mount cache to
solve cephfs fuse mount issue when cephfs plugin pod
restarts .This is not working as intended. This PR removes
the code for maintainability.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-02-11 15:11:21 +00:00
Vasyl Purchel
669dc4536f Reduce encryption KMS configuration SC parameters
* moves KMS type from StorageClass into KMS configuration itself
 * updates omapval used to identify KMS to only it's ID without the type

why?

1. when using multiple KMS configurations (not currently supported)
automated parsing of kms configuration will be failing because some
entries in configs won't comply with the requested type
2. less options are needed in the StorageClass and less data used to
identify the KMS

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
2020-02-10 15:21:11 +00:00
sophal
1695c6965d Add printing failed message when timeout
Signed-off-by: sophal <sophalHong@github.com>
2020-02-07 13:05:30 +00:00
Vasyl Purchel
419ad0dd8e Adds per volume encryption with Vault integration
- adds proposal document for PVC encryption from PR448
- adds per-volume encription by generating encryption passphrase
  for each volume and storing it in a KMS
- adds HashiCorp Vault integration as a KMS for encryption passphrases
- avoids encrypting volume second time if it was already encrypted but
  no file system created
- avoids unnecessary checks if volume is a mapped device when encryption
  was not requested
- prevents resizing encrypted volumes (it is not currently supported)
- prevents creating snapshots from encrypted volumes to prevent attack
  on encryption key (security guard until re-encryption of volumes
  implemented)

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com

Fixes #420
Fixes #744
2020-02-05 05:18:56 +00:00