Commit Graph

106 Commits

Author SHA1 Message Date
Niels de Vos
a5211dcf0e util: allow configuring VAULT_BACKEND for Vault connection
It seems that the version of the key/value engine can not always be
detected for Hashicorp Vault. In certain cases, it is required to
configure the `VAULT_BACKEND` (or `vaultBackend`) option so that a
successful connection to the service can be made.

The `kv-v2` is the current default for development deployments of
Hashicorp Vault (what we use for automated testing). Production
deployments default to version 1 for now.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 82557e3f34)
2021-08-10 10:08:16 +00:00
Niels de Vos
c61e6b3f8c e2e: disable iss validation in Hashicorp Vault
Testing encrypted PVCs does not work anymore since Kubernetes v1.21. It
seems that disabling the iss validation in Hashicorp Vault is a
relatively simple workaround that we can use instead of the more complex
securing of the environment like should be done in production
deployments.

Updates: #1963
See-also: external-secrets/kubernetes-external-secrets#721
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit fd9fee74de)
2021-06-29 11:43:11 +00:00
Humble Chirammal
1b0ebc43d4 deploy: use serviceAccountName instead of serviceAccount in yamls
serviceAccount is the depricated alias for serviceAccountName, so it
is recommended/suggested to use serviceAccountName instead.

For ex. reference:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-04-06 09:00:35 +00:00
Niels de Vos
f172e6956b doc: add configuration example for Amazon KMS
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
e1f8b1d44a doc: add example for csi-kms-connection-details ConfigMap
Updates: #1793
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Madhu Rajanna
d8f7b38d3d rbd: add exclusive-lock and journaling image features for rbd image
Current rbd plugin only supports the layering feature
for rbd image. Add exclusive-lock and journaling image
features for the rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: woohhan <woohyung_han@tmax.co.kr>
2021-03-24 09:48:04 +00:00
Prasanna Kumar Kalever
d92e084dc1 deploy: bump up the API version from v1beta1 to v1
With v4.0.0 release of external-snapshotter, we are moving towards v1
from v1beta1 API version

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-03-15 15:11:01 +00:00
Niels de Vos
c5f00a9e3d e2e: add test for the SecretsMetadataKMS provider
The new provider should be able to provision and mount an encrypted
volume.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-12 10:11:47 +00:00
Madhu Rajanna
c417a5d0ba rbd: add support for thick provisioning option
Add an option to the StorageClass to support creating fully allocated
(thick provisioned) RBD images

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-02-19 11:55:40 +00:00
liulei3
94fc931d44 doc: remove redundant word
Signed-off-by: liulei3 <liulei3@360.cn>
2021-01-25 09:57:12 +00:00
Niels de Vos
e4b16a5c72 util: allow tenants to (re)configure VaultTokens settings
A tenant can place a ConfigMap in their Kubernetes Namespace with
configuration options that differ from the global (by the Storage Admin
set) values.

The ConfigMap needs to be located in the Tenants namespace, as described
in the documentation

See-also: docs/design/proposals/encryption-with-vault-tokens.md
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-12-16 13:42:52 +00:00
Niels de Vos
6e2d16ed1e e2e: add test for VaultTokensKMS support
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-12-14 14:45:09 +00:00
Niels de Vos
b0f3b27209 ci: set imagePullPolicy for Vault to IfNotPresent
Deploying Vault still fails on occasion. It seems that the
imagePullPolicy has not been configured for the container yet.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-12-03 13:17:24 +00:00
Niels de Vos
1845f2b77d e2e: use full-qualified-image-name for vault-init-job
On occasion deploying Vault fails. It seems the vault-init-job batch job
does not use a full-qualified-image-name for the "vault" container.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-30 12:27:00 +00:00
Niels de Vos
db6d376434 deploy: add sys/mounts to Vault policy
Add "sys/mounts" so that VaultBackendKey does not need to be set. The
libopenstorage API detects the version for the key-value store in Vault
by reading "sys/mounts". Without permissions to read this endpoint, the
VaultBackendKey is required to be configured.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
04586dc733 deploy: add "list" operation to Vault policy
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
1f18e876f0 e2e: use docker.io/library as prefix for official images
Docker Hub offers a way to pull official images without any project
prefix, like "docker.io/vault:latest". This does a redirect to the
images located under "docker.io/library".

By using the full qualified image name, a redirect gets removed while
pulling the images. This reduces the likelyhood of hittin Docker Hub
pull rate-limits.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-26 13:51:02 +00:00
Niels de Vos
eaeee8ac3d deploy: use docker.io for unqualified image names
Images that have an unqualified name (no explicit registry) come from
Docker Hub. This can be made explicit by adding docker.io as prefix. In
addition, the default :latest tag has been added too.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 10:27:33 +00:00
Niels de Vos
4fd0294eb7 e2e: pull centos image from registry.centos.org
The BlockVolume PVC tests consume the example files that refer to
"centos:latest" without registry. This means that the images will get
pulled from Docker Hub, which has rate limits preventing CI jobs from
pulling images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-19 16:00:33 +00:00
Madhu Rajanna
7d229c2369 build: update imagepullpolicy for vault
this allows the image to be reused instead of pulling
it again.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-19 16:00:33 +00:00
Prasanna Kumar Kalever
2945f7b669 cleanup: stick to standards when using dollar-sign in md
MD014 - Dollar signs used before commands without showing output
The dollar signs are unnecessary, it is easier to copy and paste and
less noisy if the dollar signs are omitted. Especially when the
command doesn't list the output, but if the command follows output
we can use `$ ` (dollar+space) mainly to differentiate between
command and its ouput.

scenario 1: when command doesn't follow output
```console
cd ~/work
```

scenario 2: when command follow output (use dollar+space)
```console
$ ls ~/work
file1 file2 dir1 dir2 ...
```

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-11-11 13:18:05 +00:00
Humble Chirammal
4f9823efbb e2e: add block mode PVC clone file
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-10-15 15:37:14 +00:00
Humble Chirammal
6359e37a23 e2e: add pod yaml which consume block mode clone pvc
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-10-15 15:37:14 +00:00
Prasanna Kumar Kalever
d2f4666ebe deploy: group fields with optional/required labels
This patch with touch on the varuious other fields with in the storage class
yamls and label them with optional/required.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-10-07 13:01:45 +00:00
Prasanna Kumar Kalever
0c6d8fbd80 deploy: let the pool/fs name be required
Obviously expecting a pool with name `rbd` or CephFS name `myfs`
will be a limitation, as the pool/fs is created by admin manually,
let them choose the name that suits their requirement and come back
edit it in the storage class.

Making the pool/fs name as required field will give more attention,
else with new users it will be mostly left unedited until one hit
the errors saying no pool/fs exists.

This patch clips-off the default pool/fs name and make it a mandatory
field.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-10-07 13:01:45 +00:00
Jeremy Facchetti
b9dd54e238 doc: added precisions about snapshotting and cloning
Signed-off-by: Jeremy Facchetti <facchettos@gmail.com>
2020-09-29 12:48:30 +00:00
Madhu Rajanna
d1f175d9f3 rbd: add support for rbd map and unmap options
added support for providing map and unmap
options to rbd CLI when mapping rbd image
on the node.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-09-21 13:27:28 +00:00
Humble Chirammal
2b180d89c5 cephfs: change the volumetype to RWX instead of RWO
The intention here is to keep the example YAMLs of CephFS
with recommended Access Mode of CephFS which is RWX instead of RWO.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-13 03:37:28 +00:00
Mehdy Khoshnoody
3081eabba5 doc: Add rados namespace to option to csi-configs
A minimal documentation on how to use a rados namespace with rbd.

Signed-off-by: Mehdy Khoshnoody <mehdy.khoshnoody@gmail.com>
2020-08-12 16:22:58 +05:30
Humble Chirammal
c85d03c79e e2e: add/modify deployment files for cephfs snapshot/clone tests
Add snapshot class, snapshot, restore, clone PVC and pod yamls
for e2e test to consume

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Yug
41be151ac6 deploy: remove vault along with driver
The created vault should also be
brought down with the driver.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-04 16:00:21 +00:00
Yug
2e14116ed7 deploy: add vault creation to rbd driver deployment
Currently, the script does not deploy the driver singlehandedly;
As the vault creation needs to be done prior to that.
The script now includes the vault creation so that
one script can be sufficient to deploy the rbd driver.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-04 16:00:21 +00:00
Madhu Rajanna
a0fd805a8b rbd: Add support for smart cloning
Added support for RBD PVC to PVC cloning, below
commands are executed to create a PVC-PVC clone from
RBD side.

* Check the depth(n) of the cloned image if n>=(hard limit -2)
or ((soft limit-2) Add a task to flatten the image and return
about (to avoid image leak) **Note** will try to flatten the
temp clone image in the chain if available
* Reserve the key and values in omap (this will help us to
avoid the leak as it's not reserved earlier as we have returned
ABORT (the request may not come back))
* Create a snapshot of rbd image
* Clone the snapshot (temp clone)
* Delete the snapshot
* Snapshot the temp clone
* Clone the snapshot (final clone)
* Delete the snapshot

```bash
1) check the image depth of the parent image if flatten required
add a task to flatten image and return ABORT to avoid leak
(hardlimit-2 and softlimit-2 check will be done)
2) Reserve omap keys
2) rbd snap create <RBD image for src k8s volume>@<random snap name>
3) rbd clone --rbd-default-clone-format 2 --image-feature
layering,deep-flatten <RBD image for src k8s volume>@<random snap>
<RBD image for temporary snap image>
4) rbd snap rm <RBD image for src k8s volume>@<random snap name>
5) rbd snap create <cloned RBD image created in snapshot process>@<random snap name>
6) rbd clone --rbd-default-clone-format 2 --image-feature <k8s dst vol config>
 <RBD image for temporary snap image>@<random snap name> <RBD image for k8s dst vol>
7)rbd snap rm <RBD image for src k8s volume>@<random snap name>
```

* Delete temporary clone image created as part of clone(delete if present)
* Delete rbd image

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 14:02:12 +00:00
Humble Chirammal
c1a269191c doc: make sure configmap object referred or documented correctly
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-06-10 11:33:10 +00:00
Yug Gupta
6cf7389d97 doc: Add prefix for naming subvolumes and snapshots
Updated storageclass and snapshotclass
to include the name prefix for naming
subvolumes and snapshots.

Fixes: #1087

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-27 05:54:10 +00:00
Madhu Rajanna
a116764f4d Snapshot: Template changes for snapshot beta
Updated the deployment templates and the sidecar
images version to support snapshot beta version.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Mathias Merscher
0991cdf498 make CephFS SubvolumeGroup configurable
The name of the CephFS SubvolumeGroup for the CSI volumes was hardcoded to "csi". To make permission management in multi tenancy environments easier, this commit makes it possible to configure the CSI SubvolumeGroup.

related to #798 and #931
2020-05-04 05:50:06 +00:00
ShyamsundarR
1a8f8e3c24 Add support for erasure coded pools
This commit adds support to mention dataPool parameter for the
topology constrained pools in the StorageClass, that can be
leveraged to mention erasure coded pool names to use for RBD
data instead of the replica pools.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
5c4abf8347 Add topology support to ceph-csi
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
Madhu Rajanna
37c4e3447d Add helm chart SA to vault.yaml
we need to provide access to the Service
account created with helm charts to access
the vault service.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
bcd646ee55 Deprecate grpc metrics in ceph-csi
As kubernetes CSI sidecar is exposing the
GRPC mertics we can make use of the same in
ceph-csi we dont need to expose our own.

update: #881

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-01 11:59:37 +00:00
Niels de Vos
825448825c examples: rbd/storageclass.yaml causes Python excetion with yamllint
While running the 'make test' target and have 'yamllint' available, the
test fails with the following exception:

yamllint -s -d {extends: default, rules: {line-length: {allow-non-breakable-inline-mappings: true}},ignore: charts/*/templates/*.yaml} ./examples/rbd/storageclass.yaml
Traceback (most recent call last):
  File "/usr/local/bin/yamllint", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.6/site-packages/yamllint/cli.py", line 181, in run
    problems = linter.run(f, conf, filepath)
  File "/usr/local/lib/python3.6/site-packages/yamllint/linter.py", line 237, in run
    content = input.read()
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1947: ordinal not in range(128)

The quotes used in the comments seem to be non-ascii characters.
Replacing these with standard " makes the test pass again.

This problem occurred while running tests in a container based on the
Ceph image (CentOS-7) with Python 3. Travis CI might still use Python 2
for yamllint, and hide the problem.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 14:42:25 +00:00
Vasyl Purchel
669dc4536f Reduce encryption KMS configuration SC parameters
* moves KMS type from StorageClass into KMS configuration itself
 * updates omapval used to identify KMS to only it's ID without the type

why?

1. when using multiple KMS configurations (not currently supported)
automated parsing of kms configuration will be failing because some
entries in configs won't comply with the requested type
2. less options are needed in the StorageClass and less data used to
identify the KMS

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
2020-02-10 15:21:11 +00:00
Vasyl Purchel
419ad0dd8e Adds per volume encryption with Vault integration
- adds proposal document for PVC encryption from PR448
- adds per-volume encription by generating encryption passphrase
  for each volume and storing it in a KMS
- adds HashiCorp Vault integration as a KMS for encryption passphrases
- avoids encrypting volume second time if it was already encrypted but
  no file system created
- avoids unnecessary checks if volume is a mapped device when encryption
  was not requested
- prevents resizing encrypted volumes (it is not currently supported)
- prevents creating snapshots from encrypted volumes to prevent attack
  on encryption key (security guard until re-encryption of volumes
  implemented)

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com

Fixes #420
Fixes #744
2020-02-05 05:18:56 +00:00
Niels de Vos
7ec8e68b67 examples/rbd-block: use a container image that is supported
Fedora 26 has been End-Of-Life for a long time already, is should not be
used anymore. Instead use the latest CentOS image that gets regular
updates. The Ceph base image used by Ceph-CSI is also based on CentOS,
so uses would be familiar with the available tools.

Also use "sleep infinity" instead of a weird 'tail -f /dev/null' hack.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-01-21 13:30:45 +00:00
wilmardo
f04af5742d refact: Remove Kubernetes 1.13.x support
Signed-off-by: wilmardo <info@wilmardenouden.nl>
2020-01-20 10:32:30 +00:00
Vasyl Purchel
166eaf700f Adds PVC encryption with LUKS
Adds encryption in StorageClass as a parameter. Encryption passphrase is
stored in kubernetes secrets per StorageClass. Implements rbd volume
encryption relying on dm-crypt and cryptsetup using LUKS extension

The change is related to proposal made earlier. This is a first part of
the full feature that adds encryption with passphrase stored in secrets.

Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.com
Signed-off-by: Ioannis Papaioannou ioannis.papaioannou@workday.com
Signed-off-by: Paul Mc Auley paul.mcauley@workday.com
Signed-off-by: Sergio de Carvalho sergio.carvalho@workday.com
2019-12-16 08:12:44 +00:00
Humble Chirammal
8e437244de Enable RBD resizer in travis
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-12-13 12:40:12 +00:00
Humble Chirammal
ac09c5553c Add E2E for cephfs resize functionality
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-11-27 14:00:47 +00:00
Humble Chirammal
b44a81bdbc We were defaulting to ext4 at first and then moved toxfs.
However further testing shows that, ext4 should be
the default or preferred fs for RBD devices.

This patch bring that change

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-09-24 14:36:50 +05:30