Commit Graph

2453 Commits

Author SHA1 Message Date
Yati Padia
299979fc14 ci: add unit test for toError()
This commit adds unit test for the
func converting cephFSCloneState to error.

Fixes: #2259

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-14 15:02:12 +00:00
Yati Padia
c66872c3c6 cleanup: ineffective assignment
This commit resolves ineffective assignent of
snap.

Updates: #1586

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-14 12:39:17 +00:00
Niels de Vos
4d4a2a7814 e2e: prevent re-using empty pvc object
When an error occurs, the pvc object is overwritten in the
PollImmediate() loop. Re-using the pvc.Namespace results in error
messages like

    Error getting pvc in namespace: '': an empty namespace may not be set when a resource name is provided

and prevents the retry by PollImmediate() to never succeed. Storing the
namespace in a local variable prevents this from happening.

Reported-by: Rakshith R <rar@redhat.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-14 10:18:51 +00:00
Niels de Vos
f7ae33c67c e2e: only call error check functions when err != nil
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-14 10:18:51 +00:00
Niels de Vos
075a4087d7 e2e: mark "etcdserver: request timed out" errors as retryable
There are regular CI failures where etcdserver times out. These errors
seem not to get caught by any of the existing error comparing. Matching
the error by string should prevent temporary etcdserver issues now too.

Updates: #2218
Closes: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-14 10:18:51 +00:00
Yati Padia
f210d5758b cleanup: spell check getImageMirroingStatus
This commit corrects the spelling for
getImageMirroingStatus() -> getImageMirroringStatus

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-14 07:32:01 +00:00
Niels de Vos
d941e5abac util: make parseTenantConfig() usable for modular KMSs
parseTenantConfig() only allowed configuring a defined set of options,
and KMSs were not able to re-use the implementation. Now, the function
parses the ConfigMap from the Tenants Namespace and returns a map with
options that the KMS supports.

The map that parseTenantConfig() returns can be inspected by the KMS,
and applied to the vaultTenantConnection type by calling parseConfig().

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
96bb8bfd0e e2e: add securityContext.runAsUser to vault-init-job
Kubelet sometimes reports the following error:

    failed to "StartContainer" for "vault-init-job" with CreateContainerConfigError: container has runAsNonRoot and image will run as root

Setting securityContext.runAsUser resolves this.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
e3c7dea7d6 e2e: add test for Vault with ServiceAccount per Tenant
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
b700fa43e6 doc: add example for Tenant ServiceAccount
The ServiceAccount "ceph-csi-vault-sa" is expected to be placed in the
Namespace "tenant" so that the provisioner and node-plugin fetch the
ServiceAccount from a Namespace where Ceph-CSI is not deployed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
8662e01d2c deploy: allow RBD components to get ServiceAccounts
The provisioner and node-plugin have the capability to connect to
Hashicorp Vault with a ServiceAccount from the Namespace where the PVC
is created. This requires permissions to read the contents of the
ServiceAccount from an other Namespace than where Ceph-CSI is deployed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
3d7d48a4aa util: VaultTenantSA KMS implementation
This new KMS uses a Kubernetes ServiceAccount from a Tenant (Namespace)
to connect to Hashicorp Vault. The provisioner and node-plugin will
check for the configured ServiceAccount and use the token that is
located in one of the linked Secrets. Subsequently the Vault connection
is configured to use the Kubernetes token from the Tenant.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
6dc5bf2b29 util: split vaultTenantConnection from VaultTokensKMS
This makes the Tenant configuration for Hashicorp Vault KMS connections
more modular. Additional KMS implementations that use Hashicorp Vault
with per-Tenant options can re-use the new vaultTenantConnection.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 17:16:35 +00:00
Niels de Vos
ed298341a6 doc: proposal for KMS with ServiceAccount per Tenant
A new KMS that supports Hashicorp Vault with the Kubernetes Auth backend
and ServiceAccounts per Tenant (Kubernetes Namespace).

Updates: #2222
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-13 12:12:25 +00:00
Yati Padia
69c9e5ffb1 cleanup: resolve parallel test issue
This commit resolves parallel test issues
and also excludes internal/util/conn_pool_test.go
as those test can't run in parallel.

Updates: #1586

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-13 11:31:39 +00:00
Prasanna Kumar Kalever
8b3136e696 doc: add documentaion for rbd-nbd mounter
Closes #2124

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-07-13 10:19:17 +00:00
Yati Padia
4a649fe17f cleanup: resolve godot linter
This commit resolves godot linter issue
which says "Comment should end in a period (godot)".

Updates: #1586

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-13 06:50:03 +00:00
Niels de Vos
855f9080d1 e2e: refactor KMS configuration and usage
This adds a new `kmsConfig` interface that can be used to validate
different KMS services and setting. It makes checking for the available
support easier, and fetching the passphrase simpler.

The basicKMS mirrors the current validation of the KMS implementations
that use secrets and metadata. vaultKMS can be used to validate the
passphrase stored in a Vault service.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-12 14:57:39 +00:00
Yati Padia
42f8d8d446 doc: Updates the helm doc with secret and sc configuration
This commit updates the helm doc with
storageclass and secret configurations.

Fixes: #2255

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-12 12:03:39 +00:00
Yati Padia
f35ce3d880 cleanup: Adds t.Helper() to test helper function
This commit adds t.Helper() to the test helper
function. With this call go test prints correct
lines of code for failed tests. Otherwise,
printed lines will be inside helpers functions.
For more details check: https://github.com/kulti/thelper

Updates: #1586

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-12 11:25:55 +00:00
Yati Padia
84c1fe52c7 cleanup: resolve exhaustive linter
This commit resolves exhaustive linter
error.

Updates: #2240

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-12 04:47:08 +00:00
Jonas Zeiger
680a7bf411 util: more generic kernel version parsing
* Make kernel version parsing to support more (valid) version strings

  * Put version string parsing into a separate, testable function

  * Fixes #2248 (Kernel Subversion Parsing Failure)

Signed-off-by: Jonas Zeiger <jonas.zeiger@talpidae.net>
2021-07-09 07:36:27 +00:00
Rakshith R
b27d6319ca e2e: add e2e for user secret based metadata encryption
This commit adds e2e for user secret based metadata encryption,
adds user-secret.yaml and makes required changes in kms-connection-details,
kms-config yamls.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-07-08 17:06:02 +00:00
Rakshith R
3352d4aabd rbd: add user secret based metadata encryption
This commit adds capability to `metadata` encryption
to be able to fetch `encryptionPassphrase` from user
specified secret name and namespace(if not specified,
will default to namespace where PVC was created).

This behavior is followed if `secretName` key is found
in the encryption configuration else defaults to fetching
`encryptionPassphrase` from storageclass secrets.

Closes: 2107

Signed-off-by: Rakshith R <rar@redhat.com>
2021-07-08 17:06:02 +00:00
Rakshith R
c4060b8aa2 doc: add info about metadata rbd volume encryption
Signed-off-by: Rakshith R <rar@redhat.com>
2021-07-08 17:06:02 +00:00
Yati Padia
853d77d793 doc: update helm chart documentations
This commit updates the helm chart documentations
with the configurations available while deploying
these helm charts.

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-08 09:44:20 +00:00
Yati Padia
ffab37f44f cleanup: resolves gocritic linter issues
This commit resolves gocritic linter errors.

Updates: #2250

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-07-08 05:19:26 +00:00
Madhu Rajanna
dd0884310f rbd: set image metadata in isThickProvisioned
setting metadata in isThickProvisioned method
helps us to avoid checking thick metakey and
deprecated metakey for both thick and thin
provisioned images and also this will easily
help us to migrated the deprecated key to new key.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-07 08:31:10 +00:00
Madhu Rajanna
77135599ac rbd: make setThickProvisioned as method of rbdImage
isThickProvisioned is already method of the rbdImage
to keep similar thick provisioner related functions
common making isThickProvisioned as method of rbdImage.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-07 08:31:10 +00:00
Madhu Rajanna
708800ddc1 rbd: set thick metadata if ThickProvision is set
instead of checking the parent is thick provisioned
or not we can decide based on the rbdVol generated
from the request. If the request is to create a Thick
Image. set metadata without checking the parent.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-07 08:31:10 +00:00
Madhu Rajanna
f471a206c5 e2e: update e2e to validate thick metadata
updated e2e tests to validate the thick
metadata set on the rbd image for thick PVC.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-07 08:31:10 +00:00
Madhu Rajanna
332a47a100 rbd: deprecate .rbd.csi.ceph.com/thick-provisioned metadata key
As image metadata key starting with '.rbd' will not
be copied when we do clone or mirroring, deprecating
the old key for the same reason use
'csi.ceph.com/thick-provisioned' to set image metadata.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-07 08:31:10 +00:00
Madhu Rajanna
0837c05be0 rbd: set scheduling interval on snapshot mirrored image
Mirror-snapshots can also be automatically created on a
periodic basis if mirror-snapshot schedules are defined.
The mirror-snapshot can be scheduled globally, per-pool,
or per-image levels. Multiple mirror-snapshot schedules
can be defined at any level.

To create a mirror-snapshot schedule with rbd, specify
the mirror snapshot schedule add command along with an
optional pool or image name; interval; and optional start time:

The interval can be specified in days, hours, or minutes
using d, h, m suffix respectively. The optional start-time
can be specified using the ISO 8601 time format. For example:

```
$ rbd --cluster site-a mirror snapshot schedule
  add --pool image-pool --image image1 24h 14:00:00-05:00
```

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-06 14:41:48 +00:00
Madhu Rajanna
b1710f4c53 util: add method to get rados connection
New go-ceph admin package api's expects to
pass the rados connection as argument.
added new method called GetRBDAdmin
to get admin connection to administrate
rbd volumes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-06 14:41:48 +00:00
Madhu Rajanna
8732bec369 rebase: add go-ceph rbd admin package to vendor
added go-ceph rbd admin package to vendor. keeping
this as a separate commit which helps in review.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-07-06 14:41:48 +00:00
Yug
e14d649547 e2e: add cephfs test for sc, secret via helm
add cephfs test for validating storageclass and
secret deployment via helm.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
7d28896e36 e2e: add rbd test for testing sc, secret via helm
add rbd test for validating storageclass and
secret deployment via helm.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
d865670f97 deploy: pass --namespace flag before passing namespace
Since we parse multiple options from command line in install-helm,
script will not recognize any random string directly as namespace.
To inform script about namespace, `--namespace` flag is used before
passing namespace.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
69d1beeb07 deploy: add support to deploy storageclass and secret
Update install-helm script to accept `--deploy-sc` and
`--deploy-secret` as optional parameters when deploying
ceph-csi via helm charts.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
353fa5e65c helm: Update values for storage class and secret
Add values for storage class and secret templates for
ceph-csi-cephfs helm chart.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
a55d5efade helm: Add secret template to ceph-csi-cephfs
Deploy cephfs secret when using the ceph-csi-cephfs
helm chart.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
eddcf6b6d4 helm: Add storage class template to ceph-csi-cephfs
Deploy cephfs storage class when using helm chart.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
4c16e086e6 helm: Update values for storage class and secret
Add default values for storage class and secret
templates for helm installation.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
230f0d3f22 helm: Add csi secret template to ceph-csi-rbd
Deploy csi secret when using the helm chart.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Yug
ecf768ec41 helm: Add storage class template to ceph-csi-rbd
Deploy storage class when using the helm chart.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 10:55:41 +00:00
Rakshith R
fb365005da e2e: update e2epv.WaitOnPVandPVC() to match new parameters
Added &framework.TimeoutContext{ClaimBound: timeout, PVBound: timeout}
to parameters of function call in updated package.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-07-01 03:35:23 +00:00
Rakshith R
9eaa55506f rebase: update controller-runtime package to v0.9.2
This commit updates controller-runtime to v0.9.2 and
makes changes in persistentvolume.go to add context to
various functions and function calls made here instead of
context.TODO().

Signed-off-by: Rakshith R <rar@redhat.com>
2021-07-01 03:35:23 +00:00
Rakshith R
1b23d78113 rebase: update kubernetes to v1.21.2
Updated kubernetes packages to latest release.
resizefs package has been included into k8s.io/mount-utils
package. updated code to use the same.

Updates: #1968

Signed-off-by: Rakshith R <rar@redhat.com>
2021-07-01 03:35:23 +00:00
Niels de Vos
8ce5ae16c1 e2e: do not create a single-item list
The deployment of the Vault ConfigMap for the init-scripts job contains
a List with a single Item. This can be cleaned up to just be a ConfigMap
(without the list structure around it).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-30 16:22:32 +00:00
Niels de Vos
75f385e881 ci: require CI jobs with Kubernetes 1.21 to pass
Closes: #1963
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-30 19:10:16 +05:30