Commit Graph

2273 Commits

Author SHA1 Message Date
Madhu Rajanna
385a751b8e rebase: rename kube-storage to csi-addons
as the org github.com/kube-storage is renamed
to github.com/csi-addons as the name kube-storage
was more generic.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-06 10:59:58 +00:00
Humble Chirammal
1b0ebc43d4 deploy: use serviceAccountName instead of serviceAccount in yamls
serviceAccount is the depricated alias for serviceAccountName, so it
is recommended/suggested to use serviceAccountName instead.

For ex. reference:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-04-06 09:00:35 +00:00
Niels de Vos
c4856195a2 doc: add description for Amazon KMS provider
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
f172e6956b doc: add configuration example for Amazon KMS
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
e1f8b1d44a doc: add example for csi-kms-connection-details ConfigMap
Updates: #1793
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
78cb7af46f rebase: vendor pkgs for Amazon KMS support
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
1c1683ba20 util: add AmazonMetadata KMS provider
The new Amazon Metadata KMS provider uses a CMK stored in AWS KMS to
encrypt/decrypt the DEK which is stored in the volume metadata.

Updates: #1921
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
f3b06d4c4a util: pass Namespace as part of KMSInitializerArgs
Amazon KMS expects a Secret with sensitive account and key information
in the Kubernetes Namespace where the Ceph-CSI Pods are running. It will
fetch the contents of the Secret itself.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Niels de Vos
523ac4b975 util: move getPodNamespace() and getKMSConfigMapName() into its own helpers
These functions can now be re-used easier. The Amazon KMS needs to know
the Namespace of the Pod for reading a Secret with more key/values.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-06 07:33:54 +00:00
Humble Chirammal
314fe0e23d cleanup: correct misspelling in rbd/clone.go
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-04-05 09:34:09 +00:00
Madhu Rajanna
448be70682 rbd: early check for disabled,disabling in DisableVolumeReplication
added early check for disabling and disabled
image mirroring state in DisableVolumeReplication

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-05 08:53:40 +00:00
Madhu Rajanna
fb3f7fe202 rbd: remove todo for image not found
Incase of resync the image will get deleted, gets
recreated and its a a time consuming operation.
It makes sense to return aborted error instead
of not found as we have omap data only the image
is missing in rbd pool.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-05 08:53:40 +00:00
Madhu Rajanna
95387c3b5e rbd: check for peer site status
Do resync if the image is in unknow or in error
state.

Check for the current image state for up+stopped
or up+replaying and also all peer site status
should be un up+stopped to confirm that resyncing
is done and image can be promoted and used.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-05 08:53:40 +00:00
Madhu Rajanna
233954bc10 rbd: make replication operations as rbdImage methods
added replication related operations as a method
of rbdImage as these methods can be easily used
when we introduce volumesnaphot mirroring operations.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-05 08:53:40 +00:00
Madhu Rajanna
c822ad460d rbd: add a check for image mirror disabling state
the rbd mirror state can be in enabled,disabled
or disabling state. If the mirroring is not disabled
yet and still in disabling state. we need to
check for it and return abort error message if
the mirroring is still getting disabled.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-05 08:53:40 +00:00
Madhu Rajanna
aaf6b571b8 rbd: Add ReplicationServer struct for replication operations
added ReplicationServer struct for the replication related
operation it also embed the ControllerServer which
already implements the helper functions like locking/unlocking etc.

removed getVolumeFromID and cleanup functions for better
code readability and easy maintaince.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-05 08:53:40 +00:00
Madhu Rajanna
ce7f936551 rebase: add replication-lib-utils to go.mod
added github.com/kube-storage/replication-lib-utils
to the vendor directory which is required to avoid
secret logging in GRPC.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-01 20:05:55 +00:00
Madhu Rajanna
da840a70c5 util: avoid secret logging in GRPC Replication Request
This commit uses the helper function to avoid the
logging of secrets in Replication GRPC request.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-01 20:05:55 +00:00
Yug
4f955e474d cleanup: correct misspelling
Correct snapshots spelling in rbd.go

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-04-01 12:00:21 +00:00
Madhu Rajanna
fba6a2d0c3 deploy: add csidriver object for cephfs and rbd
csidriver object can be created on the kubernetes
for below reason.

If a CSI driver creates a CSIDriver object,
Kubernetes users can easily discover the CSI
Drivers installed on their cluster
(simply by issuing kubectl get CSIDriver)

Ref: https://kubernetes-csi.github.io/docs/csi-driver-object.html#what-is-the-csidriver-object

attachRequired is always required to be set to
true to avoid issue on RWO PVC.

more details about it at https://github.com/rook/rook/pull/4332

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-31 13:41:35 +00:00
Humble Chirammal
d80b8d7fc5 doc: correct reference links in the snapshot,release,upgrade docs
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-03-31 06:58:29 +00:00
Niels de Vos
296f751d08 e2e: verify (non)existence of keys for VaultTokensKMS
Key existence and removal is only checked for the VaultKMS provider. It
should also be done for the VaultTokensKMS provider.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-30 08:29:33 +00:00
Niels de Vos
96fcc58095 util: use transformed Vault Tokens for initialization
After translating options from the ConfigMap into the common Vault
parameters, the generated configuration is not used. Instead, the
untranslated version of the configuration is passed on to the
vaultConnection initialization function, which then can detects missing
options.

By passing the right configuration to the initializatino function,
things work as intended.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-29 13:56:40 +00:00
Niels de Vos
d0f054bb6c util: use ConfigMap.Data instead of .BinaryData
When using .BinaryData, the contents of the configuration is not parsed
correctly. Whereas the parsing works fine whet .Data is used.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-29 13:56:40 +00:00
Rakshith R
9e957520ae e2e: add test case for app deletion with unmounted cephFS volume
Checks app deletion when cephFS volume is already unmounted.
Creates app, pvc and binds them. Unmounts the volume through
umount cmd in cephFS plugin and checks app deletion.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-03-25 15:15:07 +00:00
Rakshith R
e2dc8439e4 e2e: add execCommandInDaemonsetPod()
execCommandInDaemonsetPod() executes commands inside given
container of a daemonset pod on a particular node.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-03-25 15:15:07 +00:00
Rakshith R
915437b314 e2e: add listPods()
listPods returns slice of pods matching given ListOptions and
namespace.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-03-25 15:15:07 +00:00
Prasanna Kumar Kalever
e6a0049c0b e2e: add getDaemonSetLabelSelector() to utils
getDaemonSetLabelSelector returns labels of daemonset given name and
namespace dynamically, needed since labels are not same for helm and
non-helm deployments.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
Signed-off-by: Rakshith R <rar@redhat.com>
2021-03-25 15:15:07 +00:00
Rakshith R
26c07aef02 e2e: small fixes in var name and error msg
rookTolBoxPodLabel to rookToolBoxPodLabel
podPot to podOpt
fix error message for getting PV

Signed-off-by: Rakshith R <rar@redhat.com>
2021-03-25 15:15:07 +00:00
Rakshith R
721640178b cephfs: fix unmountVolume function
This commit fixes bug in unmount function which caused
unmountVolume to fail when targetPath was already unmounted.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-03-25 15:15:07 +00:00
Humble Chirammal
82bc993b32 util: remove unwanted import string from module dependencies
There is no need for an extra import string when the go mod package
itself declared in the same.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-03-24 15:12:13 +00:00
Niels de Vos
eea97ca014 util: move GetID() from EncryptionKMS to VolumeEncryption
There is no need for each EncryptionKMS to implement the same GetID()
function. We have a VolumeEncryption type that is more suitable for
keeping track of the KMS-ID that was used to get the configuration of
the KMS.

This does not change any metadata that is stored anywhere.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-24 12:09:04 +00:00
Niels de Vos
9317e2afb4 util: rewrite GetKMS() to use KMS provider plugin API
GetKMS() is the public API that initilizes the KMS providers on demand.
Each provider identifies itself with a KMS-Type, and adds its own
initialization function to a switch/case construct. This is not well
maintainable.

The new GetKMS() can be used the same way, but uses the new kmsManager
interface to create and configure the KMS provider instances.

All existing KMS providers are converted to use the new kmsManager
plugins API.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-24 12:09:04 +00:00
Niels de Vos
b43d28d35b util: add API for KMS Provider plugins
The KMSProvider struct is a simple, extendable type that can be used to
register KMS providers with an internal kmsManager.

Helper functions for creating and configuring KMS providers will also be
located in the new kms.go file. This makes things more modular and
better maintainable.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-24 12:09:04 +00:00
Madhu Rajanna
d8f7b38d3d rbd: add exclusive-lock and journaling image features for rbd image
Current rbd plugin only supports the layering feature
for rbd image. Add exclusive-lock and journaling image
features for the rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: woohhan <woohyung_han@tmax.co.kr>
2021-03-24 09:48:04 +00:00
Niels de Vos
2f6fca0862 build: regenerate scripts/golangci.yml if build.env was updated
In case build.env was updated, scripts/golangci.yml needs to be
regenerated. It contains a reference to the build-tag that is used to
identify the Ceph version to link against. Failing to update the
scripts/golangci.yml configuration, may cause running tests fail.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-17 13:38:46 +00:00
Yati Padia
0d9548c815 Cephfs: Failed to delete snapshot
Failed to delete voluesnapshot when backend subvolume
(pvc) and ceph fs subvolume snapshot is deleted

Fixes#1647

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-03-17 10:28:08 +00:00
Niels de Vos
bbd24e52f3 cleanup: use rbdImage.Destroy() for temporary volumes
rbdVolumes can have several resources that get allocated during its
usage. Only destroying the IOContext may not be suffiecient and can
cause resource leaks.

Use rbdVolume.Destroy() when the rbdVolume is not used anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-17 07:50:09 +00:00
Niels de Vos
5c26fbb0d7 util: use ClusterConnection.Copy() for re-using connections
Connections are reference counted, so just assigning the connection to
an other object for re-use is not correct. This can cause connections to
be garbage collected while something else is still using it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-17 07:50:09 +00:00
Madhu Rajanna
6e941539b5 rbd: implement volume replication spec
implemented the volume replication spec
for the rbd mirroring.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-16 13:06:44 +00:00
Madhu Rajanna
ee9a200fcc rbd: implement mirroring helpers with go-ceph
mirror.go exposes the helper functions to
perform the mirroring operations.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-16 13:06:44 +00:00
Madhu Rajanna
5e19803b96 build: update ceph tag in Makefile
As octopus supports the rbd mirroring,
updated ceph tag from nautilus to
octopus in Makefile.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-16 13:06:44 +00:00
Madhu Rajanna
342d282780 rebase: add kube-storage/replication to go.mod
add dependent kube-storage/replication package
to the vendor.

update grpc to latest release v1.35.0.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-16 13:06:44 +00:00
Madhu Rajanna
c642637cec util: register replication controller
as RBD is implementing the replication
we are registering it. For CephFS, its
not implementing the replication we are
passing nil so we dont want to register
it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-16 13:06:44 +00:00
Madhu Rajanna
ee0576278f cleanup: move servers to a new struct
For future enhancments like adding more
servers. Moving the list of servers to a
new structure.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-03-16 13:06:44 +00:00
Yati Padia
cfb6abc067 e2e: update snapshot restore e2e to check data consistency
Currently, in rbd snapshot restore and volume clone E2E we
are not checking any data consistency after doing snapshot
restore or volume clone. Hence, this PR writes the data in
the PVC and checks the checksum of the file and verify it with
the snapshot or cloned PVC.

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-03-16 09:36:46 +00:00
Prasanna Kumar Kalever
bad7d56ef4 doc: add snapshot API version support matrix
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-03-15 15:11:01 +00:00
Prasanna Kumar Kalever
b9291c74f0 doc: adjust SNAPSHOT_VERSION
Fixes: #1803
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-03-15 15:11:01 +00:00
Prasanna Kumar Kalever
d92e084dc1 deploy: bump up the API version from v1beta1 to v1
With v4.0.0 release of external-snapshotter, we are moving towards v1
from v1beta1 API version

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-03-15 15:11:01 +00:00
Prasanna Kumar Kalever
ea4489da68 deploy: bump the snapshotter sidecar image version to v4.0.0
use the latest version of csi-snapshotter sidecar image at the
provisioner templates

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-03-15 15:11:01 +00:00