Commit Graph

3831 Commits

Author SHA1 Message Date
Humble Chirammal
0166817de4 cleanup: correct typo in travis scripts
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-04-21 09:22:30 +00:00
Madhu Rajanna
029b5004aa doc: update upgrade doc for v3.3.0
As we have v3.3 as the latest release
updating the upgrade doc in the devel
branch to point to the same.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-21 06:39:07 +00:00
Madhu Rajanna
52290333e6 rbd: modified logic to check image watchers
Before RBD map operation, we do check the
watchers on the RBD image. In the case of
RWO volume. cephcsi makes sure only one
client is using the RBD image. If the rbd
image is mirrored, by default mirroring
daemon will add a watcher on the image
and as we are using go-ceph a watcher will
be added as we have opened the image So
we will have two watchers on an image if
mirroring is enabled. This holds when the
rbd mirror daemon is running, In case if
the mirror daemon is not running there will
be only one watcher on the rbd image
(which is placed by go-ceph image open)
we should not block the map operation if
the mirroring daemon is not running as
its Async mirroring. This commit adds a
check to make sure no more than 2 watchers
if the image is mirrored or no more than 1
watcher if it is not mirrored image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-19 16:30:55 +00:00
Niels de Vos
27247d1444 e2e: remove losetup workaround for minikube
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-19 13:30:57 +00:00
Niels de Vos
e5787f24b0 rebase: update minikube to version 1.19.0
See-also: https://github.com/kubernetes/minikube/releases/tag/v1.19.0
Fixes: #1972
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-19 11:15:57 +00:00
Yug
6a46f381c2 cleanup: update description to generic
Since rbdImage is a common struct for
rbdVolume and rbdSnapshot, it description
was matching to only snapshot.
This commit makes the comments generic for
both volumes and snapshots.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-04-19 07:32:35 +00:00
Rakshith R
9f2cf498b6 cephfs: enable ceph-fuse big_writes by default
By default, the write buffer size in libfuse2 is 2KiB
`fuse_big_writes = true` option is used to override this limit.
This commit makes `fuse_big_writes = true` option as default
in ceph.conf.

Closes: #1928

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-19 07:08:57 +00:00
Humble Chirammal
54845b63c0 cleanup: better or corrected variable name in grpc prometheous code
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-04-16 10:22:35 +00:00
Humble Chirammal
0fae0e53b6 cleanup: various source code comment corrections
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-04-16 10:22:35 +00:00
Madhu Rajanna
b0c9b3e752 doc: rename master to devel
renamed master to devel in image
compatibility list.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-16 06:10:46 +00:00
Madhu Rajanna
6c21be278a doc: update readme for v3.3.0 release
updated readme with available released
image tags.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-16 06:10:46 +00:00
Madhu Rajanna
a3b3858a97 ci: add mergify rules for release 3.3
as we have a new release-v3.3 branch. adding
the mergify rules for auto merge and auto backport
based on backport-to-release-v3.3 label.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-15 17:39:49 +05:30
Madhu Rajanna
d94a7ca7e1 revert: cleanup: update mergify.yml to use merge_bot_account option
This reverts commit 31634ede3d.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-15 17:34:08 +05:30
Madhu Rajanna
eea52847bc rbd: check volumeID in PV if image not found
If the pool or few keys are missing in the omap.
GetImageAttributes function returns nil error message and few
empty items in imageAttributes struct. if the image is not
found and  the entiries are missing use
the volumeId present on the PV annotation for further operations.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-15 17:13:06 +05:30
Madhu Rajanna
cfc88c9910 rbd: discard up+unknown state in ResyncVolume
incase if the image is promoted and demoted the
image state will be set to up+unknown if the image
on the remote cluster is still in demoted state.

when user changes the state from primary to secondary
and still the image is in demoted (secondary) state
in the remote cluster. the image state on both the cluster
will be on unknown state.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-15 17:13:06 +05:30
Rakshith R
31634ede3d cleanup: update mergify.yml to use merge_bot_account option
New version of mergifyio requires the use `merge_bot_account`
instead of `bot_accout` configuration option.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-15 12:00:45 +05:30
Rakshith R
3795704340 ci: update feature gates setting from minikube.sh
BlockVolume, CSIBlockVolume(GA since k8s v1.18) & VolumeSnapshotDataSource
(GA since k8s v1.20) default to true and don't need to be set to true in
feature gates setting.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-15 05:27:16 +00:00
Niels de Vos
8b8480017b logging: report issues in rbdImage.DEKStore API with stacks
It helps to get a stack trace when debugging issues. Certain things are
considered bugs in the code (like missing attributes in a struct), and
might cause a panic in certain occasions.

In this case, a missing string will not panic, but the behaviour will
also not be correct (DEKs getting encrypted, but unable to decrypt).
Clearly logging this as a BUG is probably better than calling panic().

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
35d58a7d5a e2e: only test a single encrypted clone/snapshot
The default number for cloning and snapshot/restore is 10 volumes. This
adds to the time the test suite runs. There is no need to validate 10
copies of the encrypted volume, a single copy is sufficient.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
bb296c4f21 e2e: add verification for encrypted Snapshot/Restore operations
This moves validatePVCSnapshot() into its own function, so that it
follows the same format as validatePVCClone() does already.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
3fde636685 e2e: add validation for cloning encrypted volumes
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
b1d05a1840 rbd: repair encryption config in case it is missing
It is possible that when a provisioner restarts after a snapshot was
cloned, but before the newly restored image had its encryption metadata
set, the new image is not marked as encrypted. This will prevent
attaching/mounting the image, as the encryption key will not be fetched,
or is not available in the DEKStore.

By actively repairing the encryption configuration when needed, this
problem should be addressed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
1482105309 cleanup: use buildCreateVolumeResponse() to simplify CreateVolume()
buildCreateVolumeResponse() exists exactly for the need to create a
csi.CreateVolumeResponse based on an rbdVolume. Calling this helper
reduces the code duplication in CreateVolume().

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
52433841b4 cleanup: move copyEncryptionConfig() from CreateVolume to Exists()
The rbdVolume that needs its encryption configured is constructed in the
Exists() method. It is suitable to move the copyEncryptionConfig() call
there as well, so that the object is completely constructed in a single
place.

Golang-ci:gocyclo complained about the increased complexity of the
Exists() function. Moving the repairing of the ImageID into its own
helper function makes the code a little easier to understand.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
596410ae60 cleanup: address "nolint" comments for RBD CreateSnapshot
Introduce helper function cloneFromSnapshot() that takes care of the
procedures that are needed when an existing snapshot has been found.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
b5d0524c39 cleanup: release resources for rbdImages objects after use
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
dc990037a5 rbd: move setupEncryption() from buildCreateVolumeResponse to CreateVolume
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
bea9d56117 rbd: copyEncryptionConfig in doSnapshotClone()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
fd5f4dbafd rbd: configureEncryption() in genSnapFromSnapID()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
6fd3f57f40 rbd: set kmsID in reserveSnap()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
0a046c5b6d rbd: copy encryption configuration in CreateSnapshot
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
6b1285d38b rbd: copy passphrase for encrypted clones
When a source volume is encrypted, the passphrase needs to be copied and
stored for the newly cloned volume.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
7b332a0184 rbd: add rbdImage.copyEncryptionConfig() to copy encryption metadata
Cloning volumes requires copying the DEK from the source to the newly
cloned volume. Introduce copyEncryptionConfig() as a helper for that.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
7e6feecc25 util: add VolumeEncryption.StoreCryptoPassphrase()
The new StoreCryptoPassphrase() method makes it possible to store an
unencrypted passphrase newly encrypted in the DEKStore.

Cloning volumes will use this, as the passphrase from the original
volume will need to get copied as part of the metadata for the volume.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
b6aa19eea5 rbd: pass secrets when creating an source rbdVolume for cloning
Without this, the rbdVolume can not connect to the Ceph cluster and
configure the (optional) encryption.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
92b2e08adf rbd: improve logging in deleteImage()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
99da92cfd7 rbd: move deletion of DEK to deleteImage()
The ControllerServer should not need to care about support for
encryption, ideally it is transparantly handled by the rbdVolume type
and its internal API.

Deleting the DEK was one of the last remainders that was explicitly done
inside the ControllerServer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
151d066938 util: add logging when OpenEncryptedVolume() encounters an error
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
bd1388fb96 util: log available configs when KMS not found
When the KMS configuration can not be found, it is useful to know what
configurations are available. This aids troubleshooting when typos in
the KMS ID are made.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Niels de Vos
a7c261a394 logging: correct formatting when reporting error in createVolumeFromSnapshot()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-14 03:59:28 +00:00
Rakshith R
ae6a52a84e util: add nil check to default ControllerGetCapabilities()
Currently default ControllerGetCapabilities function is being
used which throws 'runtime error: invalid memory address or
nil pointer dereference' when `--controllerServer=true` is not
set in provisioner deployment args.
This commit adds a check to prevent it.

Fixes: 1925

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-09 10:12:48 +00:00
Rakshith R
10d539efc8 cleanup: correct nolint directive listing format
nolint directive needs to be followed by comma separated
list of linters. This commit changes to gocognit:gocyclo
which was not recognised to linters which show error for
the function.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-09 07:24:47 +00:00
Rakshith R
3f3489367c cleanup: correct linter name mnd to gomnd
Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-09 07:24:47 +00:00
Rakshith R
fb7389f478 cephfs: add stderr to mount function errors
This commit appends stderr to error in both kernel and
ceph-fuse mounter functions to better be able to debug
errors.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-08 12:18:01 +00:00
Yug
62ae17e263 doc: update dev standup timing
The Dev standup was preponed 2 hours
some time back. Updating the same in upstream.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-04-07 13:59:10 +00:00
Yug
2f7b733f7e doc: update command usage
Running the command specified `date -d 14:00 UTC`
fails with the following error:
```date: the argument ‘UTC’ lacks a leading '+';
when using an option to specify date(s), any non-option
argument must be a format string beginning with '+'
```
Add quotes to ensure expected output.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-04-07 13:59:10 +00:00
Yug
f4d9fd0e89 ci: Updated mergify rules for containerized-tests
Since github actions cover all the tests covered by
the containerized tests, disabling them in upstream
to avoid running repetitive tests and properly utilize
CI instances.
The test will still be available to run locally.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-04-07 18:26:07 +05:30
Madhu Rajanna
e2fa84357a rbd: take lock when reconciling the PV
there can be a change we can reconcile same
PV parallelly we can endup in generating and
deleting multiple omap keys. to be on safer
side taking lock to process one volumeHandle
at a time.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-07 11:46:27 +00:00
Madhu Rajanna
0f8813d89f rbd:store/Read volumeID in/from PV annotation
In the case of the Async DR, the volumeID will
not be the same if the clusterID or the PoolID
is different, With Earlier implementation, it
is expected that the new volumeID mapping is
stored in the rados omap pool. In the case of the
ControllerExpand or the DeleteVolume Request,
the only volumeID will be sent it's not possible
to find the corresponding poolID in the new cluster.

With This Change, it works as below

The csi-rbdplugin-controller will watch for the PV
objects, when there are any PV objects created it
will check the omap already exists, If the omap doesn't
exist it will generate the new volumeID and it checks for
the volumeID mapping entry in the PV annotation, if the
mapping does not exist, it will add the new entry
to the PV annotation.

The cephcsi will check for the PV annotations if the
omap does not exist if the mapping exists in the PV
annotation, it will use the new volumeID for further
operations.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-07 11:46:27 +00:00
Rakshith R
9aea701bd9 ci: enable nestif linter
The nestif linter reports deeply nested if statements.
This commit enables the nestif linter and sets
min-complexity to 7.

Closes: #1229

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-07 02:31:41 +00:00