Commit Graph

3904 Commits

Author SHA1 Message Date
Madhu Rajanna
76c2f3c109 cleanup: re-use flattenTemporaryClonedImages to reduce duplicate code
re-use flattenTemporaryClonedImages to avoid code duplication

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-15 16:10:38 +00:00
Madhu Rajanna
8fc9146056 rbd: flatten temp cloned images
If the snapshots on the parent image exceeds
maxSnapshotsOnImage count, we need to flatten
all the temporary cloned images to over come the
krbd issue of maximum number of snapshots on
an image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-15 16:10:38 +00:00
Madhu Rajanna
2fe1ee5287 rbd: create temporary snapshot with name same as temporary clone
create temporary snapshot on the parent image same as
name as the temporary clone rbd image. Naming the snapshot
and the temporary cloned image helps to flatten the temporary
cloned images when the snapshots on the parent image exceeds
the configured maxSnapshotsOnImage.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-15 16:10:38 +00:00
Humble Chirammal
285028727a ci: disable centos ci checks from mergify
CentOS CI outage block PRs to get merged even when it has enough
approvals. Disabling the centos check for now as there is no ETA
on the availability of the setup atm.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-07-15 20:10:34 +05:30
Madhu Rajanna
09ffaee7c3 cleanup: rename newVolumeOptionsFromVersion1Context for more clarity
rename newVolumeOptionsFromVersion1Context to newVolumeOptionsFromMonitorList
to provide more clarity to the function readers and also fixed comments.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-13 04:04:28 +00:00
Madhu Rajanna
d15ded88f5 cleanup: Remove support for Delete and Unmounting v1.1.0 PVC
as v1.0.0 is deprecated we need to remove the support
for it in the Next coming (v3.0.0) release. This PR
removes the support for the same.

closes #882

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 16:07:13 +00:00
Madhu Rajanna
a0fd805a8b rbd: Add support for smart cloning
Added support for RBD PVC to PVC cloning, below
commands are executed to create a PVC-PVC clone from
RBD side.

* Check the depth(n) of the cloned image if n>=(hard limit -2)
or ((soft limit-2) Add a task to flatten the image and return
about (to avoid image leak) **Note** will try to flatten the
temp clone image in the chain if available
* Reserve the key and values in omap (this will help us to
avoid the leak as it's not reserved earlier as we have returned
ABORT (the request may not come back))
* Create a snapshot of rbd image
* Clone the snapshot (temp clone)
* Delete the snapshot
* Snapshot the temp clone
* Clone the snapshot (final clone)
* Delete the snapshot

```bash
1) check the image depth of the parent image if flatten required
add a task to flatten image and return ABORT to avoid leak
(hardlimit-2 and softlimit-2 check will be done)
2) Reserve omap keys
2) rbd snap create <RBD image for src k8s volume>@<random snap name>
3) rbd clone --rbd-default-clone-format 2 --image-feature
layering,deep-flatten <RBD image for src k8s volume>@<random snap>
<RBD image for temporary snap image>
4) rbd snap rm <RBD image for src k8s volume>@<random snap name>
5) rbd snap create <cloned RBD image created in snapshot process>@<random snap name>
6) rbd clone --rbd-default-clone-format 2 --image-feature <k8s dst vol config>
 <RBD image for temporary snap image>@<random snap name> <RBD image for k8s dst vol>
7)rbd snap rm <RBD image for src k8s volume>@<random snap name>
```

* Delete temporary clone image created as part of clone(delete if present)
* Delete rbd image

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 14:02:12 +00:00
Madhu Rajanna
9077c25c15 e2e: add e2e to test pvc-pvc cloning
Added an e2e testcase to test pvc-pvc
cloning.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 14:02:12 +00:00
Madhu Rajanna
7481786c44 ci: update kubernetes v1.17 to latest patch release
kubernetes v1.17.8 is released recently,This commit
updates the kubernetes v1.17 to latest patch release

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
caa1eca8f6 ci: update kubenetes version to v1.18.5 for helm E2E
updated kubernetes version to latest version
to test cephcsi helm charts

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
c8e532aa98 ci: Add kube v1.18.5 to E2E testing
Updated travis CI to test cephcsi with
kubernetes v1.18.5

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
6786d92246 ci: remove kube 1.16 from Travis CI
As kube v1.18 is released we dont need to test
-2 version in our CI, removing the 1.16 testing
for the same reason.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
f05c9a6a93 ci: fix psp issue in minikube latest version
With minikube versions greater than 1.6.2 and less than 1.11.1, the YAML files
minikube path will not be automatically applied to the cluster. we will get
errors during bootstrap of the cluster if the admission controller is enabled.

To use Pod Security Policies with these versions of minikube, first start a
cluster without the `PodSecurityPolicy` admission controller enabled.

Next, apply the psp yaml. and stop the cluster and then restart it
with the admission controller enabled.

```
minikube start
kubectl apply -f /path/to/psp.yaml
minikube stop
minikube start --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy
```

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
8aae7b553c ci: update minikube version to latest release
updated minikube to latest release v1.12.0

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Yug
1490daed7e cleanup: Avoid usage of numbers
Add seperate functions to handle all
levels and types of logging.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-10 07:41:23 +00:00
Yug
8dc4ab6b1b rebase: update k8s.io/klog to v2.3.0
Update klog version to v2.3.0

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-10 07:41:23 +00:00
Madhu Rajanna
b773e821c8 ci: use testCephFS flag to skip cephfs tests
This commit fixes the flag check to skip
cephfs E2E tests.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-09 13:50:21 +00:00
Sven Anderson
13f291dfc6 cleanup: use errors.As() for error type checks
Replaces some remaining old-style error type checks with errors.As()

Signed-off-by: Sven Anderson <sven@redhat.com>
2020-07-09 09:58:33 +00:00
Madhu Rajanna
310c36e319 ci: separate the cephfs and rbd tests
As the maximum time allocated for the Travis
CI is 50 minutes and the cephfs and rbd E2E
takes around 45 Minutes to run completely.
This is blocking us from adding more tests in E2E.
splitting out the E2E will help us to run more tests
for each driver as we will get 50 minutes for
each Travis CI instance.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-09 06:42:28 +00:00
Madhu Rajanna
9d1c91f3c0 ci: Pass on the arguments to E2E
Pass on the arugments which are sent when
calling the scripts to the E2E.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-09 06:42:28 +00:00
Madhu Rajanna
63b795d165 e2e: provide an option to skip rbd and cephfs e2e
Added two new parameters for e2e test to skip
rbd and cephfs tests. This will help us to
run more test in Travis CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-09 06:42:28 +00:00
Madhu Rajanna
4e82384b43 rbd: check image not found error
during the checkSnapCloneExists we are checking
the image, if the image not found we are deleting
the snapshot on the parent image, This PR corrects
the comparasion. instead of snapshotNotFound we need
to check ImageNotFound error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-08 17:09:59 +00:00
Mudit Agarwal
953cfa508a util: fix tracevol.py to work with volumeNamePrefix in storageclass
tarcevol.py works with a fixed volume name prefix "csi-vol-", this could be changed by the storageclass
and hence tracevol.py should work with the dynamic value rather than the static.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-07-08 14:33:15 +00:00
Mudit Agarwal
2ef5729a28 cleanup: restructure a part of existing code
Have restructered a part of existing code so that it would be useful to get pvdata

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-07-08 14:33:15 +00:00
Madhu Rajanna
82080dcfa9 util: update unit testing for deep flatten
updated unit testing for the kernel check
for deep flatten feature for both supported
upstream kernel version (5.1.0+) and RHEL
8.2 backport

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-08 05:48:42 +00:00
Niels de Vos
969b032f1c build: error out when Podman or Docker is not available
In case Podman or Docker is not installed, running containerized tests
or builds will fail with unclear errors.

Introduce a 'need-container-cmd' make target that checks if the
CONTAINER_CMD is available (and non-empty). When running without Podman
and Docker, the following error is displayed:

    $ make containerized-build
    cephcsi image settings: quay.io/cephcsi/cephcsi version canary
    Missing container support, install Podman or Docker
    make: *** [Makefile:135: need-container-cmd] Error 1

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-07 09:49:45 +00:00
Yug
34593c9af0 cleanup: Reduce cyclomatic complexity of topology_test
Reduce redundant code to make the test simpler.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-07 06:08:55 +00:00
Madhu Rajanna
f05c8ed2a7 journal: remove unused rados CLI functions
Removed unused rados CLI functions, as rados
CLI is replaced with go-ceph bindings.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 15:03:44 +00:00
Madhu Rajanna
9bb79fbe7d journal: replace getStoredImageId with go-ceph
moved implementation of getStoredImageID to go-ceph
to get better performance.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 15:03:44 +00:00
Madhu Rajanna
341a3c9f59 journal: replace rados CLI with go-ceph to storeimage id
Moved the implementation of storeImageID from rados
CLI to go-ceph api.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 15:03:44 +00:00
Madhu Rajanna
8f758450d8 rbd: add RHEL 8.2 kernel to the list
as RHEL 8.2 supports the deep-flatten
feature, added it to the list to map
the rbd images on the node without flattening.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 12:20:00 +00:00
Madhu Rajanna
04c8c7fd4a rbd: correct upstream kernel version for deep-flatten
as v5.1.0 supports the deep-flatten feature,lowering
the required version to map rbd images which
are having deep-flatten feature

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 12:20:00 +00:00
Madhu Rajanna
6838d30aff e2e: reduce the pod cound and wait time
Reduced the number of pods created
in ROX E2E to save some time in E2E
and changed the waiting time from 2 to 1
min.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
b76d470bf1 e2e: increase e2e test timeout
as we are creating more resources
the e2e timeout is not enough we need
to increase the e2e timeout to higher
value for now.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
db1b5f9c55 e2e: move deploy timeout to build.env
moved deploy timeout from travis
scripts to build.env

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
5838b08edd e2e: Move e2e timeout to build.env
Moved e2e test timeout to build.env

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
032fb228dd rbd: take lock on parent image during snapshotcreate
we need to take lock on parent rbd image when
we are creating a snapshot from it, if the user
tries to delete/resize the rbd image when we are
taking snapshots,we may face issues. if the volume
lock is present on the rbd image, the user cannot
resize the rbd image nor delete the rbd image.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
f946efb7b7 e2e: set maxsnapshotsonimage during helm install
as we need to test the maxsnapshotsonimage we
need to set the limit to minimal value which we
can test in CI as the default limit is 450,which
cannot be tested in CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
61a81d35e8 rbd: flatten cloned images to freeup snapshot
flatten cloned images to remove the link
with the snapshot as the parent snapshot can
be removed from the trash.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
8ef7143e6c rbd: add maxsnapshotsonimage flag
Added maxsnapshotsonimage flag to flatten
the older rbd images on the chain to avoid
issue in krbd.The limit is in krbd since it
only allocate 1 4KiB page to handle all the
snapshot ids for an image.

The max limit is 510 as per
https://github.com/torvalds/linux/blob/
aaa2faab4ed8e5fe0111e04d6e168c028fe2987f/drivers/block/rbd.c#L98
in cephcsi we arekeeping the default to 450 to reserve 10%
to avoid issues.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
c04b903eca rbd: add listsnapshots function
added listsnapshots function for an
rbd image to list all the snapshots
created from an rbd images, This will
list the snapshots which are in trash also.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Sven Anderson
de74c36d8c cleanup: address err113 warnings about direct error comparisons
Several places in the code compared errors directly with the go-ceph
sentinel errors.  This change uses the errors.Is() function of go
1.13 instead.  The err113 linter reported this issue as:

err113: do not compare errors directly, use errors.Is() instead

Signed-off-by: Sven Anderson <sven@redhat.com>
2020-07-06 08:57:46 +00:00
Niels de Vos
7affb9289d ci: display a warning when GO_ARCH is not set for image-cephcsi
`make image-cephcsi` will fail when Golang is not installed. There is no
strict requirement for Golang to be available, it is only used to gather
the architecture of the OS where the image is built. It is possible to
build the image successfully with `make image-cephcsi GOARCH=amd64`.

In case Golang is not installed, GOARCH can not be detected
automatically. This will cause a failure while installing Golang in the
container image. Because the failure is not very clear, display a
warning in the case the GO_ARCH (from ${GOARCH} in the Makefile) is not
set.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-06 06:20:53 +00:00
Madhu Rajanna
826f7126cd doc: update doc to delete alpha snapshot
updated upgrade documentation to remove
the snapshot created by alpha driver before
upgrade of CSI driver as beta snapshot is not
backward compatible with the alpha snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-03 11:32:00 +00:00
Niels de Vos
267c709194 cleanup: use errors.As() in rbd/controllerserver methods
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
91ade42a32 cleanup: use errors.As() in journal.Connection methods
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
ad04e0d8c4 cleanup: use errors.As() in cephfs.checkVolExists()
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
72c74f5f8a cleanup: use errors.As() in cephfs.NodeStageVolume()
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
c9d5bc8a08 cleanup: use errors.As() in e2e/utils
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
a47cfbc6c8 cleanup: use errors.As() in rbdVolume.Exists() and rbd.checkSnapExists()
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00