Commit Graph

1505 Commits

Author SHA1 Message Date
Niels de Vos
4be946b816 e2e: pull centos image from registry.centos.org
The BlockVolume PVC tests consume the example files that refer to
"centos:latest" without registry. This means that the images will get
pulled from Docker Hub, which has rate limits preventing CI jobs from
pulling images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 4fd0294eb7)

# Conflicts:
#	examples/rbd/block-pod-clone.yaml
2020-12-07 17:04:16 +00:00
Madhu Rajanna
c449feaa95 e2e: use centos as image for normal user validation
Reduce the number of images that get pulled from Docker Hub. Use the
official CentOS container registry instead.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 168526a906)
2020-12-07 17:04:16 +00:00
Niels de Vos
9999f67a2b ci: set imagePullPolicy for Vault to IfNotPresent
Deploying Vault still fails on occasion. It seems that the
imagePullPolicy has not been configured for the container yet.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit b0f3b27209)
2020-12-03 18:20:34 +00:00
Madhu Rajanna
e8267de3c9 cephfs: fix logic to call ceph fs resize
clusterAdditionalInfo map is holding a localClusterState
for checking ceph cluster supports resize and subvolumegroup
is created or not, currently we are checking if the key is present
in a map and localClusterStatelocalClusterState.resizeSupported
is set to false to call ceph fs subvolume resize to check command is
supported or not, if a structure is  initialized all its members
are set to default value. so we will never going to check the
ceph fs subvolume  resize command is supported in backend or not, we are
always using ceph fs subvolume create to resize subvolume. in some
ceph version ceph fs subvolume create wont work to resize a subvolume.
This commit changes the resizeSupported from bool to *bool for
proper handling of this scenario.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit a35a835e9c)
2020-12-03 09:57:17 +00:00
Madhu Rajanna
75e06f9581 ci: run github work flow only for pull request
currently github action are getting trigerred for
both pull and push request, this commits removes the
action for push  events.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 00d34fdb2d)
2020-12-03 10:53:10 +05:30
Niels de Vos
cad6707071 build: use docker.io/ceph/ceph:v15 as BASE_IMAGE
Use a qualified image name, including the registry where it should come
from. This makes it possible to pull the image from the right location,
and consume it in CI jobs without trying to pull again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit bdf8fe7f93)
2020-12-02 17:56:57 +05:30
Madhu Rajanna
067f77fbe1 ci: add github action for multiarch build
added a github action for multiarch docker
build.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-02 13:07:04 +05:30
Niels de Vos
12ff7b7c98 e2e: use docker.io/library as prefix for official images
Docker Hub offers a way to pull official images without any project
prefix, like "docker.io/vault:latest". This does a redirect to the
images located under "docker.io/library".

By using the full qualified image name, a redirect gets removed while
pulling the images. This reduces the likelyhood of hittin Docker Hub
pull rate-limits.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 1f18e876f0)
2020-12-02 11:39:38 +05:30
Niels de Vos
953c33058f deploy: use docker.io for unqualified image names
Images that have an unqualified name (no explicit registry) come from
Docker Hub. This can be made explicit by adding docker.io as prefix. In
addition, the default :latest tag has been added too.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit eaeee8ac3d)
2020-12-02 11:39:38 +05:30
Niels de Vos
4830808592 e2e: pull centos image from registry.centos.org
The BlockVolume PVC tests consume the example files that refer to
"centos:latest" without registry. This means that the images will get
pulled from Docker Hub, which has rate limits preventing CI jobs from
pulling images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 4fd0294eb7)
2020-12-02 11:39:38 +05:30
Madhu Rajanna
0c1ba4ce88 build: update imagepullpolicy for vault
this allows the image to be reused instead of pulling
it again.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 7d229c2369)
2020-12-02 11:39:38 +05:30
Niels de Vos
e0881258f8 e2e: use full-qualified-image-name for vault-init-job
On occasion deploying Vault fails. It seems the vault-init-job batch job
does not use a full-qualified-image-name for the "vault" container.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 1845f2b77d)
2020-12-02 11:39:38 +05:30
Madhu Rajanna
1fd09d8637 e2e: set imagePullPolicy to ifNotPresent
If the imagePullPolicy is not set and the image
tag is empty or latest the image is always pulled.
This commit sets the policy to pull image if not
present.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 8d86bac9b7)
2020-11-30 18:36:36 +00:00
Madhu Rajanna
19204ea371 revert: update templates for v3.1.2 release
This reverts commit b5fe726b2a.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-23 13:31:17 +05:30
Madhu Rajanna
b5fe726b2a build: update templates for v3.1.2 release
updated required templates for v3.1.2 release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-19 17:46:31 +05:30
Niels de Vos
0321f983bb ci: use the Fedora container registry for cephcsi:test
This reduces the dependency on Docker, where image pull rate limits are
seen in the CI.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 880b5bb427)
2020-11-19 10:39:03 +00:00
Madhu Rajanna
4be9c0159f rbd: add minsnapshotsonimage flag
An rbd image can have a maximum number of
snapshots defined by maxsnapshotsonimage
On the limit is reached the cephcsi will
start flattening the older snapshots and
returns the ABORT error message, The Request
comes after this as to wait till all the
images are flattened (this will increase the
PVC creation time.  Instead of waiting till
the maximum snapshots on an RBD image, we can
have a soft limit, once the limit reached
cephcsi will start flattening the task to
break the chain. With this PVC  creation time
will only be affected when the hard limit
(minsnapshotsonimage) reached.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 8d3a44d0c4)
2020-11-18 20:11:43 +05:30
Madhu Rajanna
83fbbd6d93 cephfs: check only the stderror message for umount
if the cephfs mountpoint is not mounted the umount
call will fail with both stderr and err,
an error (exit status 32) and stdError (umount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e622cd2b-0b87-4f49-a709-b89664ca6ec4/globalmount: not mounted.

checking stderr is sufficient to find out the mountpoint
is not mounted.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-18 09:37:57 +05:30
Madhu Rajanna
09c303c2cb cephfs: check clone pending state
sometimes, cephfs returns pending as clone
status if we request for the subvolume clone.
cephcsi need to consider the pending state
when checking the clone status and it should
return abort error message if the clone is in
pending state or in-progress state.

Co-authored-by: Yug <yuggupta27@gmail.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-06 21:19:17 +05:30
Madhu Rajanna
75d190f430 cephfs: check stderror for CLI commands
currently in someplaces we are checking the stderror
for CLI errors and insome places we are checking for
just error. This commit fixes the issue by checking
the stderror for actual CLI errors.

Co-authored-by: Yug <yuggupta27@gmail.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-06 21:19:17 +05:30
Madhu Rajanna
eaf0058c31 util: log stderror in ExecCommand
all the CLI commands we execute logs the actual
error in stdError. logging stderror is using for
debugging.

Co-authored-by: Yug <yuggupta27@gmail.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-06 21:19:17 +05:30
John Mulligan
e99b996bea journal: fix reading omaps from objects with large key counts
The implementation of getOMapValues assumed that the number of key-value
pairs assigned to the object would be close to the number of keys
being requested. When the number of keys on the object exceeded the
"listExcess" value the function would fail to read additional keys
even if they existed in the omap.
This change sets a large fixed "chunk size" value and keeps reading
key-value pairs as long as the callback gets called and increments
the numKeys counter.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit 8a41cd03a5)
2020-11-06 07:46:07 +00:00
Madhu Rajanna
115285f970 ci: update minikube to latest version
updated minikube to latest version to
fix ci errors.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-06 11:19:01 +05:30
Madhu Rajanna
9db673afc7 ci: fix shellcheck in test-go
Fixed shellcheck in test-go script

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit fdbd487741)
2020-11-06 04:17:20 +00:00
Humble Chirammal
6e3d68ee7b cephfs: fix wrong error check in CreateVolume rollback action
Previously the purgeVolume error was ignored due to wrong error variable
check in the createVolume. With this change it checks on the proper error.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
(cherry picked from commit 9dee064b77)
2020-10-20 05:57:38 +00:00
Madhu Rajanna
77e4643231 build: revert update templates for v3.1.1 release
This reverts commit 22b631e991.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-09-30 11:13:44 +05:30
Madhu Rajanna
22b631e991 build: update templates for v3.1.1 release
updated required templates for v3.1.1 release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-09-29 12:56:15 +05:30
Madhu Rajanna
4935bc3ead e2e: add backend validation for rados namespaced images
added more backend validation for rados namespaced
images.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit dc38320fad)
2020-09-25 17:05:19 +00:00
Marton Natko
8dc3f51350 rbd: fix problem with deletion of a namespaced image
fix problem with deletion of a namespaced image

Signed-off-by: Marton Natko <mnatko@transip.nl>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 91c49c14b3)
2020-09-25 17:05:19 +00:00
Marton Natko
d999d9db0c rbd: fix typo in createVolume validation
Signed-off-by: Marton Natko <mnatko@transip.nl>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2fbca0d1e2)
2020-09-25 17:05:19 +00:00
Madhu Rajanna
764e33f249 cephfs: check ENOTEMPTY when removing subvolume
return a proper error message to the user when
the subvolume has the snapshots and it cannot
be removed until the snapshots on the subvolume
have to be deleted.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 89b326c896)
2020-09-25 09:26:48 +00:00
Niels de Vos
cb7c3080cd ci: do not assume Docker is availble on the minikube host
Instead of using the Docker command to push the image to to minikube VM,
read the image from stdin over ssh and load it with the Docker command
that is available inside the VM.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-25 13:35:08 +05:30
Prasanna Kumar Kalever
ce0b3b4f88 ci: add a way to supply CONTAINER_CMD of choice
Add a way to supply local CONTAINER_CMD option of choice via
env variable to minikube.sh

Note: we still use docker daemon env at minikube box, in the future
we can switch to podman service env '# minikube podman-env' if needed

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-09-25 13:35:08 +05:30
Niels de Vos
54aed7f899 e2e: add simple ceph-fuse StorageClass tests
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit f41cd105c4)
2020-09-23 14:39:43 +00:00
Niels de Vos
126a71e51e e2e: pass SC parameters to createCephfsStorageClass()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 85df59531b)
2020-09-23 14:39:43 +00:00
Niels de Vos
b810ca57a0 cephfs: trim spaces around fuseMountOptions
When passing

    fuseMountOptions: debug

in the StorageClass, the mount options passed on the ceph-fuse
commandline result in "-o nonempty ,debug". The additional space before
the ",debug" causes the mount command to fail.

Fixes: 1485
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 6f2f972f5b)
2020-09-23 14:39:43 +00:00
Mudit Agarwal
bef4873049 build: update vendor with latest version
Updating the version of golang.org/x/text to version v0.3.3.
It fixes the vulnerability in http://golang.org/x/text/encoding/unicode
which leads UTF-16 decoder entering an infinite loop causing
the program to crash or run out of memory.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
(cherry picked from commit 8e434bb3ee)
2020-09-21 11:57:45 +00:00
Yug
8170e12114 cephfs: Fetch subvolume features in newSnapshotOptionsFromID
Get subvolume features using subvolume info and update the
volume options with Features parameter.

Signed-off-by: Yug <yuggupta27@gmail.com>
(cherry picked from commit 0682f0e500)
2020-09-17 09:04:41 +00:00
Yug
6a64958876 cephfs: No-Op Unprotect call if "snapshot-autoprotect" feature present
The subvolume features consists the list of features, which if includes
"snapshot-autoprotect", result in query based approach to
detecting the need for protect/unprotect.
If "snapshot-autoprotect" feature is present, The UnprotectSnapshot
call should be treated as a no-op

Signed-off-by: Yug <yuggupta27@gmail.com>
(cherry picked from commit 9e959c45a3)
2020-09-17 09:04:41 +00:00
Yug
15463d8f52 cephfs: No-Op Protect call if "snapshot-autoprotect" feature present
The subvolume features consists the list of features, which if includes
"snapshot-autoprotect", result in query based approach to
detecting the need for protect/unprotect.
If "snapshot-autoprotect" feature is present, The ProtectSnapshot
call should be treated as a no-op

Signed-off-by: Yug <yuggupta27@gmail.com>
(cherry picked from commit 43a4c100ec)
2020-09-17 09:04:41 +00:00
Yug
b32569bc47 cephfs: use subvolume info if available
Use subvolume info to fetch the subvolume path.
If `subvolume info` command is not available,
use `getpath` command instead.

Signed-off-by: Yug <yuggupta27@gmail.com>
(cherry picked from commit 365eb58ff7)
2020-09-11 12:23:19 +00:00
Yug
0f51df99e8 cephfs: validate if subvolume has snapshot-retention feature
Add validateSnapshotRetention() to verify if subvolume has
`snapshot-retention` feature.

Signed-off-by: Yug <yuggupta27@gmail.com>
(cherry picked from commit 59fc4aa00f)
2020-09-11 12:23:19 +00:00
Yug
5daf21831e cephfs: Add --retain-snapshots flag to purge volume operations
Snapshots can be retained even after subvolume deletion in
Ceph 14.2.12. Adding support for the same in ceph-csi.

Signed-off-by: Yug <yuggupta27@gmail.com>
(cherry picked from commit 6351358752)
2020-09-11 12:23:19 +00:00
Madhu Rajanna
cb09a3583e e2e: update cephfs ROX test case
updated cephfs ROX test case to create
ROX Clone PVC instead of creating an empty PVC

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit be3a78afaf)
2020-09-10 15:05:38 +00:00
Madhu Rajanna
d7e3f06384 cephfs: allow readonly only for volume with contentsource
It doesnot make sense to allow the creation of empty
volumes with readonly access, this commit allows the
creation of volume which is having readonly capabilities
only if the content source is set for the volume.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 860b65c446)
2020-09-10 15:05:38 +00:00
Madhu Rajanna
5a7e188bb4 rbd: allow readonly only for volume with contentsource
It doesnot make sense to allow the creation of empty
volumes which is going to be accessed with readonly mode,
this commit allows the creation of volume which is having
readonly capabilities only if the content source is set
for the volume.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2deabcd284)
2020-09-10 15:05:38 +00:00
Madhu Rajanna
c9601189ef util: add helper to validate ROX request
added a helper function to validate is create
request for ROX is for volume from source.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit d816df4f42)
2020-09-10 15:05:38 +00:00
Madhu Rajanna
90da7961c5 e2e: rework on E2E framework
rework of E2E framework for better code
organization and add more helpful logs for
debugging.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit b4693dcffe)
2020-09-10 08:54:20 +00:00
Mudit Agarwal
6105a4f428 rbd: NodeExpandVolume() should use StagingTargetPath
Form kubernetes v1.19 onwards NodeRequest is getting volume path
in StagingTargetPath instead of VolumePath, cephcsi should also
use the same.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
(cherry picked from commit 4de1abad5e)
2020-08-25 17:20:08 +00:00
Mudit Agarwal
1792cff878 rbd: implement rbd.updateVolWithImageInfo() with go-ceph
updateVolWithImageInfo() is currently executing an rbd command
to get details about an RBD image. Replaced it with the
required go-ceph functions.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
(cherry picked from commit 8c2bdbc001)
2020-08-25 08:13:52 +00:00