Commit Graph

4208 Commits

Author SHA1 Message Date
Niels de Vos
61924adf57 util: use FatalLog() in util/httpserver.go
Hide the use of klog by using our own FatalLog() function.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-11 08:11:37 +00:00
Niels de Vos
ab033f85df util: add FatalLog(), ErrorLog() and WarningLog() functions
These functions will always log the message, irrespectively of the
log-level that has been configured.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-11 08:11:37 +00:00
Niels de Vos
44863a9d29 cleanup: move log functions to util/log.go
There is a util/log.go file, so the log functions in util/util.go can be
moved there.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-11 08:11:37 +00:00
Madhu Rajanna
b86d329d1e rbd: add stagingpath to error message
if the stagingpath is not present inside
the container send the staging path in error
message.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-10 08:51:23 +00:00
Humble Chirammal
ef1785ce4d e2e: remove volume populate test case and also change writeDataInPod()
As we are populating the volume in other two test cases for clone and
snapshot operation, we dont need a specific test case now.
WriteDataInPod() function is also changed to take the pod spec and write
some data to it

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
7d2bba1375 cephfs: introduce parsetime() to parse createdAt field in snap return
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
c3400bfb97 cephfs: add snapshot create and delete functionalilies
It also add helper routines like parsetime,doSnapshot..etc

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
20c90ddfc4 cephfs: getVolumeRootPathCeph and purgeVolume use new error strings
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
1f4839cb5e cephfs: add subvolume struct and getSubvolumeInfo calls
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
31e9288f8f cephfs: remove inValidCommmand and errNotFoundString
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
5bceb590fd cephfs: add snap reserve/unreserve and snap exist functionalities
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
38d005e4e6 cephfs: validate create volume request
Also add functionality to generate snap from request
and to get mon and clusterID from the request

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
9c000add29 cephfs: Change checkVolExist for snapshot and clone workflow
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
c773097f85 cephfs: add snapshot and clone helper functions
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
d1fe12b4f0 cephfs: change createvolume for snapshot/clone workflow
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
c31f349c6f cephfs: use delete operation lock in delete volume
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
f9257ed1cb cephfs: add snapshot create/delete capabilities in controllerserver
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
5a71949dcd cephfs: introduce newSnapshotOptionsFromID to generate volOpt and sid
From provided CSI volume ID this populate volumeOptions and snapshot
identifier after connecting to the snapJournal.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
a4f2c5617c cephfs: retrieve NewCSISnapshotJournal with namespace set
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
1746b4e6ee cephfs: add various error strings for command validation
Compared to previous version of the error strings, this change
depend on error strings like ENOENT, EEXIST, EINVAL..etc
The format of the error strings change in different cluster versions
and the error code return should not change. This also add extra
error strings for snapshot and clone operation outputs

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
6dcf03a4bc cephfs: add checkContentSource() to validate the data source
checkContentSource() validate the data source in the request
and then populate volumeOptions or snapshotshot identifier in
case of snapshot source. If the data source is volume, then
parentVolumeOption and pvID are populated.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
2bb0bbde7f cephfs: change createBackingVolume to accomodate clone operations
adjust createBackingVolume() to create a subvolume from snapshot
or existing subvolume by taking restore or clone operation locks

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
c968380134 cephfs: add snapshotlock and operationlock to controllerserver
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
66827facc3 cephfs: make use of expand operation lock in ControllerExpandVolume
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
074feddaf3 e2e: add test cases for cephfs snapshot creation,deletion and restore
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
90762342a0 e2e: introduce createCephFSSnapshotClass creation
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
2d72550a39 e2e: introduce writeDataInPod() to write data to an attached PVC
writeDataInPod() write data to the attached PVC using `dd` command
It leave the pod and pvc state as it is.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
53fa00dee8 deploy: update external provisioner version to v1.6.0 from v1.4.0
update helm chart and deploy yaml for version upgrade

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
c85d03c79e e2e: add/modify deployment files for cephfs snapshot/clone tests
Add snapshot class, snapshot, restore, clone PVC and pod yamls
for e2e test to consume

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
b40af5d1a5 helm: add deployment charts for cephfs csi snapshotter and roles
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Humble Chirammal
cd107d433d deploy: add csi-cephfsplugin provisioner deployment and role
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-08 00:11:35 +00:00
Yug
23e0874ebf deploy: add wrapper function of kubectl
Add retries to prevent ci failure instantly.
Now, the command execution will retry upto
5 times, to avoid failures in some runs.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-07 09:28:24 +00:00
Madhu Rajanna
d8df06e3d8 ci: update mergify rules for centos ci
Updated mergify rules for auto merging the PR
in master branch.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-06 13:36:43 +05:30
Madhu Rajanna
96ecb3946d ci: remove e2e testing from travis CI
Remove E2E testing from travis CI as we
are moving to run E2E tests in centos CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-06 13:36:43 +05:30
Niels de Vos
c578ab01a7 ci: pass HELM_VERSION when installing Helm in test container
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 07:43:10 +00:00
Niels de Vos
d4f18458de ci: include build.env variables when installing Helm
By default the install-helm.sh script uses "latest" as version for Helm.
Unfortunately this version does not exist. The HELM_VERSION variable is
already set in build.env, so source the configuration file as one of the
first actions in install-helm.sh.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 07:43:10 +00:00
Yug
41be151ac6 deploy: remove vault along with driver
The created vault should also be
brought down with the driver.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-04 16:00:21 +00:00
Yug
2e14116ed7 deploy: add vault creation to rbd driver deployment
Currently, the script does not deploy the driver singlehandedly;
As the vault creation needs to be done prior to that.
The script now includes the vault creation so that
one script can be sufficient to deploy the rbd driver.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-04 16:00:21 +00:00
Niels de Vos
b9d1f16360 ci: make number of CPUs for minikube VM configurable
By default minikube uses 2 CPUs, which might be too little for some of
the tests. When not passing a CPUS environment variable, use all CPUs
available on the system (detected with 'nproc').

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-04 08:46:17 +00:00
Madhu Rajanna
2458ec6573 rbd: return error if fetching cluster id fails
if we are not able to fetch the cluster-ID from
the createSnapshot request and also if we are
not able to get the monitor information from
the cluster-ID return error instead of using
the parent image information.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-03 14:25:06 +00:00
Mudit Agarwal
9ed0811422 rbd: implement rbdVolume.resize() with go-ceph
Replaced command execution with go-ceph Resize() function.
Volsize is being updated before waiting for resize() to return,
fixed it to get updated only after resize() is successful.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-08-03 10:50:01 +00:00
Mudit Agarwal
1eff20590d util: restructure tracevol.py function to get_subvol_group()
get_subvol_group() returns empty string if subvolumeGroup is not defined,
changed it to return "csi" as default subvolumeGroup.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-08-03 07:14:02 +00:00
Mudit Agarwal
5fa1be586a cephfs: fix tracevol.py to work with dynamic value of fsname
tracevol.py takes 'myfs' as default fsname, changed it so that it can work with
dynamic values of fsname.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-08-03 07:14:02 +00:00
Mudit Agarwal
6814f598ce util: fix tracevol.py to manage config map created by Rook
If the config map is created by rook then there won't be any provision to
specify subvolumeGroup, tracevol.py should skip looking for subvolumeGroup
in such case.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-08-03 07:14:02 +00:00
Mudit Agarwal
3585b21e80 util: fix tracevol.py to take config map namespace as an option
Added a new option so that user can specify the namespace for config map.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-08-03 07:14:02 +00:00
Mudit Agarwal
b2c9c589aa util: fix tracevol.py to use namespace argument
tracevol.py accepts namespace argument but doesn't use it currently, fixing the same.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-08-03 07:14:02 +00:00
Niels de Vos
74ba85f87b ci: add "make run-e2e"
The keeps the standard arguments for e2e testing in a single location
instead of spread over multiple files and CI jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
ff94ba282c ci: deploy rook with mon_warn_on_pool_no_redundancy in ceph.conf
In test environments the default pool size is set to 1, so there is no
redundancy. This causes recent Ceph versions to complain with
HEALTH_WARN as POOL_NO_REDUNDANCY get set.

By disabling the mon_warn_on_pool_no_redundancy option in ceph.conf, the
warning is not reported and the cluster is marked HEALTHY.

See-also: rook/rook#5925
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
fb60f66178 ci: use the host /sbin/losetup in minikube VM
minikube has /sbin/losetup from Busybox, and that does not work with
raw-block PVCs. Use the losetup executable from the host in the VM
instead.

See-also: kubernetes/minikube#8284
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
230dd67752 ci: increase memory in the minikube VM
While testing with tehj default 3000 MB RAM in the minikube VM, creating
a encrypted RBD volume fails because 'cryptsetup' gets killed:

[  766.072585] Out of memory: Kill process 18497 (cryptsetup) score 1182 or sacrifice child
[  766.072589] Killed process 18497 (cryptsetup) total-vm:863136kB, anon-rss:510336kB, file-rss:10788kB, shmem-rss:0kB
[  766.072688] oom_reaper: reaped process 18497 (cryptsetup), now anon-rss:510336kB, file-rss:10780kB, shmem-rss:0kB

Using 4 GB RAM should prevent this from occuring.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00