Commit Graph

111 Commits

Author SHA1 Message Date
Yug
23e0874ebf deploy: add wrapper function of kubectl
Add retries to prevent ci failure instantly.
Now, the command execution will retry upto
5 times, to avoid failures in some runs.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-07 09:28:24 +00:00
Niels de Vos
c578ab01a7 ci: pass HELM_VERSION when installing Helm in test container
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 07:43:10 +00:00
Niels de Vos
d4f18458de ci: include build.env variables when installing Helm
By default the install-helm.sh script uses "latest" as version for Helm.
Unfortunately this version does not exist. The HELM_VERSION variable is
already set in build.env, so source the configuration file as one of the
first actions in install-helm.sh.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 07:43:10 +00:00
Niels de Vos
b9d1f16360 ci: make number of CPUs for minikube VM configurable
By default minikube uses 2 CPUs, which might be too little for some of
the tests. When not passing a CPUS environment variable, use all CPUs
available on the system (detected with 'nproc').

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-04 08:46:17 +00:00
Niels de Vos
74ba85f87b ci: add "make run-e2e"
The keeps the standard arguments for e2e testing in a single location
instead of spread over multiple files and CI jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
ff94ba282c ci: deploy rook with mon_warn_on_pool_no_redundancy in ceph.conf
In test environments the default pool size is set to 1, so there is no
redundancy. This causes recent Ceph versions to complain with
HEALTH_WARN as POOL_NO_REDUNDANCY get set.

By disabling the mon_warn_on_pool_no_redundancy option in ceph.conf, the
warning is not reported and the cluster is marked HEALTHY.

See-also: rook/rook#5925
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
fb60f66178 ci: use the host /sbin/losetup in minikube VM
minikube has /sbin/losetup from Busybox, and that does not work with
raw-block PVCs. Use the losetup executable from the host in the VM
instead.

See-also: kubernetes/minikube#8284
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
230dd67752 ci: increase memory in the minikube VM
While testing with tehj default 3000 MB RAM in the minikube VM, creating
a encrypted RBD volume fails because 'cryptsetup' gets killed:

[  766.072585] Out of memory: Kill process 18497 (cryptsetup) score 1182 or sacrifice child
[  766.072589] Killed process 18497 (cryptsetup) total-vm:863136kB, anon-rss:510336kB, file-rss:10788kB, shmem-rss:0kB
[  766.072688] oom_reaper: reaped process 18497 (cryptsetup), now anon-rss:510336kB, file-rss:10780kB, shmem-rss:0kB

Using 4 GB RAM should prevent this from occuring.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
f46fb13357 ci: run "kubectl cluster-info" through minikube
In case kubectl did not get installed (VM_DRIVER != none),
scripts/minikube.sh can fail when kubectl is not in the path. By running
the "kubectl cluster-info" command through minikube, the script will
succeed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
e774ebb7f9 ci: detect available minikube executable
In case there is a minikube executable in the $PATH already, use that
for all commands. If there is none, install_minikube() will place a
newly downloaded executable in /usr/local/bin which will be used by the
full pathname, so that commands as root without /usr/local/bin in the
$PATH will work.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
fc378ac74b ci: remove weird mkdir/ln command
The command fails when PWD=/. It is unclear what the command tries to
achieve. The next command does something more useful, although it can
maybe be removed as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
04934a40e5 ci: start minikube with --force to allow running as root
When starting minikube as root with --driver=kvm2, minikube complains
that this is not the right thing to do. However, in the CentOS CI we
really want to run as root, as that makes the scripts simpler.

Add the --force option while starting, so that minikube does not abort
anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
43500fd6b8 ci: disable storage addons when starting minikube
Storage providers and the default storage class is not needed for
Ceph-CSI testing. In order to reduce resources and potential conflicts
between storage plugins, disable them.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Humble Chirammal
9e0589cf12 ci: fix rook cluster version fetching
As part of https://github.com/ceph/ceph-csi/pull/1237/ there was
a patching enabled for the ceph cluster deployed, however due to
an error in the version fetching logic, the patching was not applied

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-07-24 09:55:04 +00:00
Yug
94b745d573 ci: disable goerr113 linter
goerr113 linter checks the errors handling expressions.
It warns about using wrapped static errors in place
of dynamic at multiple places.

Disabled the linter as of now to avoid regression,
and this need to be handled in a seperate issue.
Tracker Issue: #1227

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-21 08:36:24 +00:00
Yug
9e435db454 ci: disable nestif linter
The `nestif` linter reports deeply nested if statements.

Disabled `nestif` as of now to avoid regression and
needs to addressed seperately.
Tracking Issue: #1229

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-21 08:36:24 +00:00
Niels de Vos
cb7ab307dd ci: disable 'testpackage' linter in golangci
The 'testpackage' linter recommends "black box" testing, which prevents
testing internal/non-exported functions from being tested. We have tests
that *do* test non-exported functions and types.

Disabling the linter allows us to test non-exported types and functions.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-21 08:36:24 +00:00
Humble Chirammal
9cb9020e2e ci: update e2e ceph cluster version to 14.2.10
Rook version is currently 1.1.7 in our e2e deployment which brings 14.2.4 version
of ceph cluster. To support cephfs snapshot e2e, we need latest version of Ceph Cluster
in E2E. Rook 1.2.7 is good enough which on patching bring up ceph 14.2.10 cluster.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-07-20 15:25:49 +00:00
Madhu Rajanna
f05c9a6a93 ci: fix psp issue in minikube latest version
With minikube versions greater than 1.6.2 and less than 1.11.1, the YAML files
minikube path will not be automatically applied to the cluster. we will get
errors during bootstrap of the cluster if the admission controller is enabled.

To use Pod Security Policies with these versions of minikube, first start a
cluster without the `PodSecurityPolicy` admission controller enabled.

Next, apply the psp yaml. and stop the cluster and then restart it
with the admission controller enabled.

```
minikube start
kubectl apply -f /path/to/psp.yaml
minikube stop
minikube start --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy
```

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
9d1c91f3c0 ci: Pass on the arguments to E2E
Pass on the arugments which are sent when
calling the scripts to the E2E.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-09 06:42:28 +00:00
Madhu Rajanna
db1b5f9c55 e2e: move deploy timeout to build.env
moved deploy timeout from travis
scripts to build.env

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
5838b08edd e2e: Move e2e timeout to build.env
Moved e2e test timeout to build.env

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
f946efb7b7 e2e: set maxsnapshotsonimage during helm install
as we need to test the maxsnapshotsonimage we
need to set the limit to minimal value which we
can test in CI as the default limit is 450,which
cannot be tested in CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Niels de Vos
f9f9fba26c ci: generate golangci.yml with correct CEPH_VERSION
When building against go-ceph, the most recent version of Ceph is
assumed to be available (currently Octopus). In case an older version of
the development packages is installed, building go-ceph will fail.

Golangci-lint does not accept the `-tags nautilus` parameter like other
Golang tools. Instead, the build-constraints need to be configured in a
confguration file.

This change takes care of the following:
- move the current scripts/golangci.yml to a template
- add the @@CEPH_VERSION@@ substitute
- generate the configuration file when needed

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-02 14:24:02 +00:00
Madhu Rajanna
37eb94e929 e2e: increase e2e timeout
in Travis CI the e2e tests are timing
out, 30 minutes seems less now for E2E
testing, increasing the timeout for 40
minutes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Niels de Vos
c02413a663 build: Add -tags=nautilus to go test and go build
Build `GO_TAGS` based on the `CEPH_VERSION` from build.env. In case the
version is non-empty, pass `-tags=${CEPH_VERSION}` to any of the go
commandline and script that call go programs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-30 07:55:37 +00:00
Yug Gupta
306d5b1da0 ci: add pylint as part of CI
add pylint to catch static issues of python
files in the repo.

User can now run make lint-py for pylint
check on python files.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-30 04:43:45 +00:00
Niels de Vos
3cc791c7c4 ci: document TEST_COVERAGE and GO_COVER_DIR in buid.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
4fd973b924 build: use BASE_IMAGE from build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
f1da7d9bd1 ci: move minikube variables to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
21946ae345 ci: move HELM_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
079459c247 build: move GOSEC_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
bc2ac1ccac build: move GOLANGCI_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
f83a065c8a build: move GOLANG_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
John Mulligan
d32cef5b9f ci: remove comment referring to a function that no longer exists
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-06-22 15:41:00 +00:00
John Mulligan
e111f2b613 scripts: add -mod=vendor to go run in check-env.sh
Without -mod=vendor running go run in this script may take more
resources than needed to execute. This also makes it consistent go build
(and alike) are invoked in the other scripts and the Makefile.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-06-17 04:58:32 +00:00
Yug Gupta
8b606f538f build: add check for ceph development headers
The script checks for the ceph development headers.
In case required packages are not found, script
suggests to run containerized build.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-10 09:49:55 +00:00
Niels de Vos
81b17c5157 ci: pass target filename of /etc/resolv.conf in case it is a symlink
In some Linux distributions the /etc/resolv.conf file is a symlink. This
file gets included in the Kubernetes containers and will be used for
resolving hostnames. By including the symlink, it is possible that that
target file is not available in the container(s). This will cause
problems when resolving hostnames, and Kubernetes will not get deployed.

The default minikube VM provides /run/systemd/resolve/resolv.conf, with
/etc/resolv.conf being a symlink. Therefor, it is needed to pass the
`--extra-config=kubelet.resolv-conf=..` parameter to `kubeadm`.

In case minikube is started with `--vm-driver=none` and
/run/systemd/resolve/resolv.conf does not exist, the local
/etc/resolv.conf will be used for inclusion in the Kubelet container. If
this is a symlink, the final destination should get passed with
`--extra-config=kubelet.resolv-conf=..` so that a working hostname
resolution configuration is available in the container.

Updates: #1121
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-05 08:57:19 +00:00
Mehdy Khoshnoody
79cef18f8f ci: Use bionic dist for test environments
As bionic uses a newer version of linux we'd also be using
a newer version of krbd.

Signed-off-by: Mehdy Khoshnoody <mehdy.khoshnoody@gmail.com>
2020-05-29 10:08:24 +00:00
Niels de Vos
aeb5c75da7 ci: install commitlint in the test container-image
The `commitlint` command can be used to verify the subject of commit
messages, so it is added to the $PATH.

See-also: https://commitlint.js.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 18:20:29 +00:00
Madhu Rajanna
1f13692000 ci: skip snapshot E2E if kube<1.17+
snapshot beta CRD wont work if the
kubernetes version is less than 1.17.0
as the snapshot CRD wont be installed
we cannot test the snapshot,so disabling
it if the kube version is less than 1.17

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
538b854853 script: Delete snapshot CRD created by cephcsi in rook
when we deploy rook+cephcsi for E2E, the
external-snapshotter in cephcsi deployed by
rook will create the CRD, we need to delete
the crd created by external-snapshotter.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
13438e8369 ci: Update travis functional tests
Updated travis functional tests to
install snapshot-controller for E2E

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
c533d0125a script: Add script to install snapshot-controller
Added script and snapshot-controller PSP
file to install snapshot-controller

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
3482cb7091 ci: updated shell script to run individual test
Earlier we were running all the linter for non-go
files in one short, this wont be helpful for the
users who want to run particular tests.

now the Makefile as different target to
run separate lint test for different type
of non-go files.

Fixes: #979

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-14 16:54:19 +00:00
Niels de Vos
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
Niels de Vos
99dabca319 add build step logging to multi-arch build
With extra logging, there is no need to call `travis_wait` anymore. In
addition the `travis_wait` command blocks output, so the build steps are
not reported until the script finishes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
5a4517e38e move build-multi-arch-image.sh to scripts/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
fd62a5897a tests: allow external-sources for shellcheck
While introducing scripts/build_step.inc.sh the tests start to fail as
the script is included and each shell script is tested separately. By
adding the option --external-sources to shellcheck, the related warnings
are not reported.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
52fa4f10da scripts: add build_step logging functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00