Commit Graph

100 Commits

Author SHA1 Message Date
Niels de Vos
04934a40e5 ci: start minikube with --force to allow running as root
When starting minikube as root with --driver=kvm2, minikube complains
that this is not the right thing to do. However, in the CentOS CI we
really want to run as root, as that makes the scripts simpler.

Add the --force option while starting, so that minikube does not abort
anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Niels de Vos
43500fd6b8 ci: disable storage addons when starting minikube
Storage providers and the default storage class is not needed for
Ceph-CSI testing. In order to reduce resources and potential conflicts
between storage plugins, disable them.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Humble Chirammal
9e0589cf12 ci: fix rook cluster version fetching
As part of https://github.com/ceph/ceph-csi/pull/1237/ there was
a patching enabled for the ceph cluster deployed, however due to
an error in the version fetching logic, the patching was not applied

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-07-24 09:55:04 +00:00
Yug
94b745d573 ci: disable goerr113 linter
goerr113 linter checks the errors handling expressions.
It warns about using wrapped static errors in place
of dynamic at multiple places.

Disabled the linter as of now to avoid regression,
and this need to be handled in a seperate issue.
Tracker Issue: #1227

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-21 08:36:24 +00:00
Yug
9e435db454 ci: disable nestif linter
The `nestif` linter reports deeply nested if statements.

Disabled `nestif` as of now to avoid regression and
needs to addressed seperately.
Tracking Issue: #1229

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-07-21 08:36:24 +00:00
Niels de Vos
cb7ab307dd ci: disable 'testpackage' linter in golangci
The 'testpackage' linter recommends "black box" testing, which prevents
testing internal/non-exported functions from being tested. We have tests
that *do* test non-exported functions and types.

Disabling the linter allows us to test non-exported types and functions.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-21 08:36:24 +00:00
Humble Chirammal
9cb9020e2e ci: update e2e ceph cluster version to 14.2.10
Rook version is currently 1.1.7 in our e2e deployment which brings 14.2.4 version
of ceph cluster. To support cephfs snapshot e2e, we need latest version of Ceph Cluster
in E2E. Rook 1.2.7 is good enough which on patching bring up ceph 14.2.10 cluster.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-07-20 15:25:49 +00:00
Madhu Rajanna
f05c9a6a93 ci: fix psp issue in minikube latest version
With minikube versions greater than 1.6.2 and less than 1.11.1, the YAML files
minikube path will not be automatically applied to the cluster. we will get
errors during bootstrap of the cluster if the admission controller is enabled.

To use Pod Security Policies with these versions of minikube, first start a
cluster without the `PodSecurityPolicy` admission controller enabled.

Next, apply the psp yaml. and stop the cluster and then restart it
with the admission controller enabled.

```
minikube start
kubectl apply -f /path/to/psp.yaml
minikube stop
minikube start --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy
```

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-10 09:47:21 +00:00
Madhu Rajanna
9d1c91f3c0 ci: Pass on the arguments to E2E
Pass on the arugments which are sent when
calling the scripts to the E2E.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-09 06:42:28 +00:00
Madhu Rajanna
db1b5f9c55 e2e: move deploy timeout to build.env
moved deploy timeout from travis
scripts to build.env

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
5838b08edd e2e: Move e2e timeout to build.env
Moved e2e test timeout to build.env

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Madhu Rajanna
f946efb7b7 e2e: set maxsnapshotsonimage during helm install
as we need to test the maxsnapshotsonimage we
need to set the limit to minimal value which we
can test in CI as the default limit is 450,which
cannot be tested in CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-06 10:08:31 +00:00
Niels de Vos
f9f9fba26c ci: generate golangci.yml with correct CEPH_VERSION
When building against go-ceph, the most recent version of Ceph is
assumed to be available (currently Octopus). In case an older version of
the development packages is installed, building go-ceph will fail.

Golangci-lint does not accept the `-tags nautilus` parameter like other
Golang tools. Instead, the build-constraints need to be configured in a
confguration file.

This change takes care of the following:
- move the current scripts/golangci.yml to a template
- add the @@CEPH_VERSION@@ substitute
- generate the configuration file when needed

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-02 14:24:02 +00:00
Madhu Rajanna
37eb94e929 e2e: increase e2e timeout
in Travis CI the e2e tests are timing
out, 30 minutes seems less now for E2E
testing, increasing the timeout for 40
minutes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Niels de Vos
c02413a663 build: Add -tags=nautilus to go test and go build
Build `GO_TAGS` based on the `CEPH_VERSION` from build.env. In case the
version is non-empty, pass `-tags=${CEPH_VERSION}` to any of the go
commandline and script that call go programs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-30 07:55:37 +00:00
Yug Gupta
306d5b1da0 ci: add pylint as part of CI
add pylint to catch static issues of python
files in the repo.

User can now run make lint-py for pylint
check on python files.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-30 04:43:45 +00:00
Niels de Vos
3cc791c7c4 ci: document TEST_COVERAGE and GO_COVER_DIR in buid.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
4fd973b924 build: use BASE_IMAGE from build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
f1da7d9bd1 ci: move minikube variables to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
21946ae345 ci: move HELM_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
079459c247 build: move GOSEC_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
bc2ac1ccac build: move GOLANGCI_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
f83a065c8a build: move GOLANG_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
John Mulligan
d32cef5b9f ci: remove comment referring to a function that no longer exists
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-06-22 15:41:00 +00:00
John Mulligan
e111f2b613 scripts: add -mod=vendor to go run in check-env.sh
Without -mod=vendor running go run in this script may take more
resources than needed to execute. This also makes it consistent go build
(and alike) are invoked in the other scripts and the Makefile.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-06-17 04:58:32 +00:00
Yug Gupta
8b606f538f build: add check for ceph development headers
The script checks for the ceph development headers.
In case required packages are not found, script
suggests to run containerized build.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-10 09:49:55 +00:00
Niels de Vos
81b17c5157 ci: pass target filename of /etc/resolv.conf in case it is a symlink
In some Linux distributions the /etc/resolv.conf file is a symlink. This
file gets included in the Kubernetes containers and will be used for
resolving hostnames. By including the symlink, it is possible that that
target file is not available in the container(s). This will cause
problems when resolving hostnames, and Kubernetes will not get deployed.

The default minikube VM provides /run/systemd/resolve/resolv.conf, with
/etc/resolv.conf being a symlink. Therefor, it is needed to pass the
`--extra-config=kubelet.resolv-conf=..` parameter to `kubeadm`.

In case minikube is started with `--vm-driver=none` and
/run/systemd/resolve/resolv.conf does not exist, the local
/etc/resolv.conf will be used for inclusion in the Kubelet container. If
this is a symlink, the final destination should get passed with
`--extra-config=kubelet.resolv-conf=..` so that a working hostname
resolution configuration is available in the container.

Updates: #1121
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-05 08:57:19 +00:00
Mehdy Khoshnoody
79cef18f8f ci: Use bionic dist for test environments
As bionic uses a newer version of linux we'd also be using
a newer version of krbd.

Signed-off-by: Mehdy Khoshnoody <mehdy.khoshnoody@gmail.com>
2020-05-29 10:08:24 +00:00
Niels de Vos
aeb5c75da7 ci: install commitlint in the test container-image
The `commitlint` command can be used to verify the subject of commit
messages, so it is added to the $PATH.

See-also: https://commitlint.js.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 18:20:29 +00:00
Madhu Rajanna
1f13692000 ci: skip snapshot E2E if kube<1.17+
snapshot beta CRD wont work if the
kubernetes version is less than 1.17.0
as the snapshot CRD wont be installed
we cannot test the snapshot,so disabling
it if the kube version is less than 1.17

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
538b854853 script: Delete snapshot CRD created by cephcsi in rook
when we deploy rook+cephcsi for E2E, the
external-snapshotter in cephcsi deployed by
rook will create the CRD, we need to delete
the crd created by external-snapshotter.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
13438e8369 ci: Update travis functional tests
Updated travis functional tests to
install snapshot-controller for E2E

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
c533d0125a script: Add script to install snapshot-controller
Added script and snapshot-controller PSP
file to install snapshot-controller

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
3482cb7091 ci: updated shell script to run individual test
Earlier we were running all the linter for non-go
files in one short, this wont be helpful for the
users who want to run particular tests.

now the Makefile as different target to
run separate lint test for different type
of non-go files.

Fixes: #979

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-14 16:54:19 +00:00
Niels de Vos
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
Niels de Vos
99dabca319 add build step logging to multi-arch build
With extra logging, there is no need to call `travis_wait` anymore. In
addition the `travis_wait` command blocks output, so the build steps are
not reported until the script finishes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
5a4517e38e move build-multi-arch-image.sh to scripts/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
fd62a5897a tests: allow external-sources for shellcheck
While introducing scripts/build_step.inc.sh the tests start to fail as
the script is included and each shell script is tested separately. By
adding the option --external-sources to shellcheck, the related warnings
are not reported.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
52fa4f10da scripts: add build_step logging functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
32839948ef cleanup: move pkg/ to internal/
The internal/ directory in Go has a special meaning, and indicates that
those packages are not meant for external consumption. Ceph-CSI does
provide public APIs for other projects to consume. There is no plan to
keep the API of the internally used packages stable.

Closes: #903
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 11:00:59 +00:00
Madhu Rajanna
10eec1f39c Create Namespace for E2E
create ns before the helm create to avoid
`Error: create: failed to create: namespaces
"xxx" not found` issue

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 04:43:04 +00:00
Madhu Rajanna
86500c06c3 Helm3: update scripts to install helm3
Updated scripts to install helm3

fixes #920

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 04:43:04 +00:00
Niels de Vos
2c6c0decbe containerized-build: set WORKDIR and use in make target
By setting the WORKDIR in the container image, there is no need to pass
it on the commandline in the Makefile. This makes the line for the make
target a little cleaner.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00
Niels de Vos
ceef252a24 containerized-build: use GOROOT variable everywhere
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00
Niels de Vos
f942e2175a containerized-build: use dnf instead of yum
The Ceph base image moved to CentOS 8 which uses dnf. Use that instead
of yum.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00
Niels de Vos
3c6054ddf9 add make target for containerized tests
`make containerized-test` has been added as a make target. This runs the
'make test' target in a container. All test dependencies are installed
in the container once, and the container is kept around for running
`make containerized-test` subsequently.

The test container is based on Fedora:latest so that all test tools get
easily installed and are available in a recent version. The production
container is based on the Ceph container which has CentOS as Operating
System and therefor a more stable (too old) toolset.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 05:26:32 +00:00
Niels de Vos
58c2a3f15f minikube: fix ShellCheck errors
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 05:26:32 +00:00
Madhu Rajanna
6998c0d141 Fix typo in golangci config file
This PR fixes the typo in golang CI config file

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-15 09:18:47 +00:00
ShyamsundarR
1a8f8e3c24 Add support for erasure coded pools
This commit adds support to mention dataPool parameter for the
topology constrained pools in the StorageClass, that can be
leveraged to mention erasure coded pool names to use for RBD
data instead of the replica pools.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
e73921f268 Add e2e tests for topology based provisioning
- This commit adds tests only for RBD, as CephFS still needs
an enhancement in CephFS subvolume commands to effectively use
topology based provisioning

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00