Commit Graph

75 Commits

Author SHA1 Message Date
Niels de Vos
f7c6b2bb72 ci: always fetch GIT_SINCE for commitlint
When running the commitlint CI job, the branch that the PR is based on
may not be available. That makes it impossible for commitlint to detect
the changes between the HEAD of the branch, and the commits in the PR.

By fetching GIT_SINCE unconditionally, commitlint should be able to
detect the changes and only run the tests against the commits that were
added through the PR.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-10 09:57:06 +00:00
Niels de Vos
65a0e5c40b ci: do not run commitlint for admin user/group
commitlint should not run automatically on PRs, regardless who posted
the PR. Everyone should be able to trigger a run of the job with `/test
commitlint` anyway.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-07 14:15:51 +00:00
Niels de Vos
2e2f69e04a ci: run commitlint when requested, not containeriized-tests
The jobs/commitlint.yaml refered to the containerized-test.groovy
script, not the commitlint.groovy script.

Make sure to run the righ job when requested. Add some comments to make
things a little clearer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-07 14:15:51 +00:00
Niels de Vos
120bbc66ab ci: add commitlint fallback job
Each PR gets tested by the commitlint app (which is deprecated). On
occasion the app does not pickup the change in a PR, and the only way to
start a retest is to re-push the PR. All tests will then get executed
again, even if only commitlint was needed.

The new commitlint CI job introduces '/retest commitlint' and runs in
the CentOS CI like other Jenkins jobs. However, the job does not start
automatically on each (updated) PR, but needs manual starting with the
'/test commitlint' or '/retest commitlint' comments. The normal way of
running commitlint is with the app (at least for now).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-07 12:52:05 +00:00
Niels de Vos
c0943efc64 ci: pass strings with quotes over ssh in mini-e2e-helm
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 12:32:32 +00:00
Niels de Vos
a44bb6fd98 ci: create test namespace before "helm install ceph-csi"
Without the namespace, Helm fails to install Ceph-CSI.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 10:24:56 +00:00
Madhu Rajanna
78976616ac ci: add single quotes for namespace var
added single quotes for namespace variable

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-05 09:41:47 +00:00
Madhu Rajanna
1978e9a9f1 ci: fix namespace variable
define a new variable with def for namespace

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-05 09:41:47 +00:00
Niels de Vos
d6b0fc37f1 ci: add missing " in mini-e2e-helm script
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 07:52:33 +00:00
Niels de Vos
020d3eb212 ci: add mini-e2e-helm Jenkins job
The mini-e2e-helm.groovy script is mostly a copy of mini-e2e.groovy that
does the deployment through the e2e.test executable. The new script
installs Helm and deploys Ceph-CSI through that.

Once the e2e tests have successfully finished, the deployment of
Ceph-CSI is removed again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-05 05:36:49 +00:00
Niels de Vos
3d7784f75b ci: give minikube 8GB RAM on CentOS bare-metal systems
All bare-metal systems in the CentOS CI have 16GB or more RAM. To
improve performance of the minikube VM, give is 8GB RAM.

See-also: https://wiki.centos.org/QaWiki/PubHardware
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-04 07:19:39 +00:00
Yug
4f4513fa0a ci: Correct the stage name
Curently the stage name directly
prints the name of the variable
in place of substituting it.
This is a fix for that issue.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-04 04:02:18 +00:00
Yug
a71785d0f2 ci: use double quotes for variables
Using double quotes as variables are
expanded inside them.
The script fails currently as it is
unable to expand the variables.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-03 17:42:15 +00:00
Niels de Vos
0f811e9735 ci: jjb-deploy should have a "deployment" stage
The stage is called "validate" at the moment, this is a copy/paste
mistake from the jjb-validate job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 15:36:47 +00:00
Niels de Vos
86448395a5 ci: job name should not contain a "/"
The "/" in the name may have a conflict with the existing "mini-e2e" job
name. Rename the new generated jobs to "mini-e2e_k8s-1.x.y", with a "_"
instead of "/".

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 16:58:15 +02:00
Niels de Vos
322698a313 ci: pass ghprbPullId as parameter to jjb-validate job
Currently the jjb-validate job always checks out the ci/centos branch,
not the contents of the PR. This makes testing PRs unreliable.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 16:50:14 +02:00
Niels de Vos
1fb77b0626 ci: correctly indent parameters for mini-e2e job
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 16:50:14 +02:00
Niels de Vos
1de6b0bba2 ci: pass Kubernetes version as parameter to mini-e2e job
Move the mini-e2e job into a template-job and generate two jobs out of
it: mini-e2e/k8s-1.17.8 and mini-e2e/k8s-1.18.5

By passing the k8s_version as variable to the job-template, and placing
it in the parameters for the mini-e2e.groovy script, all hard-coded
occurences of the Kubernetes version can be replaced by the
{k8s_version} placeholder.

See-also: https://jenkins-job-builder.readthedocs.io/en/latest/definition.html#job-template
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 14:12:25 +00:00
Niels de Vos
8fbeefa989 ci: use ceph-csi:ci/centos for fetching job scripts
Commit f5cba3aaa8 added the mini-e2e job, but still referred to the
temporary location that was used for testing the job. As everything is
available in the ceph-csi:ci/centos repository:branch, there is no need
to refer to the temporary location.

Reported-by: Yug <yuggupta27@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 11:22:13 +00:00
Niels de Vos
d8f098fb47 ci: disable debug sleep on failure of mini-e2e
While debugging issues with the job itself, a sleep has been very
useful. PRs that have been rebased on the master branch contain all the
deployment fixes that are needed for the job to pass. There is no need
anymore to run into the long sleep when the job fails.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 10:32:11 +00:00
Niels de Vos
96dcc08223 ci: use ci/centos/mini-e2e/k8s-1.18 as context for mini-e2e job
Restarting the CI job can now be done with the following comments in a
PR:
- /test all
- /retest all
- /test ci/centos/mini-e2e
- /retest ci/centos/mini-e2e
- /test ci/centos/mini-e2e/k8s-1.18
- /retest ci/centos/mini-e2e/k8s-1.18

This is added as preparation for enabling running the mini-e2e job with
different Kubernetes versions.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 10:13:46 +00:00
Niels de Vos
1b43d5ab11 ci: use rook-1.3.8 in single-node-k8s.sh
The CentOS CI jobs that use single-node-k8s.sh will run minikube with
additional disks. That makes it possible for Rook 1.3 and newer to
deploy the Ceph cluster.

The current Travis CI jobs run minikube on the Travis provided
environment, without any extra disks. This prevents updating Rook for
all CI jobs at the moment.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 08:11:17 +00:00
Niels de Vos
f5cba3aaa8 ci: add mini-e2e Jenkins job
The new mini-e2e jobs does the following:
 - reserve a bare-metal machine
 - checkout the git repository with the PR
 - build used artifacts (container image and e2e.test executable)
 - deploy k8s and Rook in a minikube VM
 - run the e2e tests

With-contributions-from: Yug <yuggupta27@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 08:11:17 +00:00
Niels de Vos
d6ab44fdfa ci: use dnf on CentOS-8 systems
All bare-metal hosts that get reserved for tests are CentOS-8, so use
dnf.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 08:11:17 +00:00
Madhu Rajanna
7d9ef78057 ci: install make in centos ci
in centos 8 make doesnot come as
default we need to install it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-03 06:27:14 +00:00
Niels de Vos
954b2f42ea ci: request CentOS-8 bare-metal machines
Currently CentOS-7 machines were requested. CentOS-8 has been out for a
while now, and is stable for running manual jobs. There is nothing
preventing us from using CentOS-8 bare-metal machines for testing.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-01 11:24:35 +05:30
Niels de Vos
9db5d82327 ci: OCP4 deletes pods from batch jobs automatically
On the new OCP4 cluster, deleting the batch job results in deletion of
the pod as well. Deleting the pod when it does not exist anymore results
in a failure. Ignoring the potential failure makes the job pass again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
db0313a26b ci: update links to new OCP4 deployment
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
f04e8738f0 ci: use new OCP4 registry for jjb image
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
477a1225c9 ci: remove jjb Pods after job finished
Currently the number of Pods is growing in the OpenShift Console. The
pods are all Completed, so not running anymore, but there is also no
need to keep them around.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-10 13:09:14 +00:00
Niels de Vos
a089d8eaa1 ci: use correct refspec for fetching PRs in jjb-validate job
Also use https, instead of http for the repository.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
06ef71971d ci: re-enable jjb-validate job for PRs
With the jjb/session label on the Pod, tracking the job works again.
Therefor this test can be re-enabled.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
6a631e61cb ci: track jjb jobs by a jjb/session=<uuid> label
By using a template, it becomes possible to identify the Pod that has
been started by the Batch Job.

This prevents the script from getting the logs from an incorrect (old)
container.

Fixes: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
69b16771ae ci: do not automatically run jjb-validate on PRs
The jjb-validate job is not correct, it does not always return the
results of the PR.

Prevent triggering the job automatically, and only have it run when
someone leaves the '/test ci/centos/jjb-validate' comment.

Updates: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-28 07:36:06 +00:00
Niels de Vos
d2905081b3 ci: checkout the PR when running jjb-validate
Using the 'git' function in groovy does not allow checking out a
non-branch, so use the 'checkout' function instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 18:21:59 +02:00
Niels de Vos
8f4a556dcc ci: add properties for jobs/jjb-deploy.yaml
The 'scm' section also needs to be a list. Jenkins Jobs Builder does not
detect this while validating, but it cause an error while updating the
jobs in Jenkins.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 16:22:51 +02:00
Niels de Vos
6022807f52 ci: jenkins-jobs builder requires git:name set
In case there is no 'origin', jenkins-jobs fails with an error like:

    jenkins_jobs.errors.JenkinsJobsException: Must specify a url for git remote "origin"

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:25:51 +02:00
Niels de Vos
9ce8d20da9 ci: cd into the WORKDIR before running jenkins-jobs
The directory that gets constructed can be like
`/opt/build/deploy/..//jobs` and this causes jenkins-jobs to fail.
Enternig the WORKDIR and passing the local ./jobs directory to
jenkins-jobs makes things clearer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:09:08 +02:00
Niels de Vos
7a7d45cd67 doc: correct links in ci/centos README.md
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
650d749d28 ci: move the Jenkins Job Builder .yaml files under jobs/
There is a new .commitlintrc.yml file in the root of the repository.
This causes jenkins-job-builder to complains that the file is not
confirm the Jenkins Job Builder format/specification.

    jenkins_jobs.errors.JenkinsJobsException: The topmost collection in file '/opt/build/.commitlintrc.yml' must be a list, not a <class 'collections.OrderedDict'>

Move all Jenkins Job Builder .yaml files to jobs/ so that all the
Jenkins job configurations are kept together without other files.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
b5036489c8 ci: checkout PR with modified scripts for testing
The `git` command does not allow checking out a non-branch (like a PR),
so it is needed to use `checkout` instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
6e3c1dadae ci: add --base=<branch> to checkout a base branch
When checking out a PR for a non-master branch, `git clone` should
download the last commit of the branch. Adding a `--base=..` option to
pass the cloning of a selected branch, instead of `master`.

We also want to fetch all commits in the PR, so they can get tested with
`commitlint`. Only fetch --depth=1 the initial clone, but fetch
everything in the PR itself.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
b6b517deb2 ci: add job for checking ci/centos contents
This job runs 'make' on the ci/centos branch (or Pull Request) so that
all MarkDown, shell-scripts and yaml files are validated.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
d66b31edf3 ci: add CentOS CI job for jjb-validation
This job checks if the Jenkins Job Builder .yaml files are valid.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
67a7b4e8e2 ci: automatically deploy Jenkins jobs after merge
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
65a2794e60 ci: add 'make test' target to validate CI job scripts
This merges several files from the master branch into ci/centos so that
the scripts, MarkDown and yaml files can be checked.

Run 'make' or 'make test' to validate the contents of this branch.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
1f751fd07a ci: address MarkDown linter issues in README.md
./README.md:51: MD012 Multiple consecutive blank lines
./README.md:10: MD025 Multiple top level headers in the same document

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
bb0bd4004a ci: address yamllint issues in jjb-deploy.yaml
18:5      error    wrong indentation: expected 6 but found 4  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
0dc179a2c9 ci: address yamllint issues in jjb-validate.yaml
20:11     error    wrong indentation: expected 12 but found 10  (indentation)
  22:11     error    wrong indentation: expected 12 but found 10  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
0be9362216 ci: address yamllint issues in deploy/jjb-validate.yaml
16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00