Commit Graph

302 Commits

Author SHA1 Message Date
Niels de Vos
d6ab44fdfa ci: use dnf on CentOS-8 systems
All bare-metal hosts that get reserved for tests are CentOS-8, so use
dnf.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-03 08:11:17 +00:00
Madhu Rajanna
7d9ef78057 ci: install make in centos ci
in centos 8 make doesnot come as
default we need to install it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-03 06:27:14 +00:00
Niels de Vos
954b2f42ea ci: request CentOS-8 bare-metal machines
Currently CentOS-7 machines were requested. CentOS-8 has been out for a
while now, and is stable for running manual jobs. There is nothing
preventing us from using CentOS-8 bare-metal machines for testing.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-01 11:24:35 +05:30
Niels de Vos
9db5d82327 ci: OCP4 deletes pods from batch jobs automatically
On the new OCP4 cluster, deleting the batch job results in deletion of
the pod as well. Deleting the pod when it does not exist anymore results
in a failure. Ignoring the potential failure makes the job pass again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
db0313a26b ci: update links to new OCP4 deployment
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
f04e8738f0 ci: use new OCP4 registry for jjb image
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
477a1225c9 ci: remove jjb Pods after job finished
Currently the number of Pods is growing in the OpenShift Console. The
pods are all Completed, so not running anymore, but there is also no
need to keep them around.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-10 13:09:14 +00:00
Niels de Vos
a089d8eaa1 ci: use correct refspec for fetching PRs in jjb-validate job
Also use https, instead of http for the repository.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
06ef71971d ci: re-enable jjb-validate job for PRs
With the jjb/session label on the Pod, tracking the job works again.
Therefor this test can be re-enabled.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
6a631e61cb ci: track jjb jobs by a jjb/session=<uuid> label
By using a template, it becomes possible to identify the Pod that has
been started by the Batch Job.

This prevents the script from getting the logs from an incorrect (old)
container.

Fixes: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
69b16771ae ci: do not automatically run jjb-validate on PRs
The jjb-validate job is not correct, it does not always return the
results of the PR.

Prevent triggering the job automatically, and only have it run when
someone leaves the '/test ci/centos/jjb-validate' comment.

Updates: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-28 07:36:06 +00:00
Niels de Vos
d2905081b3 ci: checkout the PR when running jjb-validate
Using the 'git' function in groovy does not allow checking out a
non-branch, so use the 'checkout' function instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 18:21:59 +02:00
Niels de Vos
8f4a556dcc ci: add properties for jobs/jjb-deploy.yaml
The 'scm' section also needs to be a list. Jenkins Jobs Builder does not
detect this while validating, but it cause an error while updating the
jobs in Jenkins.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 16:22:51 +02:00
Niels de Vos
6022807f52 ci: jenkins-jobs builder requires git:name set
In case there is no 'origin', jenkins-jobs fails with an error like:

    jenkins_jobs.errors.JenkinsJobsException: Must specify a url for git remote "origin"

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:25:51 +02:00
Niels de Vos
9ce8d20da9 ci: cd into the WORKDIR before running jenkins-jobs
The directory that gets constructed can be like
`/opt/build/deploy/..//jobs` and this causes jenkins-jobs to fail.
Enternig the WORKDIR and passing the local ./jobs directory to
jenkins-jobs makes things clearer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:09:08 +02:00
Niels de Vos
7a7d45cd67 doc: correct links in ci/centos README.md
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
650d749d28 ci: move the Jenkins Job Builder .yaml files under jobs/
There is a new .commitlintrc.yml file in the root of the repository.
This causes jenkins-job-builder to complains that the file is not
confirm the Jenkins Job Builder format/specification.

    jenkins_jobs.errors.JenkinsJobsException: The topmost collection in file '/opt/build/.commitlintrc.yml' must be a list, not a <class 'collections.OrderedDict'>

Move all Jenkins Job Builder .yaml files to jobs/ so that all the
Jenkins job configurations are kept together without other files.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
b5036489c8 ci: checkout PR with modified scripts for testing
The `git` command does not allow checking out a non-branch (like a PR),
so it is needed to use `checkout` instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
6e3c1dadae ci: add --base=<branch> to checkout a base branch
When checking out a PR for a non-master branch, `git clone` should
download the last commit of the branch. Adding a `--base=..` option to
pass the cloning of a selected branch, instead of `master`.

We also want to fetch all commits in the PR, so they can get tested with
`commitlint`. Only fetch --depth=1 the initial clone, but fetch
everything in the PR itself.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
b6b517deb2 ci: add job for checking ci/centos contents
This job runs 'make' on the ci/centos branch (or Pull Request) so that
all MarkDown, shell-scripts and yaml files are validated.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
d66b31edf3 ci: add CentOS CI job for jjb-validation
This job checks if the Jenkins Job Builder .yaml files are valid.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
67a7b4e8e2 ci: automatically deploy Jenkins jobs after merge
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
65a2794e60 ci: add 'make test' target to validate CI job scripts
This merges several files from the master branch into ci/centos so that
the scripts, MarkDown and yaml files can be checked.

Run 'make' or 'make test' to validate the contents of this branch.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
1f751fd07a ci: address MarkDown linter issues in README.md
./README.md:51: MD012 Multiple consecutive blank lines
./README.md:10: MD025 Multiple top level headers in the same document

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
bb0bd4004a ci: address yamllint issues in jjb-deploy.yaml
18:5      error    wrong indentation: expected 6 but found 4  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
0dc179a2c9 ci: address yamllint issues in jjb-validate.yaml
20:11     error    wrong indentation: expected 12 but found 10  (indentation)
  22:11     error    wrong indentation: expected 12 but found 10  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
0be9362216 ci: address yamllint issues in deploy/jjb-validate.yaml
16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
52c5c0a32d ci: address yamllint issues in jjb-deploy.yaml
./deploy/jjb-deploy.yaml
  16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)
  26:9      error    wrong indentation: expected 10 but found 8  (indentation)
  30:7      error    wrong indentation: expected 8 but found 6  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
311813448c ci: fix yamllint issues in jjb-buildconfig.yaml
./deploy/jjb-buildconfig.yaml
  10:3      error    wrong indentation: expected 4 but found 2  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
fe738e9d9f cleanup: remove trailing spaces in prepare.sh
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
1cb2eef010 ci: address ShellCheck issues in deploy/jjb.sh
In ./deploy/jjb.sh line 11:
	oc get pod/${1} --no-headers -o=jsonpath='{.status.phase}'
                   ^--^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 29:
cd $(dirname ${0})
^----------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^-------------^ SC2046: Quote this to prevent word splitting.
             ^--^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 31:
oc create -f jjb-${CMD}.yaml
                 ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 36:
	jjb_pod=$(oc get pods --no-headers -l job-name=jjb-${CMD} -o=jsonpath='{.items[0].metadata.name}')
                                                           ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 37:
	[ ${?} = 0 ] && [ -n "${jjb_pod}" ] && break
          ^--^ SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In ./deploy/jjb.sh line 46:
	[ ${?} = 0 ] && ( [ "${status}" = "Succeeded" ] || [ "${status}" = "Failed" ] ) && break
          ^--^ SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
                        ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

In ./deploy/jjb.sh line 54:
oc delete --wait -f jjb-${CMD}.yaml
                        ^----^ SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
f436143be1 ci: address ShellCheck issues in prepare.sh
In ./prepare.sh line 21:
if [ $? -ne 0 ]
     ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In ./prepare.sh line 28:
eval set -- ${opts}
            ^-----^ SC2086: Double quote to prevent globbing and word splitting.

In ./prepare.sh line 55:
        echo ${ref}
             ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./prepare.sh line 69:
git clone --depth=1 ${gitrepo} ${workdir}
                    ^--------^ SC2086: Double quote to prevent globbing and word splitting.
                               ^--------^ SC2086: Double quote to prevent globbing and word splitting.

In ./prepare.sh line 70:
cd ${workdir}
^-----------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^--------^ SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
5ce5eb5be6 ci: make prepare.sh use /bin/bash
There are a few non-POSIX shell extensions used. So use Bash to run the
script.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
c6874b6fe6 cleanup: remove unused make-containerized-build job
This job has been replaced by containerized-tests.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-25 17:22:11 +02:00
Niels de Vos
c304f333bf doc: update README.md for the containerized-tests job
The containerized-tests job is more advanced than, and replaced the
make-containerized-build job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-25 17:22:11 +02:00
Niels de Vos
8259fe953f ci: use parameters set by github-pull-request-builder
The parameters are set in the environment. It seems the best way to
check if the parameters are available, is with `params.<arg> == null`.

See-also: https://www.jenkins.io/doc/book/pipeline/getting-started/#global-variable-reference#
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-25 17:10:06 +02:00
Niels de Vos
acb7727c24 ci: enable concurrent jobs for containerized-tests
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-18 16:15:50 +02:00
Yug Gupta
748ef17e42 ci: retry if no machine is immediately available
To prevent the failure of job due to unavailability
of a machine immediately, retry mechanism is used.

If unable to reserve a machine, it will retry every
5 mins for 30 times to avoid job failure.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-18 16:14:16 +02:00
Yug Gupta
f6229ae237 ci: execute build and test stages in parallel
Execution of build and test stages in parallel
will reduce the job execution time.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-18 14:01:05 +02:00
Niels de Vos
d66ca07721 ci: trigger containerized-tests from GitHub PRs
These additional settings are required to trigger the job to run when a
GitHub PR gets created or updated. It is also possible to manually run
the job by leaving one of the following comments:

    /test ci/centos/containerized-tests
    /retest ci/centos/containerized-tests
    /test all
    /retest all

The 'all' has been added so that future jobs can catch this comment as
well and act on it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-18 11:34:28 +02:00
Niels de Vos
07679210da ci: add Jenkins configuration for containerized-tests job
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 17:03:27 +02:00
Yug Gupta
45cd8c311a ci: Add a stage to run containerized-test
Running containerized test prior to the build
stage to ensure that build starts only after
tests pass.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-15 17:01:26 +02:00
Niels de Vos
c76af3b20d ci: check for new PRs every 5 minutes
Add a cron schedule to check every 5 minutes for a new PR to run the
make-containerized-build job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 17:03:19 +02:00
Niels de Vos
b3a7e3953c ci: pass ghprPullId in make-containerized-build job
In case the environment variable is not set at all (as opposed to
empty), teh following error is reported while running the job:

    groovy.lang.MissingPropertyException: No such property: ghprbPullId for class: groovy.lang.Binding

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 17:03:19 +02:00
Yug Gupta
adece898c6 ci: check for ghprbPullId env variable
The script checks for ghprbPullId env variable
and if found unset, runs the script for master
to support regular builds.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-11 16:06:25 +02:00
Yug Gupta
917ec1e2f1 prepare: exit when passed incorrect parameters
Now script does not continue execution
when an invalid argument is passed.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-07 10:31:57 +02:00
Yug Gupta
34b07b8ad9 Update Prepare.sh in centos CI branch
The script accepts three command-line arguments:

reference to the latest pr [optional]
working directory	   [optional]
git repository	           [optional]

Sample usage:
./prepare.sh --gitrepo=https://github.com/example --workdir=opt/build --ref=pull/123/head

Fixes: #968

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-06 13:34:01 +05:30
Niels de Vos
1f8b41204a ci: add xterm logging wrapper and timestamps
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-06 09:54:33 +02:00
Niels de Vos
538cdcdba4 add make-containerized-build job
The job does not have any parameters yet, it only build for the master
branch.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 14:15:04 +02:00
Niels de Vos
9e2a7e0c8d add jjb-<cmd> jobs for validating/deploying Jenkins jobs
Thse jobs run the deploy/jjb.sh script on a Jenkins Slave that is
configured with an environment for working with the `oc` command.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 14:15:04 +02:00