Commit Graph

143 Commits

Author SHA1 Message Date
Niels de Vos
6a631e61cb ci: track jjb jobs by a jjb/session=<uuid> label
By using a template, it becomes possible to identify the Pod that has
been started by the Batch Job.

This prevents the script from getting the logs from an incorrect (old)
container.

Fixes: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
69b16771ae ci: do not automatically run jjb-validate on PRs
The jjb-validate job is not correct, it does not always return the
results of the PR.

Prevent triggering the job automatically, and only have it run when
someone leaves the '/test ci/centos/jjb-validate' comment.

Updates: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-28 07:36:06 +00:00
Niels de Vos
d2905081b3 ci: checkout the PR when running jjb-validate
Using the 'git' function in groovy does not allow checking out a
non-branch, so use the 'checkout' function instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 18:21:59 +02:00
Niels de Vos
8f4a556dcc ci: add properties for jobs/jjb-deploy.yaml
The 'scm' section also needs to be a list. Jenkins Jobs Builder does not
detect this while validating, but it cause an error while updating the
jobs in Jenkins.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 16:22:51 +02:00
Niels de Vos
6022807f52 ci: jenkins-jobs builder requires git:name set
In case there is no 'origin', jenkins-jobs fails with an error like:

    jenkins_jobs.errors.JenkinsJobsException: Must specify a url for git remote "origin"

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:25:51 +02:00
Niels de Vos
9ce8d20da9 ci: cd into the WORKDIR before running jenkins-jobs
The directory that gets constructed can be like
`/opt/build/deploy/..//jobs` and this causes jenkins-jobs to fail.
Enternig the WORKDIR and passing the local ./jobs directory to
jenkins-jobs makes things clearer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:09:08 +02:00
Niels de Vos
7a7d45cd67 doc: correct links in ci/centos README.md
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
650d749d28 ci: move the Jenkins Job Builder .yaml files under jobs/
There is a new .commitlintrc.yml file in the root of the repository.
This causes jenkins-job-builder to complains that the file is not
confirm the Jenkins Job Builder format/specification.

    jenkins_jobs.errors.JenkinsJobsException: The topmost collection in file '/opt/build/.commitlintrc.yml' must be a list, not a <class 'collections.OrderedDict'>

Move all Jenkins Job Builder .yaml files to jobs/ so that all the
Jenkins job configurations are kept together without other files.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
b5036489c8 ci: checkout PR with modified scripts for testing
The `git` command does not allow checking out a non-branch (like a PR),
so it is needed to use `checkout` instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
6e3c1dadae ci: add --base=<branch> to checkout a base branch
When checking out a PR for a non-master branch, `git clone` should
download the last commit of the branch. Adding a `--base=..` option to
pass the cloning of a selected branch, instead of `master`.

We also want to fetch all commits in the PR, so they can get tested with
`commitlint`. Only fetch --depth=1 the initial clone, but fetch
everything in the PR itself.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
b6b517deb2 ci: add job for checking ci/centos contents
This job runs 'make' on the ci/centos branch (or Pull Request) so that
all MarkDown, shell-scripts and yaml files are validated.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
d66b31edf3 ci: add CentOS CI job for jjb-validation
This job checks if the Jenkins Job Builder .yaml files are valid.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
67a7b4e8e2 ci: automatically deploy Jenkins jobs after merge
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
65a2794e60 ci: add 'make test' target to validate CI job scripts
This merges several files from the master branch into ci/centos so that
the scripts, MarkDown and yaml files can be checked.

Run 'make' or 'make test' to validate the contents of this branch.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
1f751fd07a ci: address MarkDown linter issues in README.md
./README.md:51: MD012 Multiple consecutive blank lines
./README.md:10: MD025 Multiple top level headers in the same document

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
bb0bd4004a ci: address yamllint issues in jjb-deploy.yaml
18:5      error    wrong indentation: expected 6 but found 4  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
0dc179a2c9 ci: address yamllint issues in jjb-validate.yaml
20:11     error    wrong indentation: expected 12 but found 10  (indentation)
  22:11     error    wrong indentation: expected 12 but found 10  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
0be9362216 ci: address yamllint issues in deploy/jjb-validate.yaml
16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
52c5c0a32d ci: address yamllint issues in jjb-deploy.yaml
./deploy/jjb-deploy.yaml
  16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)
  26:9      error    wrong indentation: expected 10 but found 8  (indentation)
  30:7      error    wrong indentation: expected 8 but found 6  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
311813448c ci: fix yamllint issues in jjb-buildconfig.yaml
./deploy/jjb-buildconfig.yaml
  10:3      error    wrong indentation: expected 4 but found 2  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
fe738e9d9f cleanup: remove trailing spaces in prepare.sh
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
1cb2eef010 ci: address ShellCheck issues in deploy/jjb.sh
In ./deploy/jjb.sh line 11:
	oc get pod/${1} --no-headers -o=jsonpath='{.status.phase}'
                   ^--^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 29:
cd $(dirname ${0})
^----------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^-------------^ SC2046: Quote this to prevent word splitting.
             ^--^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 31:
oc create -f jjb-${CMD}.yaml
                 ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 36:
	jjb_pod=$(oc get pods --no-headers -l job-name=jjb-${CMD} -o=jsonpath='{.items[0].metadata.name}')
                                                           ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 37:
	[ ${?} = 0 ] && [ -n "${jjb_pod}" ] && break
          ^--^ SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In ./deploy/jjb.sh line 46:
	[ ${?} = 0 ] && ( [ "${status}" = "Succeeded" ] || [ "${status}" = "Failed" ] ) && break
          ^--^ SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
                        ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

In ./deploy/jjb.sh line 54:
oc delete --wait -f jjb-${CMD}.yaml
                        ^----^ SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
f436143be1 ci: address ShellCheck issues in prepare.sh
In ./prepare.sh line 21:
if [ $? -ne 0 ]
     ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In ./prepare.sh line 28:
eval set -- ${opts}
            ^-----^ SC2086: Double quote to prevent globbing and word splitting.

In ./prepare.sh line 55:
        echo ${ref}
             ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./prepare.sh line 69:
git clone --depth=1 ${gitrepo} ${workdir}
                    ^--------^ SC2086: Double quote to prevent globbing and word splitting.
                               ^--------^ SC2086: Double quote to prevent globbing and word splitting.

In ./prepare.sh line 70:
cd ${workdir}
^-----------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^--------^ SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
5ce5eb5be6 ci: make prepare.sh use /bin/bash
There are a few non-POSIX shell extensions used. So use Bash to run the
script.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
c6874b6fe6 cleanup: remove unused make-containerized-build job
This job has been replaced by containerized-tests.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-25 17:22:11 +02:00
Niels de Vos
c304f333bf doc: update README.md for the containerized-tests job
The containerized-tests job is more advanced than, and replaced the
make-containerized-build job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-25 17:22:11 +02:00
Niels de Vos
8259fe953f ci: use parameters set by github-pull-request-builder
The parameters are set in the environment. It seems the best way to
check if the parameters are available, is with `params.<arg> == null`.

See-also: https://www.jenkins.io/doc/book/pipeline/getting-started/#global-variable-reference#
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-25 17:10:06 +02:00
Niels de Vos
acb7727c24 ci: enable concurrent jobs for containerized-tests
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-18 16:15:50 +02:00
Yug Gupta
748ef17e42 ci: retry if no machine is immediately available
To prevent the failure of job due to unavailability
of a machine immediately, retry mechanism is used.

If unable to reserve a machine, it will retry every
5 mins for 30 times to avoid job failure.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-18 16:14:16 +02:00
Yug Gupta
f6229ae237 ci: execute build and test stages in parallel
Execution of build and test stages in parallel
will reduce the job execution time.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-18 14:01:05 +02:00
Niels de Vos
d66ca07721 ci: trigger containerized-tests from GitHub PRs
These additional settings are required to trigger the job to run when a
GitHub PR gets created or updated. It is also possible to manually run
the job by leaving one of the following comments:

    /test ci/centos/containerized-tests
    /retest ci/centos/containerized-tests
    /test all
    /retest all

The 'all' has been added so that future jobs can catch this comment as
well and act on it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-18 11:34:28 +02:00
Niels de Vos
07679210da ci: add Jenkins configuration for containerized-tests job
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 17:03:27 +02:00
Yug Gupta
45cd8c311a ci: Add a stage to run containerized-test
Running containerized test prior to the build
stage to ensure that build starts only after
tests pass.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-15 17:01:26 +02:00
Niels de Vos
c76af3b20d ci: check for new PRs every 5 minutes
Add a cron schedule to check every 5 minutes for a new PR to run the
make-containerized-build job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 17:03:19 +02:00
Niels de Vos
b3a7e3953c ci: pass ghprPullId in make-containerized-build job
In case the environment variable is not set at all (as opposed to
empty), teh following error is reported while running the job:

    groovy.lang.MissingPropertyException: No such property: ghprbPullId for class: groovy.lang.Binding

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 17:03:19 +02:00
Yug Gupta
adece898c6 ci: check for ghprbPullId env variable
The script checks for ghprbPullId env variable
and if found unset, runs the script for master
to support regular builds.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-11 16:06:25 +02:00
Yug Gupta
917ec1e2f1 prepare: exit when passed incorrect parameters
Now script does not continue execution
when an invalid argument is passed.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-07 10:31:57 +02:00
Yug Gupta
34b07b8ad9 Update Prepare.sh in centos CI branch
The script accepts three command-line arguments:

reference to the latest pr [optional]
working directory	   [optional]
git repository	           [optional]

Sample usage:
./prepare.sh --gitrepo=https://github.com/example --workdir=opt/build --ref=pull/123/head

Fixes: #968

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-06 13:34:01 +05:30
Niels de Vos
1f8b41204a ci: add xterm logging wrapper and timestamps
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-06 09:54:33 +02:00
Niels de Vos
538cdcdba4 add make-containerized-build job
The job does not have any parameters yet, it only build for the master
branch.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 14:15:04 +02:00
Niels de Vos
9e2a7e0c8d add jjb-<cmd> jobs for validating/deploying Jenkins jobs
Thse jobs run the deploy/jjb.sh script on a Jenkins Slave that is
configured with an environment for working with the `oc` command.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 14:15:04 +02:00
Niels de Vos
6d906b415f add scripts and configuration to deploy with Jenkins Job Builder
These files make it possible to
- build a container image that includes Jenkins Job Builder
- store the container image in the OpenShift ImageStream
- use the container to validate or deploy the Jenkins Jobs

More details about the files are included in the README.md.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 14:15:04 +02:00
Niels de Vos
5caafe7539 ci/centos: new branch for Jenkins jobs and configuration
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 13:31:17 +02:00