Commit Graph

37 Commits

Author SHA1 Message Date
Niels de Vos
6e38694a11 ci: the order of --allow-empty-variables for jenkins-jobs is important
Jenkins Jobs Builder fails like this, indicating the order of the
parameters is important:

```
$ jenkins-jobs update --allow-empty-variables --delete-old jobs
usage: jenkins-jobs [-h] [--conf CONF] [-l LOG_LEVEL] [--ignore-cache]
                    [--flush-cache] [--version] [--allow-empty-variables]
                    [--server SECTION] [--user USER] [--password PASSWORD]
                    {delete,delete-all,get-plugins-info,list,test,update} ...
jenkins-jobs: error: unrecognized arguments: --allow-empty-variables
```

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-01-30 18:14:58 +01:00
Niels de Vos
6baa675fa0 ci: allow missing RUN_DISPLAY_URL variable in Jenkins Jobs
The `RUN_DISPLAY_URL` is automatically set by Jenkins, but it seems the
latest Jenkins Jobs Builder update does not detect it automatically.
Run the JJB commands with `--allow-empty-variables`.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-01-30 18:08:18 +01:00
karthik-us
d14bba7c12 doc: Update/remove broken links
Updates few broken links, removes a non existent link and some nit-picks.

Signed-off-by: karthik-us <ksubrahm@redhat.com>
2023-07-04 14:58:17 +00:00
Niels de Vos
5e728f51b2 build: remove urllib3 downgrade in jjb container, use timeout instead
The upstream reported issue suggests adding a `timeout` value in the
Jenkins Jobs Builder configuration. This looks like a nicer workaround
than downgrading urllib3.

Fixes: #3946
See-also: https://storyboard.openstack.org/#!/story/2010752
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-06-28 12:48:12 +00:00
Niels de Vos
3592bbedbb build: use urllib3 < 2 with Jenkins jobs Builder
jenkins-job-builder does not seem to work with urllib3 2.x, force using
version 1.x as a workaround.

Updates: #3946
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-06-27 12:51:26 +00:00
Niels de Vos
a02b38ac27 ci: use CentOS Stream 9 for Jenkins Jobs Builder image
CentOS Stream 8 does not provide a module named 'dataclasses'. It seems
current versions of Jenkins Jobs Builder require that. Python 3.7
offers the module by default, so using a new base distribution with
Python 3.9 has the module too.

See-also: #3909
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-06-21 08:11:51 +00:00
Niels de Vos
cca994eb1a ci: run registry garbage collection every week
In the old OpenShift cluster the registry consumer 800+ GB of data. Once
running the garbage collection manually, the consumption reduced to a
little over 8GB. Let's be nice users of the infrastructure and run
garbage collection weekly.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-03-20 14:20:53 +00:00
Niels de Vos
b0e9eeba36 ci: correct parameter passing in checkout-repo.sh
ShellCheck failed with the following warning:

```
SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
git config --global --add safe.directory "${PWD}"
```

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-03-20 09:21:41 +00:00
Niels de Vos
8135680065 ci: migrate to new OpenShift cluster running on AWS
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-03-20 09:21:41 +00:00
Niels de Vos
234471af33 ci: allow /opt/build for git repository when deploying jobs
When Jenkins Jobs have been modified, they should get deployed in the
Jenkins environment. This seems to fail with the following error:

```
Initialized empty Git repository in /opt/build/.git/
fatal: detected dubious ownership in repository at '/opt/build'
To add an exception for this directory, call:

	git config --global --add safe.directory /opt/build
```

By marking /opt/build as a safe directory in the global git
configuration file, this problem should not occur anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2023-01-30 15:15:06 +01:00
Niels de Vos
ed95c4aa78 ci: use CentOS Stream 8 for infra containers
The container images for Jenkins Job Builder and mirroring images were
still using the EOL CentOS 8 base image. Rebuilding those
container-images fails as the CentOS mirrors do not provide the
repositories for installing package anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-04 03:12:34 +00:00
Madhu Rajanna
92c8f723f9 ci: use official CentOS container location
registry.centos.org is not officially
maintained by the CentOS infrastructure
team. The container images on quay.io are
the official once and we should use
those instead.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-11-19 07:51:31 +00:00
Rakshith R
e6c04e1414 ci: set ttlSecondsAfterFinished=10 to ensure job status can be checked
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 14:47:18 +00:00
Rakshith R
6255ec9a7a ci: update Template yamls apiVersion to template.openshift.io/v1
This was updated to prevent the following error.
Using non-groupfied API resources is deprecated and will be removed in a
future release, update apiVersion to "template.openshift.io/v1" for your
resource.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 14:47:18 +00:00
Niels de Vos
1c2974d49e ci: the "master" branch got renamed to "devel"
Closes: #1193
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-01 05:22:06 +00:00
Niels de Vos
7cac1f7609 ci: remove docker mirror configuration
Instead of using a mirror, the CI registry is now pupulated with
container images that get pulled and tagged as if they get from Docker
Hub or other locations.

This is more of a manual mirror, as the Docker Registry mirror
functionality is not flexible enough for our usecase (push images
without providing them on docker.io).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-26 16:26:57 +00:00
Niels de Vos
5ae8fb7c9b ci: add configuration for the proxy/mirror registry
This makes it possible to pull images from Docker Hub through the local
container image registry in the CI OpenShift deployment. The registry in
the CI is configured with the 'cephcsibot' account so that pulling
images is accounted towards the account, and not anonymous consumers
within the whole CentOS CI.

There should be no need to manually sync the images between the local
registry and Docker Hub anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 05:09:38 +00:00
Niels de Vos
49c6784c5c ci: deploy/jjb.sh should fail on wrong arguments
In case the '--cmd' option is missing, usage() will be called and the
script exits with 0. A missing option is a failure, so make usage()
return and handle the exit status when usage() got called instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 10:09:57 +00:00
Niels de Vos
45130996a0 ci: flatten images for internal consumption
There is no need to re-use different layers with the images that are
built. The separation of the layers causes more time to be used while
pulling the images, so flattening makes speeds up the CI jobs a little
(about 30 seconds from 2+ minutes).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 04:43:07 +00:00
liranmauda
c281511a11 ci: Add the ability to run JJB from private branch/fork
Add the ability to run JJB from private branch/fork

jjb-deploy.yaml:
- Add GIT_REPO parameter (not required) with default value

jjb.sh:
- Changing PS4 for more indicative debug prompt
- Adding flags to get the repo and branch

jjb-deploy.yaml:
- Calling the jjb.sh with the correct flags

Signed-off-by: liranmauda <liran.mauda@gmail.com>
2020-11-03 07:26:43 +00:00
Niels de Vos
5dc6045034 ci: add BuildConfig for ceph-csi container images
The BuildConfig can be imported in the OpenShift environment so that
Ceph-CSI images can be built automatically. These images will be cached
in a private container image registry, which can speed up CI jobs that
currently rebuild the images every time.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-30 13:07:58 +00:00
Niels de Vos
b5216592e2 ci: deploy a private container image registry
With a private container image registry, the images that CI jobs use can
be cached and re-used. This speeds up the CI jobs, as building the
container images takes up the majority of the runtime for some jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-30 13:07:58 +00:00
Niels de Vos
9a01ee2019 ci: remove old jobs when deploying
Old jobs stay behind (and active) when jobs get updated with new
versions. This mostly affects the mini-e2e jobs at the moment.

There is no need to keep old job around, so delete them while deploying
updates to jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-25 08:03:00 +00:00
Niels de Vos
01de4eacfc ci: restructure 'parameters' section for upgrade-tests
The section containing the paramaters does not seem to be structured
correctly, correct it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 10:30:38 +00:00
Niels de Vos
b1d7c83c16 ci: checkout FETCH_HEAD instead of GIT_REF
GIT_REF can point to a ref that it not part of a branch and can not
easily be checked out (like a GitHub PR). Checking out FETCH_HEAD after
fetching the GIT_REF works.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Niels de Vos
5f9222d7ee ci: run jjb-validate batch job against a PR
The batch job templates for OpenShift use a GIT_REF variable that is
used by the jobs to checkout the repository. This defaults to
"ci/centos", but it never is adjusted when jjb-validate runs against a
PR.

With the new GIT_REF environment variable in the jjb.sh script, the
variable can now be used to checkout a PR from gitHub and run the
validation against that.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Niels de Vos
9db5d82327 ci: OCP4 deletes pods from batch jobs automatically
On the new OCP4 cluster, deleting the batch job results in deletion of
the pod as well. Deleting the pod when it does not exist anymore results
in a failure. Ignoring the potential failure makes the job pass again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
f04e8738f0 ci: use new OCP4 registry for jjb image
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-24 14:11:42 +00:00
Niels de Vos
477a1225c9 ci: remove jjb Pods after job finished
Currently the number of Pods is growing in the OpenShift Console. The
pods are all Completed, so not running anymore, but there is also no
need to keep them around.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-10 13:09:14 +00:00
Niels de Vos
6a631e61cb ci: track jjb jobs by a jjb/session=<uuid> label
By using a template, it becomes possible to identify the Pod that has
been started by the Batch Job.

This prevents the script from getting the logs from an incorrect (old)
container.

Fixes: #1111
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-29 09:51:16 +00:00
Niels de Vos
9ce8d20da9 ci: cd into the WORKDIR before running jenkins-jobs
The directory that gets constructed can be like
`/opt/build/deploy/..//jobs` and this causes jenkins-jobs to fail.
Enternig the WORKDIR and passing the local ./jobs directory to
jenkins-jobs makes things clearer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 14:09:08 +02:00
Niels de Vos
650d749d28 ci: move the Jenkins Job Builder .yaml files under jobs/
There is a new .commitlintrc.yml file in the root of the repository.
This causes jenkins-job-builder to complains that the file is not
confirm the Jenkins Job Builder format/specification.

    jenkins_jobs.errors.JenkinsJobsException: The topmost collection in file '/opt/build/.commitlintrc.yml' must be a list, not a <class 'collections.OrderedDict'>

Move all Jenkins Job Builder .yaml files to jobs/ so that all the
Jenkins job configurations are kept together without other files.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 13:53:43 +02:00
Niels de Vos
0be9362216 ci: address yamllint issues in deploy/jjb-validate.yaml
16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
52c5c0a32d ci: address yamllint issues in jjb-deploy.yaml
./deploy/jjb-deploy.yaml
  16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  19:9      error    wrong indentation: expected 10 but found 8  (indentation)
  26:9      error    wrong indentation: expected 10 but found 8  (indentation)
  30:7      error    wrong indentation: expected 8 but found 6  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
311813448c ci: fix yamllint issues in jjb-buildconfig.yaml
./deploy/jjb-buildconfig.yaml
  10:3      error    wrong indentation: expected 4 but found 2  (indentation)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
1cb2eef010 ci: address ShellCheck issues in deploy/jjb.sh
In ./deploy/jjb.sh line 11:
	oc get pod/${1} --no-headers -o=jsonpath='{.status.phase}'
                   ^--^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 29:
cd $(dirname ${0})
^----------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^-------------^ SC2046: Quote this to prevent word splitting.
             ^--^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 31:
oc create -f jjb-${CMD}.yaml
                 ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 36:
	jjb_pod=$(oc get pods --no-headers -l job-name=jjb-${CMD} -o=jsonpath='{.items[0].metadata.name}')
                                                           ^----^ SC2086: Double quote to prevent globbing and word splitting.

In ./deploy/jjb.sh line 37:
	[ ${?} = 0 ] && [ -n "${jjb_pod}" ] && break
          ^--^ SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In ./deploy/jjb.sh line 46:
	[ ${?} = 0 ] && ( [ "${status}" = "Succeeded" ] || [ "${status}" = "Failed" ] ) && break
          ^--^ SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
                        ^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.

In ./deploy/jjb.sh line 54:
oc delete --wait -f jjb-${CMD}.yaml
                        ^----^ SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-27 12:52:21 +02:00
Niels de Vos
6d906b415f add scripts and configuration to deploy with Jenkins Job Builder
These files make it possible to
- build a container image that includes Jenkins Job Builder
- store the container image in the OpenShift ImageStream
- use the container to validate or deploy the Jenkins Jobs

More details about the files are included in the README.md.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-01 14:15:04 +02:00