add scripts and configuration to deploy with Jenkins Job Builder

These files make it possible to
- build a container image that includes Jenkins Job Builder
- store the container image in the OpenShift ImageStream
- use the container to validate or deploy the Jenkins Jobs

More details about the files are included in the README.md.

Signed-off-by: Niels de Vos <ndevos@redhat.com>

deploy/Dockerfile

@@ -0,0 +1,22 @@
+FROM centos:latest
+
+RUN true \
+ && yum -y install git make python3-pip \
+ && pip3 install jenkins-job-builder \
+ && yum -y clean all \
+ && true
+
+ENV MAKE_TARGET=${MAKE_TARGET:-test}
+
+# Environment that needs to be set before executing checkout-repo.sh
+# ENV GIT_REPO=https://github.com/ceph/ceph-csi
+# ENV GIT_REF=ci/centos
+ADD checkout-repo.sh /opt/build/
+
+# make WORKDIR writable, otherwise git checkout fails
+RUN chmod ugo=rwx /opt/build
+
+ENV HOME=/opt/build
+WORKDIR /opt/build
+
+CMD ["sh", "-c", "./checkout-repo.sh && make -C deploy ${MAKE_TARGET}"]

deploy/Makefile

@@ -0,0 +1,8 @@
+WORKDIR ?= $(CURDIR)/../
+OUTPUT ?= $(WORKDIR)/_output/
+
+test:
+	jenkins-jobs test -o $(OUTPUT) $(WORKDIR)
+
+deploy:
+	jenkins-jobs update $(WORKDIR)

deploy/README.md

@@ -0,0 +1,46 @@
+# Deploying Jenkins Jobs through OpenShift
+
+This `deploy/` directory contains the configuration to prepare running Jenkins
+Job Builder on OpenShift and update/add Jenkins Jobs in an environment hosted
+in the same OpenShift project.
+
+The used Jenkins environment is expected to be deployed already. This is done
+by the CentOS CI team when a [request for CI resources](ci_request) is handled.
+The deploying and configuration of Jenkins is therefor not part of this
+document.
+
+## Building the Jenkins Job Builder container image
+
+OpenShift has a feature called ImageStreams. This can be used to build the
+container image that contains the `jenkins-jobs` executable to test and
+update/add jobs in a Jenkins environment.
+
+All `.yaml` files in this directory need to be pushed into OpenShift, use `oc
+create -f <file>` for that.
+
+- the `Dockerfile` uses `pip` to install `jenkins-jobs`, the BuildConfig object
+  in OpenShift can then be used to build the image
+- `checkout-repo.sh` will be included in the container image, and checks out
+  the `ci/centos` branch of the repository
+- together with the `Makefile` (checked out with `checkout-repo.sh`), the
+  Jenkins Jobs can be validated or deployed
+- `jjb-buildconfig.yaml` creates the ImageStream and the BuildConfig objects.
+  Once created with `oc create`, the OpenShift Console shows a `Build` button
+  for the `jjb` image under the Builds/Builds menu
+- `jjb-config.yaml` is the `/etc/jenkins_jobs/jenkins_jobs.ini` configuration
+  files that contains username, password/token and URL to the Jenkins instance
+  (**edit this file before pushing to OpenShift**)
+- `jjb-validate.yaml` is the OpenShift Job that creates a Pod, runs the
+  validation test and exits. The job needs to be deleted from OpenShift before
+  it can be run again.
+- `jjb-deploy.yaml` is the OpenShift Job that creates a Pod, runs
+  `jenkins-jobs` to push the new jobs to the Jenkins environment. This pod uses
+   the jjb-config ConfigMap to connect and login to the Jenkins instance. The
+   job needs to be deleted from OpenShift before it can be run again.
+- `jjb.sh` is a helper script that can be used to validate/deploy the Jenkins
+  Jobs in the parent directory. It creates the validate or deploy job, waits
+  until the job finishes, shows the log and exits with 0 on success. This
+  script can be used in Jenkins Jobs to automate the validation and deployment of
+  jobs.
+
+[ci_request]: https://wiki.centos.org/QaWiki/CI/GettingStarted

deploy/checkout-repo.sh

@@ -0,0 +1,17 @@
+#!/bin/sh
+
+fail() {
+	echo "${@}" > /dev/stderr
+	exit 1
+}
+
+[ -n "${GIT_REPO}" ] || fail 'GIT_REPO environment variable not set'
+[ -n "${GIT_REF}" ] || fail 'GIT_REF environment variable not set'
+
+# exit in case a command fails
+set -e
+
+git init .
+git remote add origin "${GIT_REPO}"
+git fetch origin "${GIT_REF}"
+git checkout "${GIT_REF}"

deploy/jjb-buildconfig.yaml

@@ -0,0 +1,30 @@
+---
+apiVersion: v1
+kind: ImageStream
+metadata:
+  name: jjb
+  labels:
+    app: jjb
+spec:
+  tags:
+  - name: latest
+---
+apiVersion: v1
+kind: BuildConfig
+metadata:
+  name: jjb
+  labels:
+    app: jjb
+spec:
+  runPolicy: Serial
+  source:
+    git:
+      uri: https://github.com/ceph/ceph-csi
+      ref: ci/centos
+    contextDir: deploy
+  strategy:
+    dockerStrategy: {}
+  output:
+    to:
+      kind: ImageStreamTag
+      name: jjb:latest

deploy/jjb-config.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: jenkins-jobs
+  labels:
+    app: jjb
+data:
+  jenkins_jobs.ini: |-
+    [jenkins]
+    user=ndevos-admin-edit-view
+    password=<token-from-user-config-in-webui>
+    url=https://jenkins-ceph-csi.apps.ci.centos.org/

deploy/jjb-deploy.yaml

@@ -0,0 +1,33 @@
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  labels:
+    app: jjb
+  name: jjb-deploy
+spec:
+  ttlSecondsAfterFinished: 0
+  backoffLimit: 1
+  template:
+    labels:
+      app: jjb
+    spec:
+      containers:
+      - name: jjb
+        image: 172.30.254.79:5000/ceph-csi/jjb:latest
+        env:
+        - name: GIT_REPO
+          value: https://github.com/ceph/ceph-csi
+        - name: GIT_REF
+          value: ci/centos
+        - name: MAKE_TARGET
+          value: deploy
+        volumeMounts:
+        - name: etc-jj
+          mountPath: /etc/jenkins_jobs
+          readOnly: true
+      volumes:
+      - name: etc-jj
+        configMap:
+          name: jenkins-jobs
+      restartPolicy: Never

deploy/jjb-validate.yaml

@@ -0,0 +1,23 @@
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  labels:
+    app: jjb-validate
+  name: jjb-validate
+spec:
+  ttlSecondsAfterFinished: 0
+  backoffLimit: 1
+  template:
+    labels:
+      app: jjb-validate
+    spec:
+      containers:
+      - name: jjb-validate
+        image: 172.30.254.79:5000/ceph-csi/jjb:latest
+        env:
+        - name: GIT_REPO
+          value: https://github.com/ceph/ceph-csi
+        - name: GIT_REF
+          value: ci/centos
+      restartPolicy: Never

deploy/jjb.sh

@@ -0,0 +1,57 @@
+#!/bin/sh
+#
+# Create a new Job in OCP that runs the jbb-validate container once. This
+# script will wait for completion of the validation, and uses the result of the
+# container to report the status.
+#
+
+CMD="${1}"
+
+get_pod_status() {
+	oc get pod/${1} --no-headers -o=jsonpath='{.status.phase}'
+}
+
+case "${CMD}" in
+	"validate")
+		;;
+	"deploy")
+		;;
+	*)
+		echo "no such command: ${CMD}"
+		exit 1
+		;;
+esac
+
+# make sure there is a valid OCP session
+oc version || exit 1
+
+# the deploy directory where this script is located, contains files we need
+cd $(dirname ${0})
+
+oc create -f jjb-${CMD}.yaml
+
+# loop until pod is available
+while true
+do
+	jjb_pod=$(oc get pods --no-headers -l job-name=jjb-${CMD} -o=jsonpath='{.items[0].metadata.name}')
+	[ ${?} = 0 ] && [ -n "${jjb_pod}" ] && break
+	sleep 1
+done
+
+# loop until the pod has finished
+while true
+do
+	status=$(get_pod_status "${jjb_pod}")
+	# TODO: is Running as a status sufficient, did it terminate yet?
+	[ ${?} = 0 ] && ( [ "${status}" = "Succeeded" ] || [ "${status}" = "Failed" ] ) && break
+	sleep 0.5
+done
+
+# show the log of the finished pod
+oc logs "${jjb_pod}"
+
+# delete the job, so a next run can create it again
+oc delete --wait -f jjb-${CMD}.yaml
+
+# return the exit status of the pod
+[ "${status}" = 'Succeeded' ]