Commit Graph

172 Commits

Author SHA1 Message Date
Niels de Vos
e4339fea72 ci: pre-pull Ceph base-image and cephcsi:devel for upgrade-tests
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
1eec3792ec ci: pre-pull Ceph base-image and cephcsi:devel for mini-e2e-helm jobs
The same changes have been made for the mini-e2e jobs yesterday, and
those seem to work well. Use the same pre-pull method for the Helm
deployment.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
7796f52043 ci: pre-pull ceph base image for containerized-tests job
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
42bfd19da6 ci: introduce and use podman helper functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
9887a08377 ci: the image in the CI registry is called "ceph-csi"
The image is named after the project, so "ceph-csi".

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 13:21:50 +00:00
Niels de Vos
e9523b3ace ci: store podman-login results in ~/.podman-auth.json
It seems that "podman pull" does not consume the authentication details
from ~/.docker/config.json, so store the results of "podman login" in
~/.podman-auth.json and use the file for all "podman pull" commands.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 13:00:36 +00:00
Niels de Vos
6b4dffbd79 ci: store results of podman-login in ~/.docker/config.json
Once the ssh command finishes, the runtime directory is removed and the
results of "podman login" are lost. By storing the results in the
standard Docker configuration file, subsequent "podman pull" commands
will be able to re-use the authentication details.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 12:37:27 +00:00
Niels de Vos
baac40c136 ci: escape command to be executed over ssh to fetch BASE_IMAGE
The ${BASE_IMAGE} variable gets expanded by running the ssh command.
This becomes an empty variable, so the "echo" part of the command does
not output anything.

By escaping the command, there is no variable substitution, and the
BASE_IMAGE variable should get stored in the variable.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 11:46:59 +00:00
Niels de Vos
ea281737fe ci: ssh into the bare-metal machine
Missed the CICO_NODE to ssh into, that has now been added too.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 11:20:32 +01:00
Niels de Vos
629a8e0166 ci: do not run commitlint on "/retest all"
The commitlint job is not used normally, so there is no need to run it
when all CI jobs need retesting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 10:16:46 +00:00
Niels de Vos
3c82db25e3 ci: correctly fetch BASE_IMAGE from build.env
The BASE_IMAGE variable was not stored in the variable so that the CI
job can consume it. By using sh(), this should be the case now.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 10:12:32 +00:00
Niels de Vos
b5123e6134 ci: use podman to login on the CI registry
The CI registry contains several container images that can be pulled
instead of using Docker Hub.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 09:51:16 +00:00
Niels de Vos
a8a6ef27a6 ci: use "podman login --username" instead of "--user"
It seems the --user option does not exist, but is called --username.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 09:19:37 +00:00
Niels de Vos
d7172453ca ci: pull cached images from local registry
This reduces the number of images that get pulled from Docker Hub where
pull limits affect the jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 08:50:03 +00:00
Niels de Vos
49c6784c5c ci: deploy/jjb.sh should fail on wrong arguments
In case the '--cmd' option is missing, usage() will be called and the
script exits with 0. A missing option is a failure, so make usage()
return and handle the exit status when usage() got called instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 10:09:57 +00:00
Niels de Vos
2ad602e9f2 ci: fix Blue Ocean status-url for JJB templates
As ${RUN_DISPLAY_URL} also matches the format for template
substitutions, it can not be used like this for all Jenkins Jobs Builder
files. Instead, $RUN_DISPLAY_URL can be used, as it does not have the
{curly_braces} format and the template engine skips it.

Fixes: 87870369 (ci: link to the job in the Blue Ocean webui)
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 09:58:46 +00:00
Niels de Vos
f67ece89a9 ci: split "go mod verify" into its own stage
In order to speed up testing, "go mod verify" can be run parallel from
other tests. Downloading and verifying all dependencies is the most time
consuming step in the containerized-test CI job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:52:01 +00:00
Niels de Vos
09c4191a6f ci: enable USE_PULLED_IMAGE to prevent image builds
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:52:01 +00:00
Niels de Vos
4ba6e51f85 ci: use ssh() helper function for containerized-tests
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:52:01 +00:00
Niels de Vos
f14bf59856 ci: use cached image for containerized-tests
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:52:01 +00:00
Niels de Vos
d32319ae4e ci: use cached image for ci-job-validation
Pull the ceph-csi:test image from the local (internal to CI) registry.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:52:01 +00:00
Niels de Vos
a9014ce2ce ci: add script to detect need of container image rebuild
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:52:01 +00:00
Niels de Vos
8787036903 ci: link to the job in the Blue Ocean webui
The new Blue Ocean webui is much easier to navigate than the traditional
Jenkins webui. Developers that want to check failures of CI jobs will be
able to find the logs much quicker as each stage/step has a clear
success or failed status.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 04:51:49 +00:00
Niels de Vos
45130996a0 ci: flatten images for internal consumption
There is no need to re-use different layers with the images that are
built. The separation of the layers causes more time to be used while
pulling the images, so flattening makes speeds up the CI jobs a little
(about 30 seconds from 2+ minutes).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 04:43:07 +00:00
Niels de Vos
9c4009bd14 ci: pass BASE_IMAGE to "oc start-build"
Because the parallel steps, the build.env is not in the local directory
where the command is executed. So fetch the BASE_IMAGE before continuing
with 'oc start-build', and pass the name of the image on the
commandline.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 04:28:28 +00:00
liranmauda
c281511a11 ci: Add the ability to run JJB from private branch/fork
Add the ability to run JJB from private branch/fork

jjb-deploy.yaml:
- Add GIT_REPO parameter (not required) with default value

jjb.sh:
- Changing PS4 for more indicative debug prompt
- Adding flags to get the repo and branch

jjb-deploy.yaml:
- Calling the jjb.sh with the correct flags

Signed-off-by: liranmauda <liran.mauda@gmail.com>
2020-11-03 07:26:43 +00:00
Niels de Vos
1700f8585c ci: add a build-images job
The new 'build-images' job rebuilds ceph-csi:canary, ceph-csi:devel and
ceph-csi:test images after a PR has been merged in the master branch.
These images can then be used by other CI jobs, to improve the speed by
reducing unneeded rebuilds of the images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-30 13:07:58 +00:00
Niels de Vos
5dc6045034 ci: add BuildConfig for ceph-csi container images
The BuildConfig can be imported in the OpenShift environment so that
Ceph-CSI images can be built automatically. These images will be cached
in a private container image registry, which can speed up CI jobs that
currently rebuild the images every time.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-30 13:07:58 +00:00
Niels de Vos
b5216592e2 ci: deploy a private container image registry
With a private container image registry, the images that CI jobs use can
be cached and re-used. This speeds up the CI jobs, as building the
container images takes up the majority of the runtime for some jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-30 13:07:58 +00:00
Niels de Vos
31c9d7d2e3 ci: use self-contained kubeconfig for running k8s e2e job
When running on Minikube, the k8s-e2e-external-storage fails with an
error related to the kubeconfig file:

    error creating client: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

It seems that this is a problem when the kubeconfig is not
self-container, but refers to additional files. This is the case when
Minikube deployed the Kubernetes cluster.

By exporting the kubeconfig into its own self-contained file, the CI job
starts running.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-28 14:03:44 +00:00
Niels de Vos
466ada3860 ci: use github/refs/pull/<pr-id>/merge instead of doing a rebase
When fetching refs/pull/<pr-id/merge from GitHub, there is no need to do
a manual rebase. This makes things easier, as a the scripted rebases
sometimes cause CI jobs to fail.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-28 04:12:48 +00:00
Niels de Vos
cd893ba376 ci: use major Kubernetes versions again
With the update to minikube v1.14.1 downloading binaries for the recent
Kubernetes patch releases works again. Enable detecttion of the latest
patch releases for use in the CI jobs.

Fixes: #1588
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-27 13:40:25 +00:00
Yug
86b80a23a5 ci: Fix shellcheck warnings for prepare.sh
Shellcheck suggests use of doube quotes to prevent
word splitting, hence adding the same.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-10-27 13:30:42 +00:00
Niels de Vos
aa7d9de45a ci: copy run-k8s-external-storage-e2e.sh to bare-metal machine
The ./run-k8s-external-storage-e2e.sh script that executes the
Kubernetes e2e external-storage tests needs to be placed on the
bare-metal machine. Currently, the k8s-e2e-external-storage job fails
with:

    bash: ./run-k8s-external-storage-e2e.sh: No such file or directory

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-23 12:55:06 +00:00
Niels de Vos
e103d4a0cc ci: add k8s-e2e-external-storage job
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-23 09:30:14 +00:00
Niels de Vos
4b04e2b426 ci: add script to run external-storage e2e tests
The run-k8s-external-storage-e2e.sh script downloads the kubernetes-test
components, creates a configuration for RBD and CephFS drivers and runs
the external-storage tests on the current cluster.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-23 09:30:14 +00:00
Niels de Vos
86375adf9e ci: run only one jjb-validate at the time
The jjb-validate job creates a Batch Job in OCP. This job has a fixed
name, and fails to get created when one with the same name already
exists. There is no need to run jjb-validate concurrently, the number of
changes for CI jobs are not blocked by this serialization.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-23 07:16:24 +00:00
Niels de Vos
51fa5cca48 ci: use refs/pull/<id>/merge to prevent need for rebases
refs/pull/<id>/head might not contain the most current state of the
branch. In case other PRs got merged, the PR under test needs rebasing.
GitHub offers refs/pull/<id>/merge to checkout the rebased PR, use that
in the CI jobs.

In case refs/pull/<id>/merge is not available, it means the PR can not
be rebased on its target branch. This will cause the CI job to fail, but
GitHub also will have a message about rebase conflicts.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-21 13:29:06 +00:00
Niels de Vos
126fcd4cb5 ci: move skip-e2e-check after GitHub API usage
When the [ci/skip/e2e] label is set on PRs, the withCredentials()
statement is aborted, but the other stages still continue. This causes
the tests to run, which is not what we want when the label is added.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-20 06:37:25 +00:00
Niels de Vos
ed671ec207 ci: use the credentials-binding plugin for the GitHub API Token
The standard credentials() function does not seem to do what we need. So
use the credentials-binding Jenkins plugin instead.

See-also: https://www.jenkins.io/doc/pipeline/steps/credentials-binding/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-19 11:28:12 +00:00
Niels de Vos
a6ca8e31dc ci: get the GitHub API Token before starting a node
It still seems that the environment is not set when the GitHub API is
called. Maybe things work better when the environment is set before
starting the cico-workspace node.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-19 09:41:35 +00:00
Niels de Vos
debb54a1dd ci: use Kubernetes 1.18.9 for CI jobs
The latest Kubernetes patch release (v1.18.10) can not be deployed with
minikube. Selecting version 1.18.9 until the problems with minikube are
addressed.

Updates: #1588
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-19 06:42:35 +00:00
Niels de Vos
ea770fb4c8 ci: use Kubernetes 1.19.2 for CI jobs
The latest Kubernetes patch release (v1.19.3) can not be deployed with
minikube. Selecting version 1.19.2 until the problems with minikube are
addressed.

Updates: #1588
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-15 09:28:39 +00:00
Niels de Vos
535315cc29 ci: prevent detecting latest Kubernetes patch release
In case the version that is passed with --version=... contains the patch
release already, the latest patch release will not be detected, but the
passed version will be returned (enables forcing a particular version in
CI jobs).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-15 09:28:39 +00:00
Niels de Vos
89783afcf5 ci: set GITHUB_API_TOKEN in the environment for all stages
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-15 07:43:53 +00:00
Niels de Vos
dd7ac1f18e ci: log rate limit in case the GitHub API returns 403
Anonymous users have a low (currently 60) rate limit of API calls.
Authorized users have a much higher (currently 5000) limit. Print the
available, used and remainling rate limit when the GitHub API returns a
403 messages like:

    Error: 403 Client Error: rate limit exceeded

In case of anonymous access to the API, and hitting the limit, the
message in the logs will also include the following:

    Rate limit (limit/used/remaining): 60/60/0

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-14 15:30:32 +02:00
Niels de Vos
2fb4e9fd22 ci: use environment{} to fetch the GitHub API token
The `credentials()` function might only work in the `environment` block
in the Pipelines. At the moment, running the 'skip ci/skip/e2e label'
stage always reports 'Error: 401 Client Error: Unauthorized'.

Fixes: e0d49908 (ci: fetch GITHUB_API_TOKEN from Jenkins credential store)
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 15:26:41 +00:00
Niels de Vos
e0d49908f8 ci: fetch GITHUB_API_TOKEN from Jenkins credential store
Fetch the named credential "github-api-token" from the Jenkins
configuration. This is a "personal access token" that has been created
with the ceph-csi-bot user account.

CC: @ceph-csi-bot
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00
Niels de Vos
b27b2ee074 ci: authenticate with GitHub API token from the environment
These script now check if GITHUB_API_TOKEN is set in the environment,
and use that for authenticating. There is no username needed when an API
token is used, but it may not be empty, so it is set to "unused".

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00
Niels de Vos
aa062513c0 ci: report connections failures with GitHub API
On occasion the scripts that use the GitHub API fail without logging any
useful error:

    ./scripts/get_github_labels.py --id=1568 --has-label=ci/skip/e2e
    Traceback (most recent call last):
      File "./scripts/get_github_labels.py", line 71, in <module>
        main()
      File "./scripts/get_github_labels.py", line 55, in main
        names = get_names(json)
      File "./scripts/get_github_labels.py", line 40, in get_names
        names.append(label['name'])
    TypeError: string indices must be integers

While debugging, it seems that the limit of API calls is reached:

    403 Client Error: rate limit exceeded

It is useful to have failure messages reported in the output of the CI
jobs for future potential troubleshooting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00