Commit Graph

129 Commits

Author SHA1 Message Date
Niels de Vos
535315cc29 ci: prevent detecting latest Kubernetes patch release
In case the version that is passed with --version=... contains the patch
release already, the latest patch release will not be detected, but the
passed version will be returned (enables forcing a particular version in
CI jobs).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-15 09:28:39 +00:00
Niels de Vos
89783afcf5 ci: set GITHUB_API_TOKEN in the environment for all stages
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-15 07:43:53 +00:00
Niels de Vos
dd7ac1f18e ci: log rate limit in case the GitHub API returns 403
Anonymous users have a low (currently 60) rate limit of API calls.
Authorized users have a much higher (currently 5000) limit. Print the
available, used and remainling rate limit when the GitHub API returns a
403 messages like:

    Error: 403 Client Error: rate limit exceeded

In case of anonymous access to the API, and hitting the limit, the
message in the logs will also include the following:

    Rate limit (limit/used/remaining): 60/60/0

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-14 15:30:32 +02:00
Niels de Vos
2fb4e9fd22 ci: use environment{} to fetch the GitHub API token
The `credentials()` function might only work in the `environment` block
in the Pipelines. At the moment, running the 'skip ci/skip/e2e label'
stage always reports 'Error: 401 Client Error: Unauthorized'.

Fixes: e0d49908 (ci: fetch GITHUB_API_TOKEN from Jenkins credential store)
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 15:26:41 +00:00
Niels de Vos
e0d49908f8 ci: fetch GITHUB_API_TOKEN from Jenkins credential store
Fetch the named credential "github-api-token" from the Jenkins
configuration. This is a "personal access token" that has been created
with the ceph-csi-bot user account.

CC: @ceph-csi-bot
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00
Niels de Vos
b27b2ee074 ci: authenticate with GitHub API token from the environment
These script now check if GITHUB_API_TOKEN is set in the environment,
and use that for authenticating. There is no username needed when an API
token is used, but it may not be empty, so it is set to "unused".

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00
Niels de Vos
aa062513c0 ci: report connections failures with GitHub API
On occasion the scripts that use the GitHub API fail without logging any
useful error:

    ./scripts/get_github_labels.py --id=1568 --has-label=ci/skip/e2e
    Traceback (most recent call last):
      File "./scripts/get_github_labels.py", line 71, in <module>
        main()
      File "./scripts/get_github_labels.py", line 55, in main
        names = get_names(json)
      File "./scripts/get_github_labels.py", line 40, in get_names
        names.append(label['name'])
    TypeError: string indices must be integers

While debugging, it seems that the limit of API calls is reached:

    403 Client Error: rate limit exceeded

It is useful to have failure messages reported in the output of the CI
jobs for future potential troubleshooting.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00
Yug
bd247db377 ci: Trigger upgrade test jobs automatically
Since upgrade tests will become a necessary to
merge PRs, it is better to trigger them
automaticaly now.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-10-01 07:19:17 +00:00
Yug
0895d864e7 ci: update k8s version for upgrade tests
As the job seems to be unstable for k8s v1.19,
reducing the version to achieve stability for
the same.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-25 12:28:34 +00:00
Niels de Vos
9a01ee2019 ci: remove old jobs when deploying
Old jobs stay behind (and active) when jobs get updated with new
versions. This mostly affects the mini-e2e jobs at the moment.

There is no need to keep old job around, so delete them while deploying
updates to jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-25 08:03:00 +00:00
Yug
31769aa7dc ci: trigger upgrade tests job only on phrase
To avoid runs of upgrade tests on all PRs till
the job is stable, trigger it only through
phrase.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-24 15:02:12 +00:00
Yug
1b661cb2c6 ci: correct indentation on mini-e2e-helm
The trailing '}' needs to be one the next line.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-24 14:50:49 +00:00
Yug
64c3af777b ci: pass REBASE=1 to have commitlint rebase the branch
Rebase current checked out PR on top of parent branch.
This will then prevent commitlint to check complete
history of 'ci/centos' branch.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-24 13:32:28 +00:00
Niels de Vos
2f2123108b ci: run minikube.sh with podman
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-24 12:52:17 +00:00
Niels de Vos
b5223f7620 ci: do not install Docker on the minikube host
Installing Docker CE on CentOS-8 currently conflicts with Podman. Both
tools can not be installed at the same time.

Instead of using the Docker command to push the image to to minikube VM,
read the image from stdin over ssh and load it with the Docker command
that is available inside the VM.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-24 12:12:17 +00:00
Yug
118015d762 ci: Fetch csi upgrade version from configuration file
Instead of accessing the upgrade version from environment
variable, set the variable directly in configuration file.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-23 08:53:15 +00:00
Yug
69b2d1b478 ci: use 'make run-e2e'
Instead of calling the functional tests script,
use 'make run-e2e' to run the tests.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-23 08:53:15 +00:00
Yug
e2e7ffa2d4 ci: Add an escape character before UPGRADE_VERSION
Since UPGRADE_VERSION is not a groovy variable, adding an
escape character is required for proper execution of
upgrade-testing job.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-22 11:47:20 +00:00
Niels de Vos
01de4eacfc ci: restructure 'parameters' section for upgrade-tests
The section containing the paramaters does not seem to be structured
correctly, correct it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 10:30:38 +00:00
Niels de Vos
926817fb54 ci: remove unused parameters section
The incomplete 'parameters:' section causes validation and deploy
issues for jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 10:30:38 +00:00
Niels de Vos
b1d7c83c16 ci: checkout FETCH_HEAD instead of GIT_REF
GIT_REF can point to a ref that it not part of a branch and can not
easily be checked out (like a GitHub PR). Checking out FETCH_HEAD after
fetching the GIT_REF works.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Niels de Vos
5f9222d7ee ci: run jjb-validate batch job against a PR
The batch job templates for OpenShift use a GIT_REF variable that is
used by the jobs to checkout the repository. This defaults to
"ci/centos", but it never is adjusted when jjb-validate runs against a
PR.

With the new GIT_REF environment variable in the jjb.sh script, the
variable can now be used to checkout a PR from gitHub and run the
validation against that.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Niels de Vos
fef7bfdf23 ci: do not set a ghprbPullId parameter
The GitHub Pull Requests Builder plugin sets the ghprbPullId parameter.
There is no need to make it configurable, and it even overloads the
value from the plugin by the default value.

This cauesed the jjb-validate job to run against the ci/centos branch
for each PR, not against the contents of the PR itself.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Niels de Vos
79225080f9 ci: sandbox is not a supported option
The Jenkins Jobs Builder files reference 'sandbox: true' as an option,
but this does not exist (anymore). Having the option causes warnings in
the Jenkins WebUI for admin users.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Niels de Vos
cb4fd5338b ci: correct referencing template for upgrade-tests jobs
Fixes: #1489
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-22 11:59:31 +02:00
Yug
2e2df0a877 ci: Add template file for upgrade-tests job
This template file contains the configuration
parameters for the upgrade-tests job.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-22 07:25:00 +00:00
Yug
447e2b1ae4 ci: Add 'upgrade-tests' job to centos ci
Add Snapshot and PVC to PVC cloning test cases
to rbd and cephfs upgrade testing.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-22 07:25:00 +00:00
Niels de Vos
e353d406eb ci: pass major k8s version to jobs, detect patch release
Jobs can now pass the wanted Kubernetes major version (like '1.19') to
the Jenkins Pipeline scripts. The Pipelines detect the most recent patch
release for the major version with the new get_patch_release.py script.

This causes the CI Job status context to not have the patch number (last
digit of the release) included anymore. Restarting a test will only need
the major version number, as does updating the Mergify configuration.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-15 17:02:04 +05:30
Niels de Vos
73244ab07b ci: add script to detect latest Kubernetes patch release
By using this script, we can pass Kubernetes versions like v1.19 to the
CI jobs. The jobs are then responsible for detecting the a full release
version, including the '.patch' suffix.

Versions are passed as 1.19, releases have the 'v' prefix as in v1.19.1.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-09-15 17:02:04 +05:30
Madhu Rajanna
8fd212a797 ci: update kubernetes version for e2e job
removed 1.17 as we are testing only two versions
in centos CI. as v1.19 is released, we will be
testing v1.18 and v1.19 in CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-09-09 14:02:45 +00:00
Niels de Vos
199f5e7551 ci: skip mini-e2e-helm jobs if PR has the 'ci/skip/e2e' label
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-24 08:51:46 +00:00
Niels de Vos
8806d8e69b ci: skip mini-e2e jobs if PR has the 'ci/skip/e2e' label
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-24 08:51:46 +00:00
Niels de Vos
d172010e52 ci: add script to fetch labels for GitHub PR
By passing `--id=<id>` the labels of the given Pull-Request or Issue get
printed. When adding `--has-label=<label>`, the script verifies if the
label is set and returns 0 on succes, or 1 on failure.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-24 08:51:46 +00:00
Madhu Rajanna
5db63ceacd ci: error out for invalid component in commit header
if the commit header is not present in the list of
the component we maintain. the commitlint check should
show error instead of showing the warning.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 52243d0140)
2020-08-19 11:15:09 +00:00
Niels de Vos
23c317b05a ci: mark .commitlintrc.yml as doc-only
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit fb9ae06b1f)
2020-08-19 10:04:19 +00:00
Niels de Vos
46d0ae5495 ci: prepare.sh should accept --history without argument
The --history does not take an argument, similar to --help. Move it out
of ARGUMENT_LIST so 'getopt' does not complain about it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 15:00:17 +02:00
Niels de Vos
64588e2b1c ci: use FETCH_HEAD like "commitlint --from" does
After fetching the target branch for the PR, the GIT_SINCE ref in the
git repository may not be set (in CI environments). This causes 'git
rebase' to fail. Use FETCH_HEAD instead, so that the checked out PR gets
rebased correctly.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 11:54:23 +00:00
Niels de Vos
6fc3706fab ci: fetch full history when cloning for commitlint job
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 11:54:23 +00:00
Niels de Vos
e9aec13761 ci: add --history option to prepare.sh
In case the history of the branch is needed (commitlint job), add an
option --history to skip cloning the repository with --depth=1.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 11:54:23 +00:00
Niels de Vos
56d545c2e2 ci: pass REBASE=1 while running commitlint job
By passing REBASE=1 to the commitlint make target, the current checked
out PR will be rebased on top of the parent branch. This should result
in a linear history that commitlint can parse correctly.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 09:05:41 +00:00
Niels de Vos
ffb4634ba5 ci: pass REBASE=1 to have commitlint rebase the branch
When Mergify adds a merge commit to the branch that is being tested with
commitlint, the tool tries to detect the most recent changes based on
the newly merged commit. This is for most PRs the master branch, and
that contains incorrect commit messages in the history. Because of this,
commitlint will fail.

By adding an option (REBASE=1) to the commitlint make target, CI jobs
can request a rebase so that the history of the PR becomes linear again
and commitlint should be able to detect only the new commits.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 09:05:41 +00:00
Yug
e106eed235 ci: Increase timout for mini-e2e-helm job
Increase timeout to provide sufficient time
for tests to execute.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-18 06:32:52 +00:00
Yug
79adaef3d1 ci: Increase timeout for mini-e2e job
Due to a strict timeout, the job
tends to abort sometimes. Increasing the
timeout to allow sufficient time for
tests to execute.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-08-18 06:32:52 +00:00
Niels de Vos
5830a2f1bf ci: use GitHub Hooks to trigger jobs
By using GitHub Hooks, jobs get triggers quickers. It also may prevent
the occasional hickup in the Jenkins GitHub-Pull-Request-Builder while
polling events for the cron-jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-17 09:45:07 +00:00
Niels de Vos
108628039e rebase: use Rook v1.3.9 for mini-e2e jobs
Rook v1.3.9 was released yesterday, so we can consume it for our CI jobs
now.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-17 03:44:37 +00:00
Niels de Vos
ca6d88b3b7 ci: use build.env:CSI_IMAGE_VERSION for mini-e2e jobs
By placing CSI_IMAGE_VERSION=v3.1.0 in build.env in the release-v3.1
branch, the CI jobs using single-node-k8s.sh can pickup the version
dynamically.

Other release branches can do the same, and the CI jobs will not need
modifications for each release.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-13 13:40:41 +00:00
Niels de Vos
12c0d07184 ci: the ci-job-validation script uses an undefined variable
It seems ci-job-validation.groovy got merged with a mistake. The
git_repo variable is used, but was not defined.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-13 18:56:41 +05:30
Niels de Vos
209b31c83a ci: use last "git fetch" output to test commitlint
When running in the CI the git repository is not completely cloned. This
causes the 'commitlint' job to be unable to resolve the history of the
commits.

By using FETCH_HEAD, the last 'git fetch' output will be used.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-12 16:04:26 +00:00
Niels de Vos
0e19f37f61 ci: skip mini-e2e for doc-only PRs
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-12 16:04:26 +00:00
Niels de Vos
44d6b17aa7 ci: skip mini-e2e-helm for doc-only PRs
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-12 16:04:26 +00:00