Commit Graph

60 Commits

Author SHA1 Message Date
Madhu Rajanna
9466bc3542 ci: use quay.io for ceph image
ceph image is using quay.io not
the docker.io, This commit updates
the same in CI.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2024-07-30 07:42:46 +00:00
Niels de Vos
999a8f7f38 ci: use CentOS 9 Stream machines for all Jenkins jobs
CentOS 8 Stream will be EOL at the end of this month, use CentOS 9
Stream instead.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-05-29 16:53:57 +02:00
karthik-us
919db7082c ci: fix test_type arg assignment
Set the test_type argument to the default value only
if no type is specified explicitly.

Signed-off-by: karthik-us <ksubrahm@redhat.com>
2024-04-02 18:05:15 +02:00
karthik-us
ea0464958c ci: Fix helm test E2E_ARGS
Fix the helm tests E2E_ARGS interpolation.

Signed-off-by: karthik-us <ksubrahm@redhat.com>
2023-11-08 11:37:52 +00:00
karthik-us
1245b49cb9 ci: Fix E2E_ARGS for helm test
The test_type and deployment related args are being passed
as make option instead of the E2E_ARGS. Pass it as E2E_ARGS.

Signed-off-by: karthik-us <ksubrahm@redhat.com>
2023-11-08 06:09:40 +00:00
karthik-us
b8c16ccbac ci: Pass test_type as E2E_ARGS
The test_type is being passed as the make option instead
of the E2E_ARGS. This fix should pass it as the E2E_ARGS.

Signed-off-by: karthik-us <ksubrahm@redhat.com>
2023-11-06 11:13:28 +00:00
karthik-us
674b1e811a ci: add capability to run specific e2e tests
The current e2e test runs will trigger all 3 types of
tests which takes a lot of time to complete. If the
patch impacta only a single type, then having an option
to trigger only that particular e2e test can save
considerable amount of time based on the test type.

This can also help in reducing the time in testing when
a patch failed on a particular e2e test type and triggering
only that e2e with the fix for the failure or re-trigger
only that test again to check for false positives.

Signed-off-by: karthik-us <ksubrahm@redhat.com>
2023-11-06 06:57:20 +00:00
Niels de Vos
8135680065 ci: migrate to new OpenShift cluster running on AWS
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2023-03-20 09:21:41 +00:00
Niels de Vos
3880d642b9 ci: use new duffy command for CentOS CI
The `cico` command is getting deprecated, and the new `duffy` command
should be used instead. The new command requires a `~/.config/duffy`
file that contains an API endpoint and credentials.

See-also: https://sigs.centos.org/guide/ci/#installing-and-configuring-duffy-client
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-12-12 08:23:30 +00:00
Humble Chirammal
b553425d85 ci: replace docker.io ceph image with quay.io path
The ceph images are served from quay.io and this commit adjust
the docker pull to quay.io pull for centos jobs.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-03-30 09:11:37 +00:00
Niels de Vos
50ac270360 ci: request CentOS Stream 8 bare-metal machines
CentOS 8 is EOL since the beginning of the year. CentOS Stream 8
installations should be used instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-02-02 12:41:11 +00:00
Madhu Rajanna
a07fb3bcfc ci: update images for vault 1.8.5
updating ci scripts to pull the
vault images from ci registry.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-11-19 07:12:54 +00:00
Niels de Vos
e122cff477 e2e: do not delete the namespace and contents after a failure
When a failure occurs, by default the test namespace is removed. This
makes it impossible to fetch the logs of the containers where the
failure was discovered. Pass --delete-namespace-on-failure=false as an
additional argument to the `run-e2e` make target, so that the namespace
is kept.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-10-18 14:48:23 +00:00
Niels de Vos
ca4ec105c2 ci: use pre-pulled image for building e2e.test
The 'mini-e2e' job uses the pre-pulled container image for building the
e2e.test executable. The building of the executable takes less than 2
minutes, while building with preparing the container image takes close
to 5 minutes.

This reduces consumption of network bandwith, and might speed up other
parallel builds.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-08-19 09:33:35 +00:00
Rakshith R
935b261be1 ci: remove create & delete namespace commands from helm jobs
Since this will be done internally by intall_helm.sh script
with kubectl_retry.

Fixes: #2309
Depends-on: #2377

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-12 04:17:29 +00:00
Rakshith R
70c6414e1e ci: add --ignore-not-found=true to delete namespace in helm cleanup
This commit is added to facilitate merging of
https://github.com/ceph/ceph-csi/pull/2377 into devel and release
branches.

https://github.com/ceph/ceph-csi/pull/2378 will remove create &
delete namespace cmds from helm e2e jobs later on.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 06:10:46 +00:00
Niels de Vos
f129719f18 ci: quote username for podman_login()
Jenkins does not like the passing of the username as variable to the
podman_login() function. Calling the function results in an error like

    Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
        Affected argument(s) used the following variable(s): [CREDS_USER]
        See https://jenkins.io/redirect/groovy-string-interpolation for details.
    + ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@n7.pufty.ci.centos.org 'podman login --authfile=~/.podman-auth.json --username=$CREDS_USER --password=**** registry-****.apps.ocp.ci.centos.org'
    Username: Error: error getting username and password: error reading username: EOF

By single quoting the username, just like the password, it may work
better.

Fixes: aca3745e2 ("ci: do not use Groovy string interpolation for credentials")
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-15 10:44:27 +02:00
Niels de Vos
aca3745e24 ci: do not use Groovy string interpolation for credentials
Jenkins warns in the output of CI jobs about the following:

    Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
        Affected argument(s) used the following variable(s): [CREDS_PASSWD, CREDS_USER]
        See https://jenkins.io/redirect/groovy-string-interpolation for details.

Variable with 'single quotes' and without the {curly brackets} are
expecred to not be affected. There is some indirection in the strings
passed to the `sh` function, so this approach might not fix it?

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-15 07:48:41 +00:00
Yug
b4fcd2b585 ci: fix helm job failure
Due to the quotes, the job is accepting
both --deploy-sc and --deploy-secret as
as single parameter.
Removing quotes to help it consider them
as diferent.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 09:08:09 +00:00
Yug
b8c039528d ci: avoid deployment of sc and secret on v3.3
For release 3.3, we will not be deploying storageclass
and secret on helm installation on ci.
Moving forward, devel and all the future release will
have the deployment enabled by default in the ci.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 07:44:47 +00:00
Yug
b9a1698a89 e2e: deploy storageclass and secret
deploy secret and storageclass with
deployment via helm.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 07:44:47 +00:00
Yug
0786d15951 ci: pass --helm-test argument for helm tests
To identify if a test runs on ceph-csi deployed
via helm charts, pass --helm-test parameter with
the E2E args.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-06-24 08:13:15 +00:00
Yug
69091b9273 ci: pass namespace after --namespace flag
Since we are adding support for deployment of sc and
secret via flags, to help script recognize when an
unknown string is passed as a namespace, use
--namespace flag before entering the namespace.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-06-24 08:13:15 +00:00
Niels de Vos
e599e95f25 ci: in case of a failure, return error after logging system status
It seems that it is required to re-throw the error after a catch{..}
block. Without this, and a successful execution of system-status.sh, the
CI jobs get marked as SUCCESS, even when there was a failure.

Fixes: e36155283 "ci: run system-status.sh in case a job fails"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-11 13:13:39 +00:00
Niels de Vos
5b03721a58 ci: copy system-status.sh script to the bare metal node
Without the script on the node, it can not be executed...

Fixes: e36155283 "ci: run system-status.sh in case a job fails"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-11 12:33:03 +00:00
Niels de Vos
e36155283b ci: run system-status.sh in case a job fails
The new `system-status.sh` script logs the status of the host and the
minikube VM. This gets executed when a CI job fails, and should aid in
troubleshooting spurious failures.

Updates: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-11 11:27:01 +00:00
Niels de Vos
c04a319aa9 ci: only abort on doc-change when running for PRs
In case a job has been started without a PR (manual, or timed), the
current checked out branch matches the original as there are not
additional changes in the tree. There is no need to abort the jobs when
the skip-doc-change.sh script did not detect any non-doc changes, as
there are no changes at all.

Updates: #1963
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-09 09:53:22 +00:00
Niels de Vos
8f84e592d5 ci: do not re-checkout current branch
When tests are started manually (through the Jenkins webui), there is no
PR associated with the job. That means the `git_since` and `ref` are
equal. Trying to create a new branch named `ref` will not work, as the
branch was already created when cloning the repository with `git_since`.

With this change, Jenkins jobs can be started manually. This makes it
possible to run regular/nightly jobs as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-08 08:38:11 +00:00
Niels de Vos
69cb6aeead ci: pre-pull ROOK_CEPH_CLUSTER_IMAGE if set
After the introduction of ROOK_CEPH_CLUSTER_IMAGE in build.env, the
additional image needs to get pulled from the CI registry mirror and
pushed into the minikube VM.

Without this addition, the Docker Hub pull limits may prevent deploying
Rook.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-03 11:29:52 +00:00
Niels de Vos
1c2974d49e ci: the "master" branch got renamed to "devel"
Closes: #1193
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-01 05:22:06 +00:00
Niels de Vos
a26772188a ci: pull BASE_IMAGE from local registry
The CI scripts pull all container images from the local CI registry. If
the image name starts with "docker.io/", the images will be pushed into
the test environment as "docker.io/docker.io/ceph/ceph:v15". This image
will not be used by the tests, so things can still fail in case Docker
Hub has reached the pull rate-limit.

By dropping the additional "docker.io/" from the BASE_IMAGE name, the
image gets pushed as "docker.io/ceph/ceph:v15" so the tests will use it
automatically.

Groovy-syntax: https://www.baeldung.com/groovy-remove-string-prefix#using-regex
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-30 08:47:18 +00:00
Niels de Vos
468b6cd67d ci: pull images from local registry directly
The mirror option of the Docker Registry container is very limited and
prevents updating or manually pushing images to the registry. Instead,
it tries to push the images to the docker.io, which is not what we need.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-26 16:26:57 +00:00
Niels de Vos
005d201f2f ci: use docker.io/library/ as prefix for nginx and vault images
docker.io/nginx:latest and docker.io/vault:latest are being redirected
to docker.io/library/. The redirection is not cached, and Docker Hub
might return an error during redirection when the pull rate-limit is
hit.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-26 12:40:48 +00:00
Niels de Vos
a9557f36f3 ci: provide qualified image tags for docker.io images
Unqualified container images are currently used for CI jobs. In the
future this is expected to change. By preparing the cache/mirror and
images in minikube with the qualified tags, transition to qualified
image names should become easier.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 05:09:38 +00:00
Niels de Vos
5ae8fb7c9b ci: add configuration for the proxy/mirror registry
This makes it possible to pull images from Docker Hub through the local
container image registry in the CI OpenShift deployment. The registry in
the CI is configured with the 'cephcsibot' account so that pulling
images is accounted towards the account, and not anonymous consumers
within the whole CentOS CI.

There should be no need to manually sync the images between the local
registry and Docker Hub anymore.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 05:09:38 +00:00
Niels de Vos
6a7e6c841f ci: pre-pull rook/ceph image from local registry
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-19 12:48:32 +00:00
Niels de Vos
b9cffc1b42 ci: pass registry to podman helper functions
Functions with Groovy can not use `def ci_registry` as the variable is
not in the scope. Pass the registry to the podman_login() and
podman_pull() functions instead.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 12:31:22 +00:00
Niels de Vos
ea5985fa3a ci: fix calling podman_login()
A typo when calling podman_log() causes CI jobs to fail.

Fixes: 1eec379 "ci: pre-pull Ceph base-image and cephcsi:devel for mini-e2e-helm jobs"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 12:02:25 +00:00
Niels de Vos
f36ef72a19 ci: pre-pull nginx and vault images from local registry
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
dd10e66a98 ci: move podman2minikube() into its own script
This way, it can easier be re-used for other container images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
1eec3792ec ci: pre-pull Ceph base-image and cephcsi:devel for mini-e2e-helm jobs
The same changes have been made for the mini-e2e jobs yesterday, and
those seem to work well. Use the same pre-pull method for the Helm
deployment.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-18 11:49:10 +00:00
Niels de Vos
466ada3860 ci: use github/refs/pull/<pr-id>/merge instead of doing a rebase
When fetching refs/pull/<pr-id/merge from GitHub, there is no need to do
a manual rebase. This makes things easier, as a the scripted rebases
sometimes cause CI jobs to fail.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-28 04:12:48 +00:00
Niels de Vos
51fa5cca48 ci: use refs/pull/<id>/merge to prevent need for rebases
refs/pull/<id>/head might not contain the most current state of the
branch. In case other PRs got merged, the PR under test needs rebasing.
GitHub offers refs/pull/<id>/merge to checkout the rebased PR, use that
in the CI jobs.

In case refs/pull/<id>/merge is not available, it means the PR can not
be rebased on its target branch. This will cause the CI job to fail, but
GitHub also will have a message about rebase conflicts.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-21 13:29:06 +00:00
Niels de Vos
126fcd4cb5 ci: move skip-e2e-check after GitHub API usage
When the [ci/skip/e2e] label is set on PRs, the withCredentials()
statement is aborted, but the other stages still continue. This causes
the tests to run, which is not what we want when the label is added.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-20 06:37:25 +00:00
Niels de Vos
ed671ec207 ci: use the credentials-binding plugin for the GitHub API Token
The standard credentials() function does not seem to do what we need. So
use the credentials-binding Jenkins plugin instead.

See-also: https://www.jenkins.io/doc/pipeline/steps/credentials-binding/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-19 11:28:12 +00:00
Niels de Vos
a6ca8e31dc ci: get the GitHub API Token before starting a node
It still seems that the environment is not set when the GitHub API is
called. Maybe things work better when the environment is set before
starting the cico-workspace node.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-19 09:41:35 +00:00
Niels de Vos
89783afcf5 ci: set GITHUB_API_TOKEN in the environment for all stages
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-15 07:43:53 +00:00
Niels de Vos
2fb4e9fd22 ci: use environment{} to fetch the GitHub API token
The `credentials()` function might only work in the `environment` block
in the Pipelines. At the moment, running the 'skip ci/skip/e2e label'
stage always reports 'Error: 401 Client Error: Unauthorized'.

Fixes: e0d49908 (ci: fetch GITHUB_API_TOKEN from Jenkins credential store)
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 15:26:41 +00:00
Niels de Vos
e0d49908f8 ci: fetch GITHUB_API_TOKEN from Jenkins credential store
Fetch the named credential "github-api-token" from the Jenkins
configuration. This is a "personal access token" that has been created
with the ceph-csi-bot user account.

CC: @ceph-csi-bot
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-10-13 12:05:57 +00:00
Yug
1b661cb2c6 ci: correct indentation on mini-e2e-helm
The trailing '}' needs to be one the next line.

Signed-off-by: Yug <yuggupta27@gmail.com>
2020-09-24 14:50:49 +00:00