Commit Graph

300 Commits

Author SHA1 Message Date
Prasanna Kumar Kalever
a373882099 ci: dump the rbd-nbd logs
print the rbd-nbd logs from the VM location /var/log/ceph/

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-08-18 15:20:32 +00:00
Rakshith R
2f24c7f47d ci: use csi sidecar image versions from build.env if available
This commit makes changes necessary to use csi sidecar image versions
from build.env if available.

PR https://github.com/ceph/ceph-csi/pull/2217 adds these
variables to `build.env`.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-16 11:17:26 +00:00
Yug Gupta
2be0306c82 ci: run tests on k8s v1.22 in place of v1.19
As kubernetes v1.19 is heading towards its EOL
on 2021-09-30, run tests on kubernetes v1.22
and require it to pass for merging.

Signed-off-by: Yug Gupta <yuggupta27@gmail.com>
2021-08-12 11:32:43 +00:00
Rakshith R
935b261be1 ci: remove create & delete namespace commands from helm jobs
Since this will be done internally by intall_helm.sh script
with kubectl_retry.

Fixes: #2309
Depends-on: #2377

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-12 04:17:29 +00:00
Rakshith R
70c6414e1e ci: add --ignore-not-found=true to delete namespace in helm cleanup
This commit is added to facilitate merging of
https://github.com/ceph/ceph-csi/pull/2377 into devel and release
branches.

https://github.com/ceph/ceph-csi/pull/2378 will remove create &
delete namespace cmds from helm e2e jobs later on.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-11 06:10:46 +00:00
Rakshith R
e6c04e1414 ci: set ttlSecondsAfterFinished=10 to ensure job status can be checked
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 14:47:18 +00:00
Rakshith R
6255ec9a7a ci: update Template yamls apiVersion to template.openshift.io/v1
This was updated to prevent the following error.
Using non-groupfied API resources is deprecated and will be removed in a
future release, update apiVersion to "template.openshift.io/v1" for your
resource.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-09 14:47:18 +00:00
Niels de Vos
c90023ad5d ci: add the root user to the libvirt group
When starting a minikube VM, there are always warning messages like
this:

    X libvirt group membership check failed:
    user is not a member of the appropriate libvirt group

The CI jobs run as root, so minikube works just fine. By adding the root
user to the libvirt group, the warning is hopefully removed.

Note that the libvirt group may not exist yet, as the packages will get
installed in a later stage. This change also adds a check to create the
libvirt group in case it is missing.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-16 09:59:45 +00:00
Niels de Vos
f129719f18 ci: quote username for podman_login()
Jenkins does not like the passing of the username as variable to the
podman_login() function. Calling the function results in an error like

    Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
        Affected argument(s) used the following variable(s): [CREDS_USER]
        See https://jenkins.io/redirect/groovy-string-interpolation for details.
    + ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@n7.pufty.ci.centos.org 'podman login --authfile=~/.podman-auth.json --username=$CREDS_USER --password=**** registry-****.apps.ocp.ci.centos.org'
    Username: Error: error getting username and password: error reading username: EOF

By single quoting the username, just like the password, it may work
better.

Fixes: aca3745e2 ("ci: do not use Groovy string interpolation for credentials")
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-15 10:44:27 +02:00
Niels de Vos
aca3745e24 ci: do not use Groovy string interpolation for credentials
Jenkins warns in the output of CI jobs about the following:

    Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
        Affected argument(s) used the following variable(s): [CREDS_PASSWD, CREDS_USER]
        See https://jenkins.io/redirect/groovy-string-interpolation for details.

Variable with 'single quotes' and without the {curly brackets} are
expecred to not be affected. There is some indirection in the strings
passed to the `sh` function, so this approach might not fix it?

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-15 07:48:41 +00:00
Yug
b4fcd2b585 ci: fix helm job failure
Due to the quotes, the job is accepting
both --deploy-sc and --deploy-secret as
as single parameter.
Removing quotes to help it consider them
as diferent.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 09:08:09 +00:00
Yug
b8c039528d ci: avoid deployment of sc and secret on v3.3
For release 3.3, we will not be deploying storageclass
and secret on helm installation on ci.
Moving forward, devel and all the future release will
have the deployment enabled by default in the ci.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 07:44:47 +00:00
Yug
f69921afb1 ci: remove temp job mini-e2e-helm-sc
Since the job passes, we can now get rid
of the temp job.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 07:44:47 +00:00
Yug
b9a1698a89 e2e: deploy storageclass and secret
deploy secret and storageclass with
deployment via helm.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-06 07:44:47 +00:00
Yug
4a102b44cb e2e: force deploy storageclass and secret
Add a temp job to test the deployment
of sc and secret via helm.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-06-30 15:55:53 +00:00
Niels de Vos
432ff08758 ci: skip Ephemeral-volume kubernetes external-storage tests
Ceph-CSI does not suport (inline) Ephemeral-volumes. Testing this will
continue to fail. The driver configuration can not be used to disable
testing of this feature, so it is done by skipping the tests by pattern
matching.

Updates: #2017
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-30 13:40:08 +00:00
Niels de Vos
2909e85e23 ci: add Kubernetes 1.22 as optional version for CI jobs
Kubernetes 1.22 is in the release process and can be used for testing
already. The CI jobs will be available and can be triggered by leaving a
comment in the PRs like

    /test ci/centos/mini-e2e-helm/k8s-1.22

See-also: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.22
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-30 09:44:54 +00:00
Niels de Vos
a3e235774a ci: run jobs with Kubernetes 1.21
Kubernetes 1.21 is the latest stable release, and Ceph-CSI should be
tested with that. Currently 1.19 is still supported too, so we will need
to run the CI jobs with 1.19, 1.20 and 1.21.

See-also: https://kubernetes.io/releases/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-30 09:44:54 +00:00
Yug
0786d15951 ci: pass --helm-test argument for helm tests
To identify if a test runs on ceph-csi deployed
via helm charts, pass --helm-test parameter with
the E2E args.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-06-24 08:13:15 +00:00
Yug
69091b9273 ci: pass namespace after --namespace flag
Since we are adding support for deployment of sc and
secret via flags, to help script recognize when an
unknown string is passed as a namespace, use
--namespace flag before entering the namespace.

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-06-24 08:13:15 +00:00
Rakshith R
eb4d9259a9 ci: get status from all namespaces
This commit adds capability to get events, pods,
and logs from all namespaces.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-05-20 11:35:19 +00:00
Rakshith R
6337a9fc06 ci: update upgrade-version to v3.3.1 for upgrade tests
This commit updates `upgrade-version` to `v3.3.1` for
e2e upgrade tests.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-05-19 05:17:58 +00:00
Niels de Vos
0dd488374f ci: get logs from all pods, not only the 1st
The JSON formatting will return the name of the first pod, not all pods
on the list. By useing `items[*]` the names of all pods will be
returned. This causes all logs to be fetched, instead of only the logs
from the 1st pod.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-18 04:42:40 +00:00
Rakshith R
ad2c02bdc0 ci: get logs from all containers in pod in system-status.sh
Signed-off-by: Rakshith R <rar@redhat.com>
2021-05-17 12:08:28 +00:00
Madhu Rajanna
4a6b08256b ci: fix running ceph -s in toolbox pod
we need to pass `--` when executing a command
in the container or else it will fail.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-05-17 11:26:20 +00:00
Madhu Rajanna
0cb5abc6fb ci: log previous instance of the container in a pod
if the pod is crashed and restarted the current logs
is not helpful. Logging with `-p` might help us
to log the previous container.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-05-17 11:13:32 +00:00
Niels de Vos
a75430ce21 ci: rebuild infra container images after merge
When the `ci/centos` branch has been merged, the container images that
are built from it may need to be rebuilt. Instead of conditionally
rebuilding images only on a change, just rebuild them always, as there
are very few changes in the CI branch anyway.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-17 10:12:28 +00:00
Niels de Vos
06409fcbd6 ci: add CronJob for automated image mirroring in the CI registry
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-17 06:25:51 +00:00
Niels de Vos
2f6400fe57 ci: add mirror-images container image for automated mirroring
The ImageStream and BuildConfig use the Containerfile to build a
container image in the OpenShift registry so that automated image
mirroring can be configured.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-17 06:25:51 +00:00
Niels de Vos
4ec5b6f045 ci: add script and config to mirror images to local CI registry
This script reads images.txt and copies the images from docker.io (or
other registries) to the local registry in the CI.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-17 06:25:51 +00:00
Niels de Vos
c40a055628 ci: log events from the rook-ceph namespace on failure
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-14 16:45:24 +00:00
Niels de Vos
0049638e64 ci: get logs from all pods in the rook-ceph namespace on failure
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-14 16:45:24 +00:00
Niels de Vos
7ebc8306ef ci: get the logs from the Ceph cluster pods
It seems that `/var/log/rook` inside the VM does not contain any files.
Getting the logs from the Pods through kubectl may not be as stable, but
it should get some logs when minikube is still/again available.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-14 13:06:17 +00:00
Niels de Vos
410db81215 ci: include status of the Rook deployment on failure
More details of the Rook (and Ceph) deployment should be useful when
troubleshooting CI failures. This now includes the status of the most
important Kubernetes objects, and all the logs Ceph stores on the host.

Updates: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-12 14:42:09 +00:00
Yug
86d5c17ba4 ci: use "-T" to display the filesystem type
Provide filesystem type information with logs

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-05-12 09:36:53 +00:00
Niels de Vos
bae519db07 ci: use "top -w" for untruncated wide output
Without the `-w` argument, the output of `top` gets truncated, and the
commandline of the processes is not comlete. It would be useful to eb
able to tell which command uses 100% CPU in an output like:

  17377 root      20   0  110.8m   8.2m   0.0   0.1   0:00.89 S  `- containerd+
  17414 167       20   0 1036.7m  59.6m   0.0   0.4   0:03.47 S      `- ceph-o+
  40875 root      20   0  283.9m  30.4m 100.0   0.2   0:00.23 R      `- ceph

Updates: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-12 05:12:09 +00:00
Niels de Vos
e599e95f25 ci: in case of a failure, return error after logging system status
It seems that it is required to re-throw the error after a catch{..}
block. Without this, and a successful execution of system-status.sh, the
CI jobs get marked as SUCCESS, even when there was a failure.

Fixes: e36155283 "ci: run system-status.sh in case a job fails"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-11 13:13:39 +00:00
Niels de Vos
5b03721a58 ci: copy system-status.sh script to the bare metal node
Without the script on the node, it can not be executed...

Fixes: e36155283 "ci: run system-status.sh in case a job fails"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-11 12:33:03 +00:00
Niels de Vos
e36155283b ci: run system-status.sh in case a job fails
The new `system-status.sh` script logs the status of the host and the
minikube VM. This gets executed when a CI job fails, and should aid in
troubleshooting spurious failures.

Updates: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-11 11:27:01 +00:00
Niels de Vos
4ef36aed0c ci: increase memory for minikube VM to 14GB
The e2e tests very regulary hit a timeout where the Kubernetes API
becomes unreachable for 3 minutes. Hopefully it helps when more RAM is
available to the VM.

Updates: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-07 02:04:35 +00:00
Niels de Vos
2fb24c3c4c ci: provide k8s-e2e-external-storage jobs for different k8s versions
The Kubernetes e2e external storage tests from v1.21 do not work yet
with Ceph-CSI. In order to address the issues, the job is now provided
and can be run with:

     /test ci/centos/k8s-e2e-external-storage/1.21

The job for v1.20 is enabled by default, and identified by the
ci/centos/k8s-e2e-external-storage/1.20 context in PRs.

Updates: #2017
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-03 07:30:50 +00:00
Rakshith R
ddd10c3245 ci: pull and push busybox image after installing minikube
k8s-e2e-external-storage fails with error
`./podman2minikube.sh: line 16: minikube: command not found`.
This commit fixes it by starting minikube before calling
./podman2minikube.sh.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-04-29 10:50:07 +00:00
Madhu Rajanna
1cc12b1a1c ci: pre-pull busybox container image
pre-pull the required busy box container
image in k8s-e2e-external-storage.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-28 12:10:40 +00:00
Madhu Rajanna
c5ce8e1a95 ci: add missing ci_registry k8s-e2e-external-storage groovy
added missing ci_registry variable to the
k8s-e2e-external-storage groovy file.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-27 13:02:51 +00:00
Madhu Rajanna
43dd2a20e6 ci: pre-pull the required container images
added code to pre-pull the required container
images to run the k8s-e2e-external-storage E2E.

fixes: #2023

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-04-27 11:08:16 +00:00
Niels de Vos
918d5e0870 ci: enable running of k8s-e2e-external-storage job by default
The job seems stable, and can be run by default now. Once it has been
run on several PRs, the `ci/centos/k8s-e2e-external-storage` job ID
can be added to the Mergify configuration.

See-also: https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/k8s-e2e-external-storage/activity
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-26 04:18:39 +00:00
Niels de Vos
955559a235 ci: use Helm 3.1.2 in test container
By default a version of Helm is used that does not want to get
installed. Using the same version as the devel branch makes the testing
work again.

See-also: helm/helm#9617
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-20 07:08:16 +00:00
Niels de Vos
b5aa0b11d7 ci: add optional Kubernetes 1.21 job
To test a PR with Kubernetes 1.21, leave a comment in the PR like:

    /test ci/centos/mini-e2e-helm/k8s-1.21

The status of the job will be recorded in the PR, but running this job
is not required (yet).

Updates: #1963
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-20 07:08:16 +00:00
Niels de Vos
c04a319aa9 ci: only abort on doc-change when running for PRs
In case a job has been started without a PR (manual, or timed), the
current checked out branch matches the original as there are not
additional changes in the tree. There is no need to abort the jobs when
the skip-doc-change.sh script did not detect any non-doc changes, as
there are no changes at all.

Updates: #1963
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-09 09:53:22 +00:00
Niels de Vos
8f84e592d5 ci: do not re-checkout current branch
When tests are started manually (through the Jenkins webui), there is no
PR associated with the job. That means the `git_since` and `ref` are
equal. Trying to create a new branch named `ref` will not work, as the
branch was already created when cloning the repository with `git_since`.

With this change, Jenkins jobs can be started manually. This makes it
possible to run regular/nightly jobs as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-04-08 08:38:11 +00:00