Commit Graph

12 Commits

Author SHA1 Message Date
Niels de Vos
03413a53fd e2e: retry when a pod does not have a host assigned (yet)
Errors like "pod nfs-820 does not have a host assigned" seem to get
reported when a Pod is not completely started yet, or was restarted
while trying to access it.

Reported: https://github.com/ceph/ceph-csi/pull/4656#issuecomment-2151794926
Signed-off-by: Niels de Vos <ndevos@ibm.com>
2024-06-06 19:23:15 +00:00
Niels de Vos
a04a0ecc9f ci: retry command in Pod on "unable to upgrade connection" error
Sometimes executing a command in a Pod fails with "unable to upgrade
connection". This is most likely a temporary situation, and retrying
hopefully reduces the number of spurious failures because of it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-07-29 16:39:26 +00:00
Niels de Vos
eab2ec42bf e2e: allow kubectlDelete to succeed with NotFound error
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2022-05-10 00:43:43 +00:00
Niels de Vos
ba60566cfb e2e: retry on "transport: missing content-type field" error
The e2e sometimes fail getting objects like PVCs from the Kubernetes API
server, and log the following error:

    Error getting pvc "rbd-6940" in namespace "rbd-694": rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field

By checking the error message, and initiating a retry on this failure,
CI jobs should fail less regulary.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-11-23 17:13:56 +00:00
Niels de Vos
1fa8939e84 e2e: retry when a "transport is closing" error is hit
There have been occasional CI job failures due to "transport is closing"
errors. Adding this error to the isRetryableAPIError() function should
make sure to retry the request until the connection is restored.

Fixes: #2613
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-11-17 14:07:07 +00:00
Rakshith R
dbf2eb3905 e2e: ignore lines with 'Warning' in isAlreadyExistsCliError()
Signed-off-by: Rakshith R <rar@redhat.com>
2021-08-27 07:14:10 +00:00
Madhu Rajanna
dd9fabf747 e2e: add isAlreadyExistsCLIError to check known error
added isAlreadyExistsCLIError to check for known error.
if error is already exists we are considering it
as a success.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-06 08:03:18 +00:00
Niels de Vos
e75d308b9c e2e: isRetryableAPIError() should match any etcdserver timeout
framework.RunKubectl() returns an error that does not end with
"etcdserver: request timed out", but contains the text somewhere in the
middle:

    error running /usr/bin/kubectl --server=https://192.168.39.57:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-a44ec4b4 create -f -:
    Command stdout:

    stderr:
    Error from server: error when creating "STDIN": etcdserver: request timed out

    error:
    exit status 1

isRetryableAPIError() should  return `true` for this case as well, so
instead of using HasSuffix(), we'll use Contains().

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-23 12:20:16 +00:00
Niels de Vos
5c016b4b94 e2e: retry on "connect: connection refused" errors
Sometimes there are failures in the e2e suite when connecting to the
etcdserver fails. The following error was caught:

    INFO: Error getting pvc "rbd-pvc" in namespace "rbd-1318": Get "https://192.168.39.222:8443/api/v1/namespaces/rbd-1318/persistentvolumeclaims/rbd-pvc": dial tcp 192.168.39.222:8443: connect: connection refused
    FAIL: failed to create PVC with error failed to get pvc: Get "https://192.168.39.222:8443/api/v1/namespaces/rbd-1318/persistentvolumeclaims/rbd-pvc": dial tcp 192.168.39.222:8443: connect: connection refused

If etcdserver was only briefly unavailable, one or more retries might be
sufficient to have the test pass.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-21 13:08:41 +00:00
Niels de Vos
075a4087d7 e2e: mark "etcdserver: request timed out" errors as retryable
There are regular CI failures where etcdserver times out. These errors
seem not to get caught by any of the existing error comparing. Matching
the error by string should prevent temporary etcdserver issues now too.

Updates: #2218
Closes: #1969
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-07-14 10:18:51 +00:00
Madhu Rajanna
3e9172ab31 e2e: check isRetryableAPIError during polling
check isRetryableAPIError error when the
function is polling for the resouce.

updates: #1969

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-06-30 10:00:44 +00:00
Madhu Rajanna
1db3f03856 e2e: IsRetryableAPIError is not available in kubernetes 1.20.0
IsRetryableAPIError is not available in latest
kubernetes release ie 1.20.0 created a internal
function called isRetryableAPIError for the same.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-17 16:04:54 +00:00