Commit Graph

1005 Commits

Author SHA1 Message Date
Niels de Vos
ed90535c23 minikube: fix ShellCheck errors
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-11 14:05:19 +05:30
Niels de Vos
60a8e879e5 ci: add make target for containerized tests
`make containerized-test` has been added as a make target. This runs the
'make test' target in a container. All test dependencies are installed
in the container once, and the container is kept around for running
`make containerized-test` subsequently.

The test container is based on Fedora:latest so that all test tools get
easily installed and are available in a recent version. The production
container is based on the Ceph container which has CentOS as Operating
System and therefor a more stable (too old) toolset.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-11 14:05:19 +05:30
Madhu Rajanna
fbc7d09670 Revert "deploy: Template changes for release v2.1.2"
This reverts commit a03675e3ae.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-10 12:50:43 +05:30
Madhu Rajanna
a03675e3ae deploy: Template changes for release v2.1.2
Updated the templates and the required scripts
to make a v2.1.2 release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-05 13:56:13 +05:30
Madhu Rajanna
958d2b3974 e2e: Add testcase of invalid mount option
Added a test case to cover unmap of rbd image
if the mounting fails. if we pass the invalid
mount option the expectation is that mounting
of rbd image to stagingpath fails. as the unmap
happens it should not block the rbd pvc deletion
saying rbd image is in-use.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 44da7ffb4e)
2020-06-05 12:20:47 +05:30
Madhu Rajanna
c06533de6b e2e: Add option to set mountoptions in sc
Updated E2E code to pass mountoptions when
creating a storageclass.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 78cdf9d4db)
2020-06-05 12:20:47 +05:30
Madhu Rajanna
c2d0cc9ec1 rbd: unmap rbd image if the mounting fails
There is a bug in current code where the devicePath
is always empty and the rbd image unmap never
happens if nodeplugin fails to mount the rbd image
to the stagingpath.
This is a fix to unmap the rbd image if some issue
occurs after rbd image is mapped.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2f7e51076b)
2020-06-05 12:20:47 +05:30
Madhu Rajanna
4c6d384991 cephfs: use new 'ceph fs resize' command when available
Use ceph fs resize command when its available
in the cluster.if its not available fallback to
old style of resizing the subvolume

ceph change log:https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-8-nautilus

Fixes #1002

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit b065726f19)
2020-05-13 15:39:31 +05:30
Madhu Rajanna
dfe7f1b100 template: revert back changes done for v2.1.1 release
This PR reverts the templates changes and other
changes done for release-v2.1.1.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-12 18:35:37 +05:30
Humble Chirammal
66645cf6f5 cephfs: use new 'ceph fs resize' command when available
This reverts commit 9022d899eb.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-12 18:34:52 +05:30
Madhu Rajanna
9022d899eb cephfs: use new 'ceph fs resize' command when available
Use ceph fs resize command when its available
in the cluster.if its not available fallback to
old style of resizing the subvolume

ceph change log:https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-8-nautilus

Fixes #1002

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit b065726f19)
2020-05-12 14:49:01 +05:30
Madhu Rajanna
d503729fc4 Prepare for v2.1.1 release
updated required files to do a v2.1.1 release

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 12:53:40 +05:30
Madhu Rajanna
f4d6e31764 rbd: default nouuid if the formattype is xfs
The problem happens when multiple PVCs with the
same UUID are attached/mounted on a node. This
can happen after creating a PVC from a snapshot,
or cloning a PVC.

make nouuid as the default mount option if
the format type is xfs to avoid mounting
issues.

updates: #966

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 22a86c568e)
2020-05-05 12:51:56 +05:30
Niels de Vos
3005fde6f7 travis: group build/e2e testing in stages
This reduces the number of concurrent jobs that get executed. The "build
test" jobs are relatively quick, and give developers the 1st feedback.
Longer e2e testing now only happens once the build tests have passed.

See-also: https://docs.travis-ci.com/user/build-stages/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 43fbc00d49)
2020-05-04 15:12:41 +05:30
Madhu Rajanna
e1c919bddb revert back the ceph base image to 14.2
as we are facing the xfs format issue ie ` wrong fs type,
bad option, bad superblock on /dev/rbdx, missing codepage or
helper program, or other error` as the xfsprogs
version has changed in ceph:v15 base image.
due to this cephcsi is not working in older
kernel versions

updates #966

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-04 14:37:36 +05:30
Niels de Vos
b6b8be3b20 add build step logging to multi-arch build
With extra logging, there is no need to call `travis_wait` anymore. In
addition the `travis_wait` command blocks output, so the build steps are
not reported until the script finishes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 99dabca319)
2020-04-28 11:01:28 +00:00
Niels de Vos
b8c17b096f move build-multi-arch-image.sh to scripts/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 5a4517e38e)
2020-04-28 11:01:28 +00:00
Niels de Vos
a4f4458c02 travis: remove 'travis_wait' from deploy section
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 9fd4005819)
2020-04-28 11:01:28 +00:00
Niels de Vos
b615e2b61f deploy: log steps while tasks are running
By repeated logging of the current step, Travis CI should receive some
terminal activity and is expected to not timeout anymore.

Fixes: #982
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 20801d6563)
2020-04-28 11:01:28 +00:00
Niels de Vos
0d12fb6a4e tests: allow external-sources for shellcheck
While introducing scripts/build_step.inc.sh the tests start to fail as
the script is included and each shell script is tested separately. By
adding the option --external-sources to shellcheck, the related warnings
are not reported.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit fd62a5897a)
2020-04-28 11:01:28 +00:00
Niels de Vos
00324171f1 scripts: add build_step logging functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 52fa4f10da)
2020-04-28 11:01:28 +00:00
Madhu Rajanna
d0369e9fdf Add grpc 1.27 version require section in go.mod to avoid CI issue
This PR addes GRPC 1.27 to required section in go.mod
file. this is overridden by 1.26 in replace section

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 3b608f130a)
2020-04-23 17:11:04 +00:00
Niels de Vos
f766bb37e5 tests: fail when "make mod-check" detects a modified go.mod file
`go mod verify` does not fail when all moduled under vendor/ are
updated. However it does update `go.mod` in case it does not reflect the
downloaded versions.

By adding this additional verification in `make mod-check`, future
updates to vendor/ will require a correct go.mod as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit 22c0baa4b0)
2020-04-23 08:37:30 +00:00
Niels de Vos
917a75de99 go.mod: go not list grpc as direct dependency
The recent update to k8s 1.18.0 causes an issue in go.mod that gets
corrected by running `go mod verify`. grpc should be used in version
1.26, and not in the expected update 1.27.

By removing the dependency and keeping grpc in the 'replace' section,
`go mod verify` seems to be happy.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
(cherry picked from commit b3664cd631)
2020-04-23 08:37:30 +00:00
Madhu Rajanna
bad06a1751 Add support to build multi architecture image
This is back port fo https://github.com/ceph/ceph-csi/pull/866
to release-v2.1 branch

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 12:05:30 +05:30
Madhu Rajanna
67be05f547 Revert back the changes made for release 2.1.0
As the release is out, This PR revert back the
changes to help pushing v2.1-canary image and
helm charts

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-16 11:01:39 +05:30
Humble Chirammal
b38f2c5c31 Prepare for v2.1.0 release
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-15 18:47:33 +05:30
Madhu Rajanna
e4a4850056 Upgrade: Upgrade doc for v2.1.0
This PR updates the upgrade document
from v2.0.x to v2.1.0

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 2099600aa6)
2020-04-15 12:10:32 +00:00
Madhu Rajanna
38ba4c1cfd Fix goimports issue in CI
Fix below error in current codebase
File is not `goimports`-ed with -local
github.com/ceph/ceph-csi

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 5b14cc9272)
2020-04-15 10:53:58 +00:00
Madhu Rajanna
d2e26d466b Fix typo in golangci config file
This PR fixes the typo in golang CI config file

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 6998c0d141)
2020-04-15 10:53:58 +00:00
Madhu Rajanna
774ceea04d RUN CI on release-v2.1 branch
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-15 14:50:00 +05:30
Kevin Fox
19c5c1d5e6 Support externally managed configmap.
https://github.com/ceph/ceph-csi/issues/927

Signed-off-by: Kevin Fox <Kevin.Fox@pnnl.gov>
(cherry picked from commit d4c4954c9d)
(cherry picked from commit 6f73d4e3ca)
2020-04-15 13:17:45 +05:30
ShyamsundarR
1a8f8e3c24 Add support for erasure coded pools
This commit adds support to mention dataPool parameter for the
topology constrained pools in the StorageClass, that can be
leveraged to mention erasure coded pool names to use for RBD
data instead of the replica pools.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
3f06fedf61 Add unit tests for pkg/util/topology.go file
NOTE: If GetTopologyFromDomainLabels took a k8s client as an
argument, we could use the kubernetes fake client to test this
function in addition. This would though break the layers, exposing
k8s semantics in the code into the higher layers. Hence not
changing the same.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
e73921f268 Add e2e tests for topology based provisioning
- This commit adds tests only for RBD, as CephFS still needs
an enhancement in CephFS subvolume commands to effectively use
topology based provisioning

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
d7ea523540 Update Helm charts to support topology
- Added requires chart variables
- Updated templates with required additions based on the
variables
- Re-added older nodeplugin cluster role/rolebinding as needed

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
5c4abf8347 Add topology support to ceph-csi
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
chenxu1990
5475022bc3 Document about stale resource cleanup
1. when user delete pv manual, it will result in stale metadata and image in ceph
2020-04-14 11:29:07 +00:00
Humble Chirammal
34fc1d847e Changes to accommodate client-go changes and kube vendor update
to v1.18.0

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-14 10:50:12 +00:00
Humble Chirammal
4c96ad3c85 Changes to e2e to accomodate client-go changes and RunKubectlInput
With client-go v1.18.0 there is a change where Signatures on methods
in generated clientsets, dynamic, metadata, and scale clients have been
modified to accept context.Context as a first argument.
Signatures of Create, Update, and Patch methods have been updated to accept
CreateOptions, UpdateOptions and PatchOptions respectively.
Signatures of Delete and DeleteCollection methods now accept DeleteOptions
by value instead of by reference

The framework.RunkubectlInput now accepts namespace as the first parameter
which is also accommodated with this PR.

Signed-off-by: Humble Chirammal hchiramm@redhat.com
2020-04-14 10:50:12 +00:00
Madhu Rajanna
60a394b397 Fix helm clean in functional test
there was an issue in helm cleanup
script in functional tests. This Fixes
the same.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-10 14:10:01 +00:00
Niels de Vos
14276bf642 rbd: fallback to inline image deletion if adding it as a task fails
Fixes: #858
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-10 11:16:37 +00:00
Madhu Rajanna
22f3dc3556 E2E: Add Ability to test with different user #919
Updated E2E to use a normal user other than admin user,
who is having access to create and mount cephfs PVC and
create and map rbd PVC

we will use the user created by rook who is having above
access.

Signed-off-by: Madhu Rajanna madhupr007@gmail.com

This PR adds a test case for #904

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Madhu Rajanna
a1f5425b88 Update cephcsi secret created by rook
as we need to include the encryption
secret key inside the secret created by the
rook, This PR will add the key and value required
for the encryption inside the secrets.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Niels de Vos
c3cf6be6a7 util/conn_pool: open a connection with requested user
Use the Credentials.ID in combination with the keyfile to connect to the
Ceph cluster. This makes it possible to use different users for
different tasks on the cluster.

Fixes: #904
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-09 15:10:55 +00:00
Humble Chirammal
2cc59ca411 Install go from release tar ball and update to 1.13.9
The current version of go ( 1.12.x) is causing issues
on some method call under errors package. This patch
could help to overcome the same. More details about the failure
is @https://github.com/ceph/ceph-csi/pull/917#issuecomment-609998502

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-09 08:12:37 +00:00
Niels de Vos
19cc28ddea util/cephcmds: have GetOMapValue() return ErrPoolNotFound
On occasion the e2e tests fail as there is an unexpected error while
deleting an RBD image. The particular tests forcefully removes the pool
where the RBD image is stored. Deleting a volume that has been removed
already (or when its parent pool has been wiped), should succeed.

By catching the error that a pool does not exist (anymore), the
provisioner responds to the DeleteVolume request with succes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-08 12:44:42 +00:00
Humble Chirammal
02367c4a3f Cleanup vendor dependencies
- `go mod {tidy and vendor}.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-08 08:12:13 +00:00
Madhu Rajanna
58765e27a0 Resizer: Update resizer image version
Recently resizer 0.5.0 has been released.
This PR updated the resizer container from
v0.4.0 to v0.5.0

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 12:06:54 +00:00
Madhu Rajanna
8b70e68f65 Fix issue related to http status code in vault
the status code for success should be between
200 to 300.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00