Commit Graph

1336 Commits

Author SHA1 Message Date
wilmardo
157e8f4c75 util: use MiB and GiB constants from k8s helpers
Signed-off-by: wilmardo <info@wilmardenouden.nl>
2020-04-21 13:56:40 +00:00
Niels de Vos
4d60e76241 travis: split static checks and continue with tests in case one fails
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 07:01:10 +00:00
Niels de Vos
bd6393cf76 travis: add "make containerized-test" job
Add the containerized test to the existing containerized-build job so
that resources are shared a little.

Both tests use a different distribution, similar to what a developer has
on their system, and production deployments on a server.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 05:26:32 +00:00
Niels de Vos
3c6054ddf9 add make target for containerized tests
`make containerized-test` has been added as a make target. This runs the
'make test' target in a container. All test dependencies are installed
in the container once, and the container is kept around for running
`make containerized-test` subsequently.

The test container is based on Fedora:latest so that all test tools get
easily installed and are available in a recent version. The production
container is based on the Ceph container which has CentOS as Operating
System and therefor a more stable (too old) toolset.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 05:26:32 +00:00
Niels de Vos
58c2a3f15f minikube: fix ShellCheck errors
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 05:26:32 +00:00
chenxu1990
b7d46c1b79 Insert backend storage name into volumeAttributes
1. insert "imageName" field to pv with storage rbd
2. insert "subvolumeName" field to pv with storage cephfs
2020-04-20 07:30:03 +00:00
Niels de Vos
7bdb42c14a doc: update to current release tracking method
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-17 09:16:58 +00:00
hchiramm
a219b2f9d4 Update README for v2.1.0 release.
Signed-off-by: hchiramm <hchiramm@redhat.com>
2020-04-15 11:19:57 +00:00
Madhu Rajanna
2099600aa6 Upgrade: Upgrade doc for v2.1.0
This PR updates the upgrade document
from v2.0.x to v2.1.0

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-15 09:48:31 +00:00
Madhu Rajanna
5b14cc9272 Fix goimports issue in CI
Fix below error in current codebase
File is not `goimports`-ed with -local
github.com/ceph/ceph-csi

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-15 09:18:47 +00:00
Madhu Rajanna
6998c0d141 Fix typo in golangci config file
This PR fixes the typo in golang CI config file

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-15 09:18:47 +00:00
Madhu Rajanna
a3d2980311 Update Mergify for v2.1.0 backports
Updated Mergify rules for backporting PR's
to release-v2.1 branch and to auto merge the
PR if author is mergify bot

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-15 13:00:22 +05:30
Kevin Fox
6f73d4e3ca Support externally managed configmap.
https://github.com/ceph/ceph-csi/issues/927

Signed-off-by: Kevin Fox <Kevin.Fox@pnnl.gov>
(cherry picked from commit d4c4954c9d)
2020-04-15 07:21:20 +00:00
ShyamsundarR
1a8f8e3c24 Add support for erasure coded pools
This commit adds support to mention dataPool parameter for the
topology constrained pools in the StorageClass, that can be
leveraged to mention erasure coded pool names to use for RBD
data instead of the replica pools.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
3f06fedf61 Add unit tests for pkg/util/topology.go file
NOTE: If GetTopologyFromDomainLabels took a k8s client as an
argument, we could use the kubernetes fake client to test this
function in addition. This would though break the layers, exposing
k8s semantics in the code into the higher layers. Hence not
changing the same.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
e73921f268 Add e2e tests for topology based provisioning
- This commit adds tests only for RBD, as CephFS still needs
an enhancement in CephFS subvolume commands to effectively use
topology based provisioning

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
d7ea523540 Update Helm charts to support topology
- Added requires chart variables
- Updated templates with required additions based on the
variables
- Re-added older nodeplugin cluster role/rolebinding as needed

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
ShyamsundarR
5c4abf8347 Add topology support to ceph-csi
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-14 14:14:29 +00:00
chenxu1990
5475022bc3 Document about stale resource cleanup
1. when user delete pv manual, it will result in stale metadata and image in ceph
2020-04-14 11:29:07 +00:00
Humble Chirammal
34fc1d847e Changes to accommodate client-go changes and kube vendor update
to v1.18.0

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-14 10:50:12 +00:00
Humble Chirammal
4c96ad3c85 Changes to e2e to accomodate client-go changes and RunKubectlInput
With client-go v1.18.0 there is a change where Signatures on methods
in generated clientsets, dynamic, metadata, and scale clients have been
modified to accept context.Context as a first argument.
Signatures of Create, Update, and Patch methods have been updated to accept
CreateOptions, UpdateOptions and PatchOptions respectively.
Signatures of Delete and DeleteCollection methods now accept DeleteOptions
by value instead of by reference

The framework.RunkubectlInput now accepts namespace as the first parameter
which is also accommodated with this PR.

Signed-off-by: Humble Chirammal hchiramm@redhat.com
2020-04-14 10:50:12 +00:00
Madhu Rajanna
60a394b397 Fix helm clean in functional test
there was an issue in helm cleanup
script in functional tests. This Fixes
the same.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-10 14:10:01 +00:00
Niels de Vos
14276bf642 rbd: fallback to inline image deletion if adding it as a task fails
Fixes: #858
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-10 11:16:37 +00:00
Madhu Rajanna
22f3dc3556 E2E: Add Ability to test with different user #919
Updated E2E to use a normal user other than admin user,
who is having access to create and mount cephfs PVC and
create and map rbd PVC

we will use the user created by rook who is having above
access.

Signed-off-by: Madhu Rajanna madhupr007@gmail.com

This PR adds a test case for #904

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Madhu Rajanna
a1f5425b88 Update cephcsi secret created by rook
as we need to include the encryption
secret key inside the secret created by the
rook, This PR will add the key and value required
for the encryption inside the secrets.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-09 15:10:55 +00:00
Niels de Vos
c3cf6be6a7 util/conn_pool: open a connection with requested user
Use the Credentials.ID in combination with the keyfile to connect to the
Ceph cluster. This makes it possible to use different users for
different tasks on the cluster.

Fixes: #904
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-09 15:10:55 +00:00
Humble Chirammal
2cc59ca411 Install go from release tar ball and update to 1.13.9
The current version of go ( 1.12.x) is causing issues
on some method call under errors package. This patch
could help to overcome the same. More details about the failure
is @https://github.com/ceph/ceph-csi/pull/917#issuecomment-609998502

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-09 08:12:37 +00:00
Niels de Vos
19cc28ddea util/cephcmds: have GetOMapValue() return ErrPoolNotFound
On occasion the e2e tests fail as there is an unexpected error while
deleting an RBD image. The particular tests forcefully removes the pool
where the RBD image is stored. Deleting a volume that has been removed
already (or when its parent pool has been wiped), should succeed.

By catching the error that a pool does not exist (anymore), the
provisioner responds to the DeleteVolume request with succes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-08 12:44:42 +00:00
Humble Chirammal
02367c4a3f Cleanup vendor dependencies
- `go mod {tidy and vendor}.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-04-08 08:12:13 +00:00
Madhu Rajanna
58765e27a0 Resizer: Update resizer image version
Recently resizer 0.5.0 has been released.
This PR updated the resizer container from
v0.4.0 to v0.5.0

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 12:06:54 +00:00
Madhu Rajanna
8b70e68f65 Fix issue related to http status code in vault
the status code for success should be between
200 to 300.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
37c4e3447d Add helm chart SA to vault.yaml
we need to provide access to the Service
account created with helm charts to access
the vault service.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
d09ffbd6de helm: add helm charts E2E
This PR adds the support for helm
installation, and cephcsi helm charts
deployment and teardown and also runs E2E
on for helm charts.

Add socat to provide port forwadring access for helm

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Madhu Rajanna
b2dfcae802 Update configmap if already present
in e2e if the configmap is ready present,
we need to update it to make life simpler
for helm chart e2e.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Xiaopei Yuan
22c432cece Fix typos in charts values comments
Signed-off-by: Xiaopei Yuan <yuanxiaopei@kpaas.io>
2020-04-06 09:45:06 +00:00
Niels de Vos
a1de56dbd3 tests: in case 'go test' is run in a container, skip TestGetPIDLimit()
In (standard, non-privileged) container environments the /sys/fs/cgroup
mountpoint is not available. This would cause the tests to fail, as
TestGetPIDLimit() tries to write to the cgroup configuration.

The test will work when run as root on a privileged container or
directly on a host (as Travis CI does).

Setting the CEPH_CSI_RUN_ALL_TESTS environment variable to a non-empty
value will cause the test to be executed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-02 06:08:03 +00:00
Madhu Rajanna
15578432a4 Move cephfs support from alpha to beta
Moving cephfs support from Alpha
to Beta in cephcsi.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-01 12:19:22 +00:00
Madhu Rajanna
bcd646ee55 Deprecate grpc metrics in ceph-csi
As kubernetes CSI sidecar is exposing the
GRPC mertics we can make use of the same in
ceph-csi we dont need to expose our own.

update: #881

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-01 11:59:37 +00:00
Madhu Rajanna
f2e98a19e8 Install git in devel dockerfile
looks like git is not installed by default
in v15 base image.This PR installs the
git which is required to make containerized
build.

set GO111MODULE=on in dockerfile

we need to set GO111MODULE=on to fix
"build flag -mod=vendor only valid when using modules"
issue

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-01 10:35:23 +00:00
Madhu Rajanna
c629c3bf52 update base image in Dockerfile
As we have the octopus as the latest
release base image,this PR updates the
base image in Dockerfile

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-01 10:35:23 +00:00
Madhu Rajanna
3c3a624d1a Log error message for cephfs
If the loading of cephfs fuse or
the mount.ceph command fails log it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-01 09:35:40 +00:00
Wong Hoi Sing Edison
ebe5aa00cf Upgrade: csi-node-driver-registrar from v1.2.0 to v1.3.0
See https://github.com/kubernetes-csi/node-driver-registrar/releases/tag/v1.3.0
See https://github.com/kubernetes-csi/node-driver-registrar/blob/v1.3.0/CHANGELOG-1.3.md
2020-04-01 08:39:37 +00:00
Wong Hoi Sing Edison
d67ad47c45 Upgrade: csi-attacher from v2.1.0 to v2.1.1
This PR follow up the missing files from ceph-csi:v2.0.1, besides:

  - deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml
  - deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml

See https://github.com/kubernetes-csi/external-attacher/releases/tag/v2.1.1
See https://github.com/kubernetes-csi/external-attacher/blob/v2.1.1/CHANGELOG-2.1.md
2020-03-31 05:33:18 +00:00
Madhu Rajanna
84aa1ba392 Use Error instead of Errorf
If the string formatting is not required
use Error.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-30 08:53:16 +00:00
Humble Chirammal
b1dfcb4d7e Correct static errors and source code comments.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-03-30 05:13:24 +00:00
xu.chen
399f0b0d89 Audit log and follow klog standard 2020-03-27 09:24:52 +00:00
Niels de Vos
2124f6e775 travis: add job to test 'make containerized-build'
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-26 08:45:53 +00:00
Niels de Vos
7381253ee0 build: add an option to compile in a container
This makes it possible to build on any platform that supports Linux
containers. The container image used for building is created once, or on
updating the `scripts/Dockerfile.build` and is cached afterwards.

To build the executable in a container, use `make containerized-build`
and everything will be done automatically. The executable will also be
available on the usual location.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-26 08:45:53 +00:00
root
ae4d269836 fix typos 2020-03-24 15:43:03 +00:00
Niels de Vos
825448825c examples: rbd/storageclass.yaml causes Python excetion with yamllint
While running the 'make test' target and have 'yamllint' available, the
test fails with the following exception:

yamllint -s -d {extends: default, rules: {line-length: {allow-non-breakable-inline-mappings: true}},ignore: charts/*/templates/*.yaml} ./examples/rbd/storageclass.yaml
Traceback (most recent call last):
  File "/usr/local/bin/yamllint", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.6/site-packages/yamllint/cli.py", line 181, in run
    problems = linter.run(f, conf, filepath)
  File "/usr/local/lib/python3.6/site-packages/yamllint/linter.py", line 237, in run
    content = input.read()
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1947: ordinal not in range(128)

The quotes used in the comments seem to be non-ascii characters.
Replacing these with standard " makes the test pass again.

This problem occurred while running tests in a container based on the
Ceph image (CentOS-7) with Python 3. Travis CI might still use Python 2
for yamllint, and hide the problem.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-24 14:42:25 +00:00