Commit Graph

1070 Commits

Author SHA1 Message Date
Mudit Agarwal
fb1e689648 test: add e2e test for cephFS static PV
With the change in https://github.com/ceph/ceph-csi/issues/382, support for static PV for CephFS was added.
This change is to add e2e test case for the same to ensure that we do not regress the functionality.

Issue: https://github.com/ceph/ceph-csi/issues/669

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-05-18 08:30:50 +00:00
Niels de Vos
eff4f4ba50 ci: add commitlint target to the Makefile
When passing `GIT_SINCE` as an argument, the commits from then on
(defaults to origin/master) are used. A CI job can pass the base branch
or ID so a range of patches gets checked.

To test the last four commits, one can run:

    $ make containerized-test TARGET=commitlint GIT_SINCE=HEAD~4

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 18:20:29 +00:00
Niels de Vos
aeb5c75da7 ci: install commitlint in the test container-image
The `commitlint` command can be used to verify the subject of commit
messages, so it is added to the $PATH.

See-also: https://commitlint.js.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 18:20:29 +00:00
Niels de Vos
ead02ea8e6 ci: add .commitlintrc.yml for forcing commitlint CI checks
The rules are documented at https://commitlint.js.org/#/reference-rules
and can easily be added to the configuration.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 18:20:29 +00:00
John Mulligan
8dea42a624 journal: rename CSIJournal to Config
This better reflects what it actually does now.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-15 15:14:48 +00:00
John Mulligan
52603d595a journal: split journal types creating a new Connection type
Before, the one CSIJournal type was handling both configuration and
providing methods to make changes to the journal. This created the
temptation to modify the state of the global configuration object to
enact changes through the method calls.

This change creates a new type `journal.Connection` that takes the
monitors and credentials to create a short(er)-lived object to actually
read and make changes on the journal. This also avoid mixing the
arguments needed to connect to the cluster with the arguments needed
for the various journal read & update calls.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-15 15:14:48 +00:00
Madhu Rajanna
f0b3cee94a e2e: define replicapool as constant
the static check is failing as the replicapool
is used in 3 or more places, we need to define
a variable and use it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
76a55350aa e2e: Pass context.TODO() to kubernetes snapshot client
in external-snapshotter the client has changed we
need to pass the context to snapshot api calls.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
1f13692000 ci: skip snapshot E2E if kube<1.17+
snapshot beta CRD wont work if the
kubernetes version is less than 1.17.0
as the snapshot CRD wont be installed
we cannot test the snapshot,so disabling
it if the kube version is less than 1.17

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
538b854853 script: Delete snapshot CRD created by cephcsi in rook
when we deploy rook+cephcsi for E2E, the
external-snapshotter in cephcsi deployed by
rook will create the CRD, we need to delete
the crd created by external-snapshotter.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
4606b5042f vendor: Vendor changes to add external-snapshotter
Few other depedencies got updated which might
be required for snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
c5bf95b625 e2e: Enable E2E for rbd snapshot
Updated E2E to use beta client and beta
Snapshot resources.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
13438e8369 ci: Update travis functional tests
Updated travis functional tests to
install snapshot-controller for E2E

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
c533d0125a script: Add script to install snapshot-controller
Added script and snapshot-controller PSP
file to install snapshot-controller

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
1a3b07994e doc: Add Upgrade documentation for snapshot
Added step to identify alpha snapshot CRD.
Added step to delete alpha CRD and link
for installing beta CRD.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Madhu Rajanna
a116764f4d Snapshot: Template changes for snapshot beta
Updated the deployment templates and the sidecar
images version to support snapshot beta version.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
Yug Gupta
2cdf5c3b9f util: make util.ClusterInfo usable outside util package
functions like getClusterInfo() returns struct
instead of a set of strings.

Fix: #998

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-14 19:43:21 +00:00
Niels de Vos
63c458bd63 rbd: use rbdVolume.open() for metadata operations
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 18:20:15 +00:00
Niels de Vos
7a18e68a6e rbd: add rbdVolume.open() to get access to an image
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 18:20:15 +00:00
Madhu Rajanna
3482cb7091 ci: updated shell script to run individual test
Earlier we were running all the linter for non-go
files in one short, this wont be helpful for the
users who want to run particular tests.

now the Makefile as different target to
run separate lint test for different type
of non-go files.

Fixes: #979

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-14 16:54:19 +00:00
Niels de Vos
f889dc20bd cleanup: use switch/case for genVolFromVolID() error checking
By using switch/case it is easier to follow the error checking of the
genVolFromVolID() function. In case a new error is added as a return of
the function, it will be simpler to add checking for it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 14:43:05 +00:00
Niels de Vos
25ea25368a doc: add list of valid components to the development guide
The commitlint CI job uses the configuration from .commitlintrc.yaml
which contains the different components that Ceph-CSI uses. A short
description of each component has been added, so that contributors
understand what component to mention in the prefix of the subject in
commit messages.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 08:45:26 +00:00
Humble Chirammal
3be9d99573 bot: Disable semantic bot from the repo
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-14 07:04:44 +00:00
Humble Chirammal
d362dffd62 doc: add v2.1.1 release to the release matrix
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-13 09:20:37 +00:00
John Mulligan
f091374ba9 journal: remove SetNamespace setter function
The SetNamespace setter function was called only once, immediately after
the creation of a volume journal object in cephfs only.
Remove this function so that it is no longer implied that this field can
be mutated after the journal is created. In it's place, use an extended
"constructor" NewCSIVolumeJournalWithNamespace that takes a namespace
value at create-time only.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 17:57:36 +00:00
Yug Gupta
9394ac887e rbd: change image pull policy
unlike other containers, image pull policy of
csi-snapshotter was set to "Always", which can
be changed to pull only if not present.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-12 13:44:52 +00:00
Niels de Vos
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
John Mulligan
78a6de2bd0 journal: remove SetCSIDirectorySuffix to simplify journal creation
The function SetCSIDirectorySuffix was used only one per (long-lived,
gloabl) journal object. It is simpler to construct the journal objects
with this needed parameter:
1. As it is required to function and non-optional AFAICT
2. Removes the temptation to mutate global object
3. Reduces LOC with exact same functionality
4. SetCSIDirectorySuffix would not behave correctly if called a 2nd time
   anyway.

Point 4. means that if you called the function twice to change the
suffix when you previously had "csi.volumes.alice", you'd get
"csi.volumes.alice.bob" instead of "csi.volumes.bob" what one would
expect.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 08:19:09 +00:00
Madhu Rajanna
b065726f19 cephfs: use new 'ceph fs resize' command when available
Use ceph fs resize command when its available
in the cluster.if its not available fallback to
old style of resizing the subvolume

ceph change log:https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-8-nautilus

Fixes #1002

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-12 06:57:44 +00:00
Niels de Vos
3f0d2b3c4f ci: run e2e tests on Kubernetes 1.16.9
Kubernetes 1.16 still is maintained, so run e2e tests with that.
Currently 1.16.9 is the most current release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#116
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
Niels de Vos
23faa600c1 ci: run e2e tests with current Kubernetes 1.17.5 release
Kubernetes 1.17.0 has had several updates, 1.17.5 is the latest patch
release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#117
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
Niels de Vos
8a1e633403 ci: remove unmaintained k8s versions
Version 1.18.x is currently out, and the k8s community does not maintain
versions 1.14 and 1.15 anymore. No need to test on versions that users
should not deploy.

Fixes: #1023
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
Niels de Vos
7e303b6e97 ci: add configuration for "Semantic Pull Request" bot
This configuration enforces the format of commits that are part of a
Pull Request. The title of the PR can be checked as well, that will not
be required for now.

The format of a subject of all commit messages needs to be:

    <component>: <description>

For example:

    ci: add configuration for "Semantic Pull Request" bot

The valid components in the commit message are listed in the
.github/semantic.yaml file as 'types'.

See-also: https://github.com/zeke/semantic-pull-requests#semantic-pull-requests
See-also: https://www.conventionalcommits.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 11:32:25 +00:00
Humble Chirammal
0457fbe851 ci: add initial configuration for stale bot
The rules are defined here which could enable stale bot.

Additional Ref# https://probot.github.io/apps/stale/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-11 09:04:32 +00:00
Madhu Rajanna
d7a02b14c2 Doc: Updated github PR template
Updated github PR template about the rules
to be followed when developer is creating a PR

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
Madhu Rajanna
9ec4479234 DOC: Added document for DOC and commit message
Added document on the standard user need to follow
when writting the commit message and to include
sign-off in commit message.

source: https://probot.github.io/apps/dco/

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
Madhu Rajanna
ea33ccdb1a CI: update mergify rules
Updated mergify rules to consider
DCO and commitlint status

Fixes #1016

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 21:11:13 +05:30
Madhu Rajanna
cf4d5e5244 doc: Update github issue template
Added more details about required versions
and what logs need to be included when raising the bug

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 15:35:35 +00:00
Niels de Vos
43857a3b75 rbd: implement deleteImage() with go-ceph
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-06 14:55:47 +00:00
Madhu Rajanna
0ca07e4657 README: Update community dev stand meeting details
Updated community dev standup details in readme

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 09:14:28 +00:00
Madhu Rajanna
919f3b6d85 Doc: update development documentation
Updated golang version to 1.13.x and
also updated user to set GO111MODULE=on
and CGO_ENABLED=1 when doing development
in cephcsi

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:56:38 +00:00
Madhu Rajanna
22a86c568e rbd: default nouuid if the formattype is xfs
The problem happens when multiple PVCs with the
same UUID are attached/mounted on a node. This
can happen after creating a PVC from a snapshot,
or cloning a PVC.

make nouuid as the default mount option if
the format type is xfs to avoid mounting
issues.

updates: #966

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:07:16 +00:00
Niels de Vos
45d1404d3e rbd: split buildCreateVolumeResponse() from CreateVolume()
The gocyclo linter complains about the high complexity of the
CreateVolume() function:

> pkg/rbd/controllerserver.go:133:1: cyclomatic complexity 21 of func `(*ControllerServer).CreateVolume` is high (> 20) (gocyclo)

By splitting it up and separeting the creation of an exisint CSI Volume
object in buildCreateVolumeResponse(), the gocyclic linter does not
complain any longer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
c89c68e9ea rbd: use 'rv' as name of the rbdVolume (golint)
golint has a pretty struct stylechek, it down not allow different
variable names for methods on an object:

    pkg/rbd/rbd_util.go:970:1: receiver name rbdVol should be consistent with previous receiver name rv for rbdVolume (golint)
    func (rbdVol *rbdVolume) ensureEncryptionMetadataSet(ctx context.Context) error {
    ^
    pkg/rbd/rbd_journal.go:166:26: ST1016: methods on the same type should have the same receiver name (seen 2x "rbdVol", 3x "rv") (stylecheck)
    func (rbdVol *rbdVolume) Exists(ctx context.Context) (bool, error) {
                             ^

Rename the 'rbdVol' variable to 'rv' to make it consistent.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
f814bd72e5 rbd/go-ceph: add GetMetadata() and GetMetadata() functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
ea51b04017 rbd: setup connection in genVolFromVolID()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
12130123ac rbd: do not pass Credentials to checkVolExists()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
805f10fd71 rbd: remove unused rbdVolume.open()
rbdVolume.open() was split from commit 5dd34732e1e while moving part of
the functionality to util.ClusterConnection. It seems that .open() is
not used anywhere at the moment, so drop it until follow-up patches
require it again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
01edaf8a71 move rbdVolume connection details to utils.ClusterConnection
The shared util.ClusterConnection can be used for rbd.rbdVolume and
cephfs.volumeOptions to connect to the Ceph cluster. This will then use
the shared ConnPool, and functions for obtaining connection details will
be the same across cephfs and rbd packages.

The ClusterConnection.Creds credentials are temporarily available until
all the functions have been adapted to use go-ceph and the connection
from the ConnPool.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Mathias Merscher
0991cdf498 make CephFS SubvolumeGroup configurable
The name of the CephFS SubvolumeGroup for the CSI volumes was hardcoded to "csi". To make permission management in multi tenancy environments easier, this commit makes it possible to configure the CSI SubvolumeGroup.

related to #798 and #931
2020-05-04 05:50:06 +00:00