Commit Graph

2312 Commits

Author SHA1 Message Date
1f13692000 ci: skip snapshot E2E if kube<1.17+
snapshot beta CRD wont work if the
kubernetes version is less than 1.17.0
as the snapshot CRD wont be installed
we cannot test the snapshot,so disabling
it if the kube version is less than 1.17

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
538b854853 script: Delete snapshot CRD created by cephcsi in rook
when we deploy rook+cephcsi for E2E, the
external-snapshotter in cephcsi deployed by
rook will create the CRD, we need to delete
the crd created by external-snapshotter.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
4606b5042f vendor: Vendor changes to add external-snapshotter
Few other depedencies got updated which might
be required for snapshot.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
c5bf95b625 e2e: Enable E2E for rbd snapshot
Updated E2E to use beta client and beta
Snapshot resources.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
13438e8369 ci: Update travis functional tests
Updated travis functional tests to
install snapshot-controller for E2E

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
c533d0125a script: Add script to install snapshot-controller
Added script and snapshot-controller PSP
file to install snapshot-controller

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
1a3b07994e doc: Add Upgrade documentation for snapshot
Added step to identify alpha snapshot CRD.
Added step to delete alpha CRD and link
for installing beta CRD.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
a116764f4d Snapshot: Template changes for snapshot beta
Updated the deployment templates and the sidecar
images version to support snapshot beta version.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-15 08:19:32 +00:00
2cdf5c3b9f util: make util.ClusterInfo usable outside util package
functions like getClusterInfo() returns struct
instead of a set of strings.

Fix: #998

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-14 19:43:21 +00:00
63c458bd63 rbd: use rbdVolume.open() for metadata operations
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 18:20:15 +00:00
7a18e68a6e rbd: add rbdVolume.open() to get access to an image
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 18:20:15 +00:00
3482cb7091 ci: updated shell script to run individual test
Earlier we were running all the linter for non-go
files in one short, this wont be helpful for the
users who want to run particular tests.

now the Makefile as different target to
run separate lint test for different type
of non-go files.

Fixes: #979

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-14 16:54:19 +00:00
f889dc20bd cleanup: use switch/case for genVolFromVolID() error checking
By using switch/case it is easier to follow the error checking of the
genVolFromVolID() function. In case a new error is added as a return of
the function, it will be simpler to add checking for it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 14:43:05 +00:00
25ea25368a doc: add list of valid components to the development guide
The commitlint CI job uses the configuration from .commitlintrc.yaml
which contains the different components that Ceph-CSI uses. A short
description of each component has been added, so that contributors
understand what component to mention in the prefix of the subject in
commit messages.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 08:45:26 +00:00
3be9d99573 bot: Disable semantic bot from the repo
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-14 07:04:44 +00:00
d362dffd62 doc: add v2.1.1 release to the release matrix
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-13 09:20:37 +00:00
f091374ba9 journal: remove SetNamespace setter function
The SetNamespace setter function was called only once, immediately after
the creation of a volume journal object in cephfs only.
Remove this function so that it is no longer implied that this field can
be mutated after the journal is created. In it's place, use an extended
"constructor" NewCSIVolumeJournalWithNamespace that takes a namespace
value at create-time only.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 17:57:36 +00:00
9394ac887e rbd: change image pull policy
unlike other containers, image pull policy of
csi-snapshotter was set to "Always", which can
be changed to pull only if not present.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-12 13:44:52 +00:00
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
78a6de2bd0 journal: remove SetCSIDirectorySuffix to simplify journal creation
The function SetCSIDirectorySuffix was used only one per (long-lived,
gloabl) journal object. It is simpler to construct the journal objects
with this needed parameter:
1. As it is required to function and non-optional AFAICT
2. Removes the temptation to mutate global object
3. Reduces LOC with exact same functionality
4. SetCSIDirectorySuffix would not behave correctly if called a 2nd time
   anyway.

Point 4. means that if you called the function twice to change the
suffix when you previously had "csi.volumes.alice", you'd get
"csi.volumes.alice.bob" instead of "csi.volumes.bob" what one would
expect.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 08:19:09 +00:00
b065726f19 cephfs: use new 'ceph fs resize' command when available
Use ceph fs resize command when its available
in the cluster.if its not available fallback to
old style of resizing the subvolume

ceph change log:https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-8-nautilus

Fixes #1002

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-12 06:57:44 +00:00
3f0d2b3c4f ci: run e2e tests on Kubernetes 1.16.9
Kubernetes 1.16 still is maintained, so run e2e tests with that.
Currently 1.16.9 is the most current release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#116
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
23faa600c1 ci: run e2e tests with current Kubernetes 1.17.5 release
Kubernetes 1.17.0 has had several updates, 1.17.5 is the latest patch
release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#117
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
8a1e633403 ci: remove unmaintained k8s versions
Version 1.18.x is currently out, and the k8s community does not maintain
versions 1.14 and 1.15 anymore. No need to test on versions that users
should not deploy.

Fixes: #1023
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
7e303b6e97 ci: add configuration for "Semantic Pull Request" bot
This configuration enforces the format of commits that are part of a
Pull Request. The title of the PR can be checked as well, that will not
be required for now.

The format of a subject of all commit messages needs to be:

    <component>: <description>

For example:

    ci: add configuration for "Semantic Pull Request" bot

The valid components in the commit message are listed in the
.github/semantic.yaml file as 'types'.

See-also: https://github.com/zeke/semantic-pull-requests#semantic-pull-requests
See-also: https://www.conventionalcommits.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 11:32:25 +00:00
0457fbe851 ci: add initial configuration for stale bot
The rules are defined here which could enable stale bot.

Additional Ref# https://probot.github.io/apps/stale/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-11 09:04:32 +00:00
d7a02b14c2 Doc: Updated github PR template
Updated github PR template about the rules
to be followed when developer is creating a PR

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
9ec4479234 DOC: Added document for DOC and commit message
Added document on the standard user need to follow
when writting the commit message and to include
sign-off in commit message.

source: https://probot.github.io/apps/dco/

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
ea33ccdb1a CI: update mergify rules
Updated mergify rules to consider
DCO and commitlint status

Fixes #1016

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 21:11:13 +05:30
cf4d5e5244 doc: Update github issue template
Added more details about required versions
and what logs need to be included when raising the bug

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 15:35:35 +00:00
43857a3b75 rbd: implement deleteImage() with go-ceph
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-06 14:55:47 +00:00
0ca07e4657 README: Update community dev stand meeting details
Updated community dev standup details in readme

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 09:14:28 +00:00
919f3b6d85 Doc: update development documentation
Updated golang version to 1.13.x and
also updated user to set GO111MODULE=on
and CGO_ENABLED=1 when doing development
in cephcsi

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:56:38 +00:00
22a86c568e rbd: default nouuid if the formattype is xfs
The problem happens when multiple PVCs with the
same UUID are attached/mounted on a node. This
can happen after creating a PVC from a snapshot,
or cloning a PVC.

make nouuid as the default mount option if
the format type is xfs to avoid mounting
issues.

updates: #966

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:07:16 +00:00
45d1404d3e rbd: split buildCreateVolumeResponse() from CreateVolume()
The gocyclo linter complains about the high complexity of the
CreateVolume() function:

> pkg/rbd/controllerserver.go:133:1: cyclomatic complexity 21 of func `(*ControllerServer).CreateVolume` is high (> 20) (gocyclo)

By splitting it up and separeting the creation of an exisint CSI Volume
object in buildCreateVolumeResponse(), the gocyclic linter does not
complain any longer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
c89c68e9ea rbd: use 'rv' as name of the rbdVolume (golint)
golint has a pretty struct stylechek, it down not allow different
variable names for methods on an object:

    pkg/rbd/rbd_util.go:970:1: receiver name rbdVol should be consistent with previous receiver name rv for rbdVolume (golint)
    func (rbdVol *rbdVolume) ensureEncryptionMetadataSet(ctx context.Context) error {
    ^
    pkg/rbd/rbd_journal.go:166:26: ST1016: methods on the same type should have the same receiver name (seen 2x "rbdVol", 3x "rv") (stylecheck)
    func (rbdVol *rbdVolume) Exists(ctx context.Context) (bool, error) {
                             ^

Rename the 'rbdVol' variable to 'rv' to make it consistent.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
f814bd72e5 rbd/go-ceph: add GetMetadata() and GetMetadata() functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
ea51b04017 rbd: setup connection in genVolFromVolID()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
12130123ac rbd: do not pass Credentials to checkVolExists()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
805f10fd71 rbd: remove unused rbdVolume.open()
rbdVolume.open() was split from commit 5dd34732e1e while moving part of
the functionality to util.ClusterConnection. It seems that .open() is
not used anywhere at the moment, so drop it until follow-up patches
require it again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
01edaf8a71 move rbdVolume connection details to utils.ClusterConnection
The shared util.ClusterConnection can be used for rbd.rbdVolume and
cephfs.volumeOptions to connect to the Ceph cluster. This will then use
the shared ConnPool, and functions for obtaining connection details will
be the same across cephfs and rbd packages.

The ClusterConnection.Creds credentials are temporarily available until
all the functions have been adapted to use go-ceph and the connection
from the ConnPool.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
0991cdf498 make CephFS SubvolumeGroup configurable
The name of the CephFS SubvolumeGroup for the CSI volumes was hardcoded to "csi". To make permission management in multi tenancy environments easier, this commit makes it possible to configure the CSI SubvolumeGroup.

related to #798 and #931
2020-05-04 05:50:06 +00:00
47226ccdf7 util: fix golint warnings in csiconfig, volid
golint warns about the following statements:

ceph-csi/internal/util/csiconfig.go
Line 49: warning: exported function Mons should have comment or be unexported (golint)

ceph-csi/pkg/util/volid.go :
Line 72: warning: exported method CSIIdentifier.ComposeCSIID should have comment
or be unexported (golint)

Reported-by: https://goreportcard.com/report/github.com/ceph/ceph-csi
Updates: #975

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-04-29 11:36:53 +00:00
b6ea4ecdf0 travis: run deploy as final stage
The `deploy:` section is executed as part of each successful job. That
means the container images and Helm charts are built and pushed many
times during testing after a push/merge event.

By creating a deploy stage and listing it after the tests, it is
guaranteed to run only once after all tests have succeeded. By adding a
condition to only run the stage for push events, the images and Helm
charts should only get built+pushed after a PR has been merged.

See-also: https://docs.travis-ci.com/user/conditional-builds-stages-jobs#Specifying-conditions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 15:05:15 +00:00
c8df7aa475 create temp directory to push helm charts
Instead of creating the tmp directory in
ceph-csi create temp directory outside
and use it to push helm charts

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-28 11:45:56 +00:00
d1e7d1eab3 Skip saving helm to temp file
install helm directly from the script
instead of storing it in a temp file
and then executing it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-28 11:45:56 +00:00
99dabca319 add build step logging to multi-arch build
With extra logging, there is no need to call `travis_wait` anymore. In
addition the `travis_wait` command blocks output, so the build steps are
not reported until the script finishes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
5a4517e38e move build-multi-arch-image.sh to scripts/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
9fd4005819 travis: remove 'travis_wait' from deploy section
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
20801d6563 deploy: log steps while tasks are running
By repeated logging of the current step, Travis CI should receive some
terminal activity and is expected to not timeout anymore.

Fixes: #982
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00