Commit Graph

3245 Commits

Author SHA1 Message Date
Yug Gupta
9394ac887e rbd: change image pull policy
unlike other containers, image pull policy of
csi-snapshotter was set to "Always", which can
be changed to pull only if not present.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-12 13:44:52 +00:00
Niels de Vos
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
John Mulligan
78a6de2bd0 journal: remove SetCSIDirectorySuffix to simplify journal creation
The function SetCSIDirectorySuffix was used only one per (long-lived,
gloabl) journal object. It is simpler to construct the journal objects
with this needed parameter:
1. As it is required to function and non-optional AFAICT
2. Removes the temptation to mutate global object
3. Reduces LOC with exact same functionality
4. SetCSIDirectorySuffix would not behave correctly if called a 2nd time
   anyway.

Point 4. means that if you called the function twice to change the
suffix when you previously had "csi.volumes.alice", you'd get
"csi.volumes.alice.bob" instead of "csi.volumes.bob" what one would
expect.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 08:19:09 +00:00
Madhu Rajanna
b065726f19 cephfs: use new 'ceph fs resize' command when available
Use ceph fs resize command when its available
in the cluster.if its not available fallback to
old style of resizing the subvolume

ceph change log:https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-8-nautilus

Fixes #1002

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-12 06:57:44 +00:00
Niels de Vos
3f0d2b3c4f ci: run e2e tests on Kubernetes 1.16.9
Kubernetes 1.16 still is maintained, so run e2e tests with that.
Currently 1.16.9 is the most current release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#116
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
Niels de Vos
23faa600c1 ci: run e2e tests with current Kubernetes 1.17.5 release
Kubernetes 1.17.0 has had several updates, 1.17.5 is the latest patch
release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#117
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
Niels de Vos
8a1e633403 ci: remove unmaintained k8s versions
Version 1.18.x is currently out, and the k8s community does not maintain
versions 1.14 and 1.15 anymore. No need to test on versions that users
should not deploy.

Fixes: #1023
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
Niels de Vos
7e303b6e97 ci: add configuration for "Semantic Pull Request" bot
This configuration enforces the format of commits that are part of a
Pull Request. The title of the PR can be checked as well, that will not
be required for now.

The format of a subject of all commit messages needs to be:

    <component>: <description>

For example:

    ci: add configuration for "Semantic Pull Request" bot

The valid components in the commit message are listed in the
.github/semantic.yaml file as 'types'.

See-also: https://github.com/zeke/semantic-pull-requests#semantic-pull-requests
See-also: https://www.conventionalcommits.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 11:32:25 +00:00
Humble Chirammal
0457fbe851 ci: add initial configuration for stale bot
The rules are defined here which could enable stale bot.

Additional Ref# https://probot.github.io/apps/stale/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-11 09:04:32 +00:00
Madhu Rajanna
d7a02b14c2 Doc: Updated github PR template
Updated github PR template about the rules
to be followed when developer is creating a PR

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
Madhu Rajanna
9ec4479234 DOC: Added document for DOC and commit message
Added document on the standard user need to follow
when writting the commit message and to include
sign-off in commit message.

source: https://probot.github.io/apps/dco/

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
Madhu Rajanna
ea33ccdb1a CI: update mergify rules
Updated mergify rules to consider
DCO and commitlint status

Fixes #1016

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 21:11:13 +05:30
Madhu Rajanna
cf4d5e5244 doc: Update github issue template
Added more details about required versions
and what logs need to be included when raising the bug

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 15:35:35 +00:00
Niels de Vos
43857a3b75 rbd: implement deleteImage() with go-ceph
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-06 14:55:47 +00:00
Madhu Rajanna
0ca07e4657 README: Update community dev stand meeting details
Updated community dev standup details in readme

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 09:14:28 +00:00
Madhu Rajanna
919f3b6d85 Doc: update development documentation
Updated golang version to 1.13.x and
also updated user to set GO111MODULE=on
and CGO_ENABLED=1 when doing development
in cephcsi

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:56:38 +00:00
Madhu Rajanna
22a86c568e rbd: default nouuid if the formattype is xfs
The problem happens when multiple PVCs with the
same UUID are attached/mounted on a node. This
can happen after creating a PVC from a snapshot,
or cloning a PVC.

make nouuid as the default mount option if
the format type is xfs to avoid mounting
issues.

updates: #966

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:07:16 +00:00
Niels de Vos
45d1404d3e rbd: split buildCreateVolumeResponse() from CreateVolume()
The gocyclo linter complains about the high complexity of the
CreateVolume() function:

> pkg/rbd/controllerserver.go:133:1: cyclomatic complexity 21 of func `(*ControllerServer).CreateVolume` is high (> 20) (gocyclo)

By splitting it up and separeting the creation of an exisint CSI Volume
object in buildCreateVolumeResponse(), the gocyclic linter does not
complain any longer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
c89c68e9ea rbd: use 'rv' as name of the rbdVolume (golint)
golint has a pretty struct stylechek, it down not allow different
variable names for methods on an object:

    pkg/rbd/rbd_util.go:970:1: receiver name rbdVol should be consistent with previous receiver name rv for rbdVolume (golint)
    func (rbdVol *rbdVolume) ensureEncryptionMetadataSet(ctx context.Context) error {
    ^
    pkg/rbd/rbd_journal.go:166:26: ST1016: methods on the same type should have the same receiver name (seen 2x "rbdVol", 3x "rv") (stylecheck)
    func (rbdVol *rbdVolume) Exists(ctx context.Context) (bool, error) {
                             ^

Rename the 'rbdVol' variable to 'rv' to make it consistent.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
f814bd72e5 rbd/go-ceph: add GetMetadata() and GetMetadata() functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
ea51b04017 rbd: setup connection in genVolFromVolID()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
12130123ac rbd: do not pass Credentials to checkVolExists()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
805f10fd71 rbd: remove unused rbdVolume.open()
rbdVolume.open() was split from commit 5dd34732e1e while moving part of
the functionality to util.ClusterConnection. It seems that .open() is
not used anywhere at the moment, so drop it until follow-up patches
require it again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Niels de Vos
01edaf8a71 move rbdVolume connection details to utils.ClusterConnection
The shared util.ClusterConnection can be used for rbd.rbdVolume and
cephfs.volumeOptions to connect to the Ceph cluster. This will then use
the shared ConnPool, and functions for obtaining connection details will
be the same across cephfs and rbd packages.

The ClusterConnection.Creds credentials are temporarily available until
all the functions have been adapted to use go-ceph and the connection
from the ConnPool.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
Mathias Merscher
0991cdf498 make CephFS SubvolumeGroup configurable
The name of the CephFS SubvolumeGroup for the CSI volumes was hardcoded to "csi". To make permission management in multi tenancy environments easier, this commit makes it possible to configure the CSI SubvolumeGroup.

related to #798 and #931
2020-05-04 05:50:06 +00:00
Yug Gupta
47226ccdf7 util: fix golint warnings in csiconfig, volid
golint warns about the following statements:

ceph-csi/internal/util/csiconfig.go
Line 49: warning: exported function Mons should have comment or be unexported (golint)

ceph-csi/pkg/util/volid.go :
Line 72: warning: exported method CSIIdentifier.ComposeCSIID should have comment
or be unexported (golint)

Reported-by: https://goreportcard.com/report/github.com/ceph/ceph-csi
Updates: #975

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-04-29 11:36:53 +00:00
Niels de Vos
b6ea4ecdf0 travis: run deploy as final stage
The `deploy:` section is executed as part of each successful job. That
means the container images and Helm charts are built and pushed many
times during testing after a push/merge event.

By creating a deploy stage and listing it after the tests, it is
guaranteed to run only once after all tests have succeeded. By adding a
condition to only run the stage for push events, the images and Helm
charts should only get built+pushed after a PR has been merged.

See-also: https://docs.travis-ci.com/user/conditional-builds-stages-jobs#Specifying-conditions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 15:05:15 +00:00
Madhu Rajanna
c8df7aa475 create temp directory to push helm charts
Instead of creating the tmp directory in
ceph-csi create temp directory outside
and use it to push helm charts

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-28 11:45:56 +00:00
Madhu Rajanna
d1e7d1eab3 Skip saving helm to temp file
install helm directly from the script
instead of storing it in a temp file
and then executing it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-28 11:45:56 +00:00
Niels de Vos
99dabca319 add build step logging to multi-arch build
With extra logging, there is no need to call `travis_wait` anymore. In
addition the `travis_wait` command blocks output, so the build steps are
not reported until the script finishes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
5a4517e38e move build-multi-arch-image.sh to scripts/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
9fd4005819 travis: remove 'travis_wait' from deploy section
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
20801d6563 deploy: log steps while tasks are running
By repeated logging of the current step, Travis CI should receive some
terminal activity and is expected to not timeout anymore.

Fixes: #982
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
fd62a5897a tests: allow external-sources for shellcheck
While introducing scripts/build_step.inc.sh the tests start to fail as
the script is included and each shell script is tested separately. By
adding the option --external-sources to shellcheck, the related warnings
are not reported.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Niels de Vos
52fa4f10da scripts: add build_step logging functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
Yaniv Kaul
1f7d9760d5 Makefile: allow building with Podman
If Docker does not exist, use Podman.
Use all CPUs for the build, which hopefully will make it faster.

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
Fixes: #987
2020-04-27 05:21:26 +00:00
Niels de Vos
d1e12cb0e2 add optional TARGET=.. parameter for 'make containerized-test'
Running the containerized-tests is relatively time consuming as it runs
all the tests. By adding a TARGET=.. option to the Makefile, it is now
possible to run a selected test, for example:

    make containerized-test TARGET=static-check

This reduces the time the tests take, so it is more friendly to the
developers while addressing problems.

When no TARGET=.. parameter is passed, the default tests are run.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-24 15:41:46 +00:00
Niels de Vos
7d642b791c e2e: fix IneffAssign warnings in checkPVSelectorValuesForPVC()
IneffAssign warns about the two following statements:

  Line 1342: warning: ineffectual assignment to rFound (ineffassign)
  Line 1350: warning: ineffectual assignment to zFound (ineffassign)

rFound and zFound should be set before entering the loop, otherwise the
initial value will overwrite the updated value on each iteration.

Reported-by: https://goreportcard.com/report/github.com/ceph/ceph-csi
Updates: #975
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-24 08:35:36 +00:00
John Mulligan
c8271fe64c journal: move voljournal.go to a new package
This new journal package isolates journal logic from the rest of util
and helps draw bright lines between what is a generic utility function
and what is csi journal logic.

Done partly as preparation for making use of go-ceph in journal.

No functional changes are made except to update references to allow the
code to compile.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-04-24 07:36:38 +00:00
John Mulligan
22d1476bba util: create a NewErrSnapNameConflict function
The NewErrSnapNameConflict will allow packages outside of "util" to
create new instances of the ErrSnapNameConflict error.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-04-24 07:36:38 +00:00
Madhu Rajanna
3b608f130a Add grpc 1.27 version require section in go.mod to avoid CI issue
This PR addes GRPC 1.27 to required section in go.mod
file. this is overridden by 1.26 in replace section

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 16:02:55 +00:00
Niels de Vos
32839948ef cleanup: move pkg/ to internal/
The internal/ directory in Go has a special meaning, and indicates that
those packages are not meant for external consumption. Ceph-CSI does
provide public APIs for other projects to consume. There is no plan to
keep the API of the internally used packages stable.

Closes: #903
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 11:00:59 +00:00
Madhu Rajanna
d0abc3f5e6 Add travis_wait to deploy section
the deploy section is getting timeout to build and push
the images. see https://travis-ci.org/github/ceph/ceph-csi/jobs/678228818#L1617-L1622

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 09:09:28 +00:00
Madhu Rajanna
6b3e6e67e4 update the helm documentation for cephfs and rbd
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 04:43:04 +00:00
Madhu Rajanna
10eec1f39c Create Namespace for E2E
create ns before the helm create to avoid
`Error: create: failed to create: namespaces
"xxx" not found` issue

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 04:43:04 +00:00
Madhu Rajanna
86500c06c3 Helm3: update scripts to install helm3
Updated scripts to install helm3

fixes #920

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 04:43:04 +00:00
Madhu Rajanna
f696567517 Update Travis CI to install helm3
Updated CI scripts to install helm3

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 04:43:04 +00:00
Niels de Vos
2c6c0decbe containerized-build: set WORKDIR and use in make target
By setting the WORKDIR in the container image, there is no need to pass
it on the commandline in the Makefile. This makes the line for the make
target a little cleaner.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00
Niels de Vos
ceef252a24 containerized-build: use GOROOT variable everywhere
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00
Niels de Vos
f942e2175a containerized-build: use dnf instead of yum
The Ceph base image moved to CentOS 8 which uses dnf. Use that instead
of yum.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00