Commit Graph

1804 Commits

Author SHA1 Message Date
2cdf5c3b9f util: make util.ClusterInfo usable outside util package
functions like getClusterInfo() returns struct
instead of a set of strings.

Fix: #998

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-14 19:43:21 +00:00
63c458bd63 rbd: use rbdVolume.open() for metadata operations
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 18:20:15 +00:00
7a18e68a6e rbd: add rbdVolume.open() to get access to an image
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 18:20:15 +00:00
3482cb7091 ci: updated shell script to run individual test
Earlier we were running all the linter for non-go
files in one short, this wont be helpful for the
users who want to run particular tests.

now the Makefile as different target to
run separate lint test for different type
of non-go files.

Fixes: #979

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-14 16:54:19 +00:00
f889dc20bd cleanup: use switch/case for genVolFromVolID() error checking
By using switch/case it is easier to follow the error checking of the
genVolFromVolID() function. In case a new error is added as a return of
the function, it will be simpler to add checking for it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 14:43:05 +00:00
25ea25368a doc: add list of valid components to the development guide
The commitlint CI job uses the configuration from .commitlintrc.yaml
which contains the different components that Ceph-CSI uses. A short
description of each component has been added, so that contributors
understand what component to mention in the prefix of the subject in
commit messages.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-14 08:45:26 +00:00
3be9d99573 bot: Disable semantic bot from the repo
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-14 07:04:44 +00:00
d362dffd62 doc: add v2.1.1 release to the release matrix
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-13 09:20:37 +00:00
f091374ba9 journal: remove SetNamespace setter function
The SetNamespace setter function was called only once, immediately after
the creation of a volume journal object in cephfs only.
Remove this function so that it is no longer implied that this field can
be mutated after the journal is created. In it's place, use an extended
"constructor" NewCSIVolumeJournalWithNamespace that takes a namespace
value at create-time only.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 17:57:36 +00:00
9394ac887e rbd: change image pull policy
unlike other containers, image pull policy of
csi-snapshotter was set to "Always", which can
be changed to pull only if not present.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-05-12 13:44:52 +00:00
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
78a6de2bd0 journal: remove SetCSIDirectorySuffix to simplify journal creation
The function SetCSIDirectorySuffix was used only one per (long-lived,
gloabl) journal object. It is simpler to construct the journal objects
with this needed parameter:
1. As it is required to function and non-optional AFAICT
2. Removes the temptation to mutate global object
3. Reduces LOC with exact same functionality
4. SetCSIDirectorySuffix would not behave correctly if called a 2nd time
   anyway.

Point 4. means that if you called the function twice to change the
suffix when you previously had "csi.volumes.alice", you'd get
"csi.volumes.alice.bob" instead of "csi.volumes.bob" what one would
expect.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-05-12 08:19:09 +00:00
b065726f19 cephfs: use new 'ceph fs resize' command when available
Use ceph fs resize command when its available
in the cluster.if its not available fallback to
old style of resizing the subvolume

ceph change log:https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-8-nautilus

Fixes #1002

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-12 06:57:44 +00:00
3f0d2b3c4f ci: run e2e tests on Kubernetes 1.16.9
Kubernetes 1.16 still is maintained, so run e2e tests with that.
Currently 1.16.9 is the most current release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#116
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
23faa600c1 ci: run e2e tests with current Kubernetes 1.17.5 release
Kubernetes 1.17.0 has had several updates, 1.17.5 is the latest patch
release.

See-also: https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#117
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
8a1e633403 ci: remove unmaintained k8s versions
Version 1.18.x is currently out, and the k8s community does not maintain
versions 1.14 and 1.15 anymore. No need to test on versions that users
should not deploy.

Fixes: #1023
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 19:30:39 +00:00
7e303b6e97 ci: add configuration for "Semantic Pull Request" bot
This configuration enforces the format of commits that are part of a
Pull Request. The title of the PR can be checked as well, that will not
be required for now.

The format of a subject of all commit messages needs to be:

    <component>: <description>

For example:

    ci: add configuration for "Semantic Pull Request" bot

The valid components in the commit message are listed in the
.github/semantic.yaml file as 'types'.

See-also: https://github.com/zeke/semantic-pull-requests#semantic-pull-requests
See-also: https://www.conventionalcommits.org
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-11 11:32:25 +00:00
0457fbe851 ci: add initial configuration for stale bot
The rules are defined here which could enable stale bot.

Additional Ref# https://probot.github.io/apps/stale/

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-05-11 09:04:32 +00:00
d7a02b14c2 Doc: Updated github PR template
Updated github PR template about the rules
to be followed when developer is creating a PR

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
9ec4479234 DOC: Added document for DOC and commit message
Added document on the standard user need to follow
when writting the commit message and to include
sign-off in commit message.

source: https://probot.github.io/apps/dco/

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 16:54:15 +00:00
ea33ccdb1a CI: update mergify rules
Updated mergify rules to consider
DCO and commitlint status

Fixes #1016

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 21:11:13 +05:30
cf4d5e5244 doc: Update github issue template
Added more details about required versions
and what logs need to be included when raising the bug

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-06 15:35:35 +00:00
43857a3b75 rbd: implement deleteImage() with go-ceph
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-06 14:55:47 +00:00
0ca07e4657 README: Update community dev stand meeting details
Updated community dev standup details in readme

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 09:14:28 +00:00
919f3b6d85 Doc: update development documentation
Updated golang version to 1.13.x and
also updated user to set GO111MODULE=on
and CGO_ENABLED=1 when doing development
in cephcsi

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:56:38 +00:00
22a86c568e rbd: default nouuid if the formattype is xfs
The problem happens when multiple PVCs with the
same UUID are attached/mounted on a node. This
can happen after creating a PVC from a snapshot,
or cloning a PVC.

make nouuid as the default mount option if
the format type is xfs to avoid mounting
issues.

updates: #966

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-05 07:07:16 +00:00
45d1404d3e rbd: split buildCreateVolumeResponse() from CreateVolume()
The gocyclo linter complains about the high complexity of the
CreateVolume() function:

> pkg/rbd/controllerserver.go:133:1: cyclomatic complexity 21 of func `(*ControllerServer).CreateVolume` is high (> 20) (gocyclo)

By splitting it up and separeting the creation of an exisint CSI Volume
object in buildCreateVolumeResponse(), the gocyclic linter does not
complain any longer.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
c89c68e9ea rbd: use 'rv' as name of the rbdVolume (golint)
golint has a pretty struct stylechek, it down not allow different
variable names for methods on an object:

    pkg/rbd/rbd_util.go:970:1: receiver name rbdVol should be consistent with previous receiver name rv for rbdVolume (golint)
    func (rbdVol *rbdVolume) ensureEncryptionMetadataSet(ctx context.Context) error {
    ^
    pkg/rbd/rbd_journal.go:166:26: ST1016: methods on the same type should have the same receiver name (seen 2x "rbdVol", 3x "rv") (stylecheck)
    func (rbdVol *rbdVolume) Exists(ctx context.Context) (bool, error) {
                             ^

Rename the 'rbdVol' variable to 'rv' to make it consistent.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
f814bd72e5 rbd/go-ceph: add GetMetadata() and GetMetadata() functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
ea51b04017 rbd: setup connection in genVolFromVolID()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
12130123ac rbd: do not pass Credentials to checkVolExists()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
805f10fd71 rbd: remove unused rbdVolume.open()
rbdVolume.open() was split from commit 5dd34732e1e while moving part of
the functionality to util.ClusterConnection. It seems that .open() is
not used anywhere at the moment, so drop it until follow-up patches
require it again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
01edaf8a71 move rbdVolume connection details to utils.ClusterConnection
The shared util.ClusterConnection can be used for rbd.rbdVolume and
cephfs.volumeOptions to connect to the Ceph cluster. This will then use
the shared ConnPool, and functions for obtaining connection details will
be the same across cephfs and rbd packages.

The ClusterConnection.Creds credentials are temporarily available until
all the functions have been adapted to use go-ceph and the connection
from the ConnPool.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-04 13:21:03 +00:00
0991cdf498 make CephFS SubvolumeGroup configurable
The name of the CephFS SubvolumeGroup for the CSI volumes was hardcoded to "csi". To make permission management in multi tenancy environments easier, this commit makes it possible to configure the CSI SubvolumeGroup.

related to #798 and #931
2020-05-04 05:50:06 +00:00
47226ccdf7 util: fix golint warnings in csiconfig, volid
golint warns about the following statements:

ceph-csi/internal/util/csiconfig.go
Line 49: warning: exported function Mons should have comment or be unexported (golint)

ceph-csi/pkg/util/volid.go :
Line 72: warning: exported method CSIIdentifier.ComposeCSIID should have comment
or be unexported (golint)

Reported-by: https://goreportcard.com/report/github.com/ceph/ceph-csi
Updates: #975

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-04-29 11:36:53 +00:00
b6ea4ecdf0 travis: run deploy as final stage
The `deploy:` section is executed as part of each successful job. That
means the container images and Helm charts are built and pushed many
times during testing after a push/merge event.

By creating a deploy stage and listing it after the tests, it is
guaranteed to run only once after all tests have succeeded. By adding a
condition to only run the stage for push events, the images and Helm
charts should only get built+pushed after a PR has been merged.

See-also: https://docs.travis-ci.com/user/conditional-builds-stages-jobs#Specifying-conditions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 15:05:15 +00:00
c8df7aa475 create temp directory to push helm charts
Instead of creating the tmp directory in
ceph-csi create temp directory outside
and use it to push helm charts

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-28 11:45:56 +00:00
d1e7d1eab3 Skip saving helm to temp file
install helm directly from the script
instead of storing it in a temp file
and then executing it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-28 11:45:56 +00:00
99dabca319 add build step logging to multi-arch build
With extra logging, there is no need to call `travis_wait` anymore. In
addition the `travis_wait` command blocks output, so the build steps are
not reported until the script finishes.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
5a4517e38e move build-multi-arch-image.sh to scripts/
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
9fd4005819 travis: remove 'travis_wait' from deploy section
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
20801d6563 deploy: log steps while tasks are running
By repeated logging of the current step, Travis CI should receive some
terminal activity and is expected to not timeout anymore.

Fixes: #982
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
fd62a5897a tests: allow external-sources for shellcheck
While introducing scripts/build_step.inc.sh the tests start to fail as
the script is included and each shell script is tested separately. By
adding the option --external-sources to shellcheck, the related warnings
are not reported.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
52fa4f10da scripts: add build_step logging functions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-28 09:13:55 +00:00
1f7d9760d5 Makefile: allow building with Podman
If Docker does not exist, use Podman.
Use all CPUs for the build, which hopefully will make it faster.

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
Fixes: #987
2020-04-27 05:21:26 +00:00
d1e12cb0e2 add optional TARGET=.. parameter for 'make containerized-test'
Running the containerized-tests is relatively time consuming as it runs
all the tests. By adding a TARGET=.. option to the Makefile, it is now
possible to run a selected test, for example:

    make containerized-test TARGET=static-check

This reduces the time the tests take, so it is more friendly to the
developers while addressing problems.

When no TARGET=.. parameter is passed, the default tests are run.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-24 15:41:46 +00:00
7d642b791c e2e: fix IneffAssign warnings in checkPVSelectorValuesForPVC()
IneffAssign warns about the two following statements:

  Line 1342: warning: ineffectual assignment to rFound (ineffassign)
  Line 1350: warning: ineffectual assignment to zFound (ineffassign)

rFound and zFound should be set before entering the loop, otherwise the
initial value will overwrite the updated value on each iteration.

Reported-by: https://goreportcard.com/report/github.com/ceph/ceph-csi
Updates: #975
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-24 08:35:36 +00:00
c8271fe64c journal: move voljournal.go to a new package
This new journal package isolates journal logic from the rest of util
and helps draw bright lines between what is a generic utility function
and what is csi journal logic.

Done partly as preparation for making use of go-ceph in journal.

No functional changes are made except to update references to allow the
code to compile.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-04-24 07:36:38 +00:00
22d1476bba util: create a NewErrSnapNameConflict function
The NewErrSnapNameConflict will allow packages outside of "util" to
create new instances of the ErrSnapNameConflict error.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2020-04-24 07:36:38 +00:00
3b608f130a Add grpc 1.27 version require section in go.mod to avoid CI issue
This PR addes GRPC 1.27 to required section in go.mod
file. this is overridden by 1.26 in replace section

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-23 16:02:55 +00:00