Commit Graph

1356 Commits

Author SHA1 Message Date
Niels de Vos
c9d5bc8a08 cleanup: use errors.As() in e2e/utils
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
a47cfbc6c8 cleanup: use errors.As() in rbdVolume.Exists() and rbd.checkSnapExists()
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
d4dad7c189 cleanup: use errors.As() in rbd.NodeUnstageVolume()
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
7126e92b4f cleanup: use errors.As() in rbd.rbdStatus()
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
f93730dee7 cleanup: use errors.As() to compare errors in cephfs/controllerserver
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
57d0989593 cleanup: use errors.As() to compare errors in util/cephcmds
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
8c4379862a rebase: remove "github.com/pkg/errors" from go.mod
There is no direct dependency on "github.com/pkg/errors" anymore, so it
can be removed from go.mod.

After running `go mod tidy`, the "github.com/pkg/errors" gets downgraded
to a version that is referenced in other depndencies. This was
unexpected, but seems needed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
e393e298d0 doc: do not mention "github.com/pkg/errors"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
92aae4834e cleanup: use standard Golang errors package
"github.com/pkg/errors" does not offer more functionlity than that we
need from the standard "errors" package. With Golang v1.13 errors can be
wrapped with `fmt.Errorf("... %w", err)`. `errors.Is()` and
`errors.As()` are available as well.

See-also: https://tip.golang.org/doc/go1.13#error_wrapping
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
8effa0cd3e util: add Unwrap() to error types
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#i-have-a-type-that-implements-error-and-holds-a-nested-error-how-should-i-adapt-it-to-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
6d6b5bab3c rbd: add Unwrap() to error types
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#i-have-a-type-that-implements-error-and-holds-a-nested-error-how-should-i-adapt-it-to-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
fad0cf7f1c cephfs: add Unwrap() to error types
See-also: https://github.com/golang/go/wiki/ErrorValueFAQ#i-have-a-type-that-implements-error-and-holds-a-nested-error-how-should-i-adapt-it-to-the-new-features
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
0bafa2be8c doc: add comments to rbd error methods
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
393569482e doc: add comments to cephfs error methods
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-03 09:12:48 +00:00
Niels de Vos
ec1a009dee rbd: allow DeleteVolume() to be too complex according to gocyclo
By fixing the golangci-lint runs, this now gets reported as a problem.
Instead of addressing the compexity of the DeleteVolume() method here,
mark it as a TODO.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-02 14:24:02 +00:00
Niels de Vos
f9f9fba26c ci: generate golangci.yml with correct CEPH_VERSION
When building against go-ceph, the most recent version of Ceph is
assumed to be available (currently Octopus). In case an older version of
the development packages is installed, building go-ceph will fail.

Golangci-lint does not accept the `-tags nautilus` parameter like other
Golang tools. Instead, the build-constraints need to be configured in a
confguration file.

This change takes care of the following:
- move the current scripts/golangci.yml to a template
- add the @@CEPH_VERSION@@ substitute
- generate the configuration file when needed

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-02 14:24:02 +00:00
Madhu Rajanna
7a87db39e6 rbd: check rbd image status only duing DeleteVolume RPC
currently, various calls to deleteImage does not
need the rbdStatus check. That is only required
when calling from DeleteVolume. This PR optimizes the
rbd image deletion by removing the status check.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-02 11:43:25 +00:00
chenxu1990
44d79e3bda deploy: Missing quotes
Missing quotes in deploy/cephfs/kubernetes/csi-provisioner-rbac.yaml

Signed-off-by: chenxu1990 <xuchen1990xx@gmail.com>
2020-07-02 09:13:44 +00:00
Madhu Rajanna
f711d15551 rbd: store imageID if image is present
Added a check to make sure that the image
ID is stored if the image is already found.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
b085577a4f rbd: add skipForceFlatten flag
added skipForceFlatten flag to skip
the image deptha and skip image flattening.
This will be very useful if the kernel is
not listed in cephcsi which supports deep
flatten fauture.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
e3a63029a3 rbd: Implement snapshot and clone from snapshot
This Adds a support for create,delete snapshot
and creating a new rbd image from the snapshot.

* Create a snapshot

* Create a temporary snapshot from the parent volume
* Clone a new image from a temporary snapshot with options
  --rbd-default-clone-format 2 --image-feature layering,deep-flatten
* Delete temporary snapshot created
* Create a new snapshot from cloned image
* Check the image chain depth, if the Softlimit is reached Add a
  task Flatten the cloned image and return success. if the depth
  is reached hard limit Add a task Flatten the cloned image and
  return snapshot status ready as false

```bash
1) rbd snap create <RBD image for src k8s volume>@<random snap name>
2) rbd clone --rbd-default-clone-format 2 --image-feature
   layering,deep-flatten <RBD image for src k8s volume>@<random snap>
   <RBD image for temporary snap image>
3) rbd snap rm <RBD image for src k8s volume>@<random snap name>
4) rbd snap rm <RBD image for temporary snap image>@<random snap name>
5) check the depth, if the depth is greater than configured hard
   limit add a task to flatten the cloned image return snapshot status
   ready as false if the depth is greater than soft limit add a task
   to flatten the image and return success
```

* Create a clone from snapshot

* Clone a new image from the snapshot with user-provided options
* Check the depth(n) of the cloned image if n>=(hard limit)
  Add task to flatten the image and return ABORT (to avoid image leak)

```bash
1) rbd clone --rbd-default-clone-format 2 --image-feature
   <k8s dst vol config> <RBD image for temporary snap image>@<random snap name>
    <RBD image for k8s dst vol>
2) check the depth, if the depth is greater than configured hard limit
    add a task to flatten the cloned image return ABORT error if the depth is
    greater than soft limit add a task to flatten the image and return success
```

* Delete snapshot or pvc

* Move the temporary cloned image to the trash
* Add task to remove the image from the trash

```bash
1) rbd trash mv <cloned image>
2) ceph rbd task trash remove <cloned image>
```

With earlier implementation to delete the image, we used to add
a task to remove the image with new changes this cannot be done
as the image may contain snapshots or linking.so we will be
doing below steps to delete an image(this will be
applicable for both normal image and cloned image)

* Move the rbd image to the trash
* Add task to remove the image from the trash

```bash
1) rbd trash mv <image>
2) ceph rbd task trash remove <image>
```

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
d95b07e39b rbd: implement rbd clone with go-ceph
moved the implementation of clone
from CLI to go-ceph

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
c30f91eead rbd: implement snapinfo with go-ceph
moved snapinfo function from CLI to go-ceph
for better performance

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
b3f1499cd4 rbd: remove unused snapshot function
we dont need to call the snapshot CLI functions
to get snapshot details. as these details are not
requried with new snapshot design.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
f08118a8e5 rbd: Implement create and delete snapshot with go-ceph
Implement create and delete snapshot with go-ceph

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
6ba0cd58b9 rbd: add code to store/retrieve the image ID
cephcsi need to store and retrieve the rbd image ID
in the omap as we need the image ID to add a
task to remove the image from the Trash.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
3dcef09676 rbd: Use rbd trash commands to delete image
as with snapshot and cloning implementation
the rbd images cannot be deleted with rbd
remove or add a task to delete the rbd image
as it might contains the snapshots and clones.
we need to make use of the rbd mv trask and
add a task to remove the image from trash
once all its clones and snapshots links
are broken and there will no longer any
dependency between parent and child images.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
9edb3b8e72 rbd: Add getImageID to get id of an image
add a new function called getImageID to fetch
the image id of an image which need to be stored
and retrived for Delete operation.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
9b518726ab rbd: add hardlimt and softlimit flag
added Hardlimit and Softlimit flags for cephcsi
arguments. When the Softlimit is reached cephcsi
will start a background task to flatten the rbd
image and return success and if the hardlimit
is reached it will start a background task
to flatten the rbd image and return ready
to use as false to make sure that the image
will not be used until it is flatten.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
3e7fa93256 rbd: Remove protect and unprotect function
As we are using v2 cloning we dont need to
do protect the snapshot before cloning and
unprotect it before deleting.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
47fb5f2299 e2e: update E2E testing for snapshot
with new implemntation when user creates a snapshot
in backend we are creating rbd image, we need to
validate the total images count in backend when
creating snapshots and clones.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
567b2ab280 util: Rename KernelVersion to GetKernelVersion
Renames KernelVersion to GetKernelVersion for
more readibility.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
12c3be9974 util: Move supported version check to util
as we need to reuse the same code for both cephfs
and rbd moving the supported version check function
to util package, for better readability renamed
the function.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Madhu Rajanna
37eb94e929 e2e: increase e2e timeout
in Travis CI the e2e tests are timing
out, 30 minutes seems less now for E2E
testing, increasing the timeout for 40
minutes.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-01 08:21:47 +00:00
Niels de Vos
834ea92185 rebase: use gosec-2.3.0
gosec-2.3.0 is the current latest version, use that so more checks are
done on the code.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-01 05:56:32 +00:00
Niels de Vos
1b89f86d51 util: handle Close() errors in pidlimit
A new version of gosec insists on handling errors returned by Close():

    [/go/src/github.com/ceph/ceph-csi/internal/util/pidlimit.go:44] - G307 (CWE-): Deferring unsafe method "*os.File" on type "Close" (Confidence: HIGH, Severity: MEDIUM)
      > defer cgroup.Close()

    [/go/src/github.com/ceph/ceph-csi/internal/util/pidlimit.go:78] - G307 (CWE-): Deferring unsafe method "*os.File" on type "Close" (Confidence: HIGH, Severity: MEDIUM)
      > defer f.Close()

    [/go/src/github.com/ceph/ceph-csi/internal/util/pidlimit.go:113] - G307 (CWE-): Deferring unsafe method "*os.File" on type "Close" (Confidence: HIGH, Severity: MEDIUM)
      > defer f.Close()

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-01 05:56:32 +00:00
Niels de Vos
30dc83c3bb util: create ceph.conf with 0600 permissions
The generated ceph.conf does not need readable by the group, there is
only one (system) user consuming the configurations file.

This addresses the following gosec warning:

    [/go/src/github.com/ceph/ceph-csi/internal/util/cephconf.go:52] - G306 (CWE-): Expect WriteFile permissions to be 0600 or less (Confidence: HIGH, Severity: MEDIUM)
      > ioutil.WriteFile(CephConfigPath, cephConfig, 0640)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-01 05:56:32 +00:00
Niels de Vos
16c4e33b09 util: log error when closing temp fail fails in GetOMapValue()
gosec-2.3.0 complains about the following:

    [/go/src/github.com/ceph/ceph-csi/internal/util/cephcmds.go:146] - G307 (CWE-): Deferring unsafe method "*os.File" on type "Close" (Confidence: HIGH, Severity: MEDIUM)
      > defer tmpFile.Close()

By logging the error from Close(), the warning is gone.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-01 05:56:32 +00:00
Madhu Rajanna
21bd26ebc4 e2e: Add E2E for ROX PVC
Added E2E testing for creation
and mounting of ROX PVC, if the
PVC accessmode is ReadOnlyMany
the application pod should not get
write access to it.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-30 17:43:41 +00:00
Madhu Rajanna
37979da5b2 e2e: Add E2E for app readonly options
If the mount option is readonly in app
pod, the pod should not get the write
access to the mounted cephfs subvolume.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-30 17:43:41 +00:00
Madhu Rajanna
5be2f62d38 doc: update readme for ROX PVC
Updated readme for support of ROX PVC

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-30 17:43:41 +00:00
Madhu Rajanna
e1fd12fb29 cephfs: do chmod on stagingpath
instead of doing chmod on the targetpath always
do it once at the stagintpath.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-30 17:43:41 +00:00
Madhu Rajanna
4d11490a2b cephfs: Add support for ROX PVC
cephcsi need to add mount the cephfs subvolume
as the readonly when the PVC type is ROX to
provide only readonly access to the users

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-06-30 17:43:41 +00:00
Niels de Vos
bb561c5da3 ci: select OS for Travis CI
Travis logs a message while parsing its configuration:

    root: missing os, using the default linux

By adding the "os: linux" option, the unneeded message is removed.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-30 16:02:01 +00:00
Niels de Vos
234ec7b1e0 ci: remove "sudo" from travis configuration
Address the following warning Travis reports:

    root: deprecated key sudo (The key `sudo` has no effect anymore.)

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-30 16:02:01 +00:00
Niels de Vos
c02413a663 build: Add -tags=nautilus to go test and go build
Build `GO_TAGS` based on the `CEPH_VERSION` from build.env. In case the
version is non-empty, pass `-tags=${CEPH_VERSION}` to any of the go
commandline and script that call go programs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-30 07:55:37 +00:00
Humble Chirammal
58bf45a13e rebase: Make use of latest go ceph library
The go-ceph version 0.4.0 is available now which got some important
library changes required for ceph csi project.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-06-30 07:55:37 +00:00
Yug Gupta
306d5b1da0 ci: add pylint as part of CI
add pylint to catch static issues of python
files in the repo.

User can now run make lint-py for pylint
check on python files.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-30 04:43:45 +00:00
Niels de Vos
6ea163af21 cleanup: remove unused GO_METALINTER_THREADS
Go Metalinter is not used anymore, we moved to Golangci.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
9b48598fca build: document CEPH_CSI_RUN_ALL_TESTS in build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00