Commit Graph

2838 Commits

Author SHA1 Message Date
Madhu Rajanna
2232067e99 cephfs: use rados.ErrNotFound in subvolume info
go-ceph rados.ErrNotFound for not found errors,cephcsi
need to check same for not found errors.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-01 09:46:52 +00:00
Madhu Rajanna
a0283ef7f9 cephfs: use rados.ErrNotFound in getVolumeRootPathCeph
go-ceph rados.ErrNotFound for not found errors,cephcsi
need to check same for not found errors.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-01 09:46:52 +00:00
Madhu Rajanna
b40d561228 cephfs: make getVolumeRootPathCeph as a method
getVolumeRootPathCeph is a method of volumeOptions

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-01 09:46:52 +00:00
Madhu Rajanna
e4713bc9ed cephfs: remove unused Credentials argument from getVolumeRootPathCeph
remove unused Credentials argument from getVolumeRootPathCeph
function.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-01 09:46:52 +00:00
Madhu Rajanna
b6f3ba885e cephfs: implement getVolumeRootPathCeph with go-ceph
instead of ceph fs CLI commands using go-ceph library
to get subvolume path.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-01 09:46:52 +00:00
Niels de Vos
5565a902ac rebase: split vaultConnection from VaultKMS
In order to re-use the configuration of Vault, split a new
vaultConnection type from the VaultKMS type.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-30 19:16:39 +00:00
Madhu Rajanna
b29af806f7 ci: remove multi stage build from travis ci
removing multi stage build from travis CI
as we are moving to github action.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-30 16:27:52 +01:00
Madhu Rajanna
40ae60cbe4 ci: add github action for multiarch build
added a github action for multiarch docker
build.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-30 16:27:52 +01:00
Niels de Vos
1845f2b77d e2e: use full-qualified-image-name for vault-init-job
On occasion deploying Vault fails. It seems the vault-init-job batch job
does not use a full-qualified-image-name for the "vault" container.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-30 12:27:00 +00:00
Niels de Vos
bdf8fe7f93 build: use docker.io/ceph/ceph:v15 as BASE_IMAGE
Use a qualified image name, including the registry where it should come
from. This makes it possible to pull the image from the right location,
and consume it in CI jobs without trying to pull again.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-30 10:30:20 +00:00
Madhu Rajanna
591d097320 ci: update mergify rule for codespell
added codespell as required to auto merge PR.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-30 08:46:48 +01:00
Madhu Rajanna
8ebb9a1ba0 cleanup: fix misspell words
fixed misspell words detected by  codespell

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-30 08:46:48 +01:00
Madhu Rajanna
28793efc90 doc: design document for rbd mirroring
This document outlines the internal cephcsi
design to handle mirrored RBD images.

Co-authored-by: ShyamsundarR <srangana@redhat.com>
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-29 07:49:11 +00:00
Madhu Rajanna
39b1f2b4d3 cleanup: fix mispell words
fixed mispell words in the repo.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-29 12:47:46 +05:30
Madhu Rajanna
eee49a6e0a ci: add codespell for spell check
adding a github action to check the spelling
mistakes in the code.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-29 12:47:46 +05:30
Niels de Vos
8d08e8b37e cephfs: remove unused Credentials argument from getCloneState()
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 07:13:35 +00:00
Niels de Vos
055975f790 cephfs: implement getCloneState() with go-ceph
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 07:13:35 +00:00
Niels de Vos
476ad18d2e cephfs: make getCloneState() a method of volumeOptions
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 07:13:35 +00:00
Niels de Vos
66aa595e77 cephfs: do not export internal CloneStatus type
getCloneInfo() does not need to return a full CloneStatus struct that
only has one member. Instead, it can just return the value of the single
member, so the JSON type/struct does not need to be exposed.

This makes the API for getCloneInfo() a little simpler, so it can be
replaced by a go-ceph implementation later on.

As the function does not return any of the unused attributes anymore, it
is renamed to getCloneStatu() as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 07:13:35 +00:00
Niels de Vos
91774fc936 rebase: vendor dependencies for Vault API
Uses github.com/libopenstorage/secrets to communicate with Vault. This
removes the need for maintaining our own limited Vault APIs.

By adding the new dependency, several other packages got updated in the
process. Unused indirect dependencies have been removed from go.mod.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
7824cb5ed7 util: use %w when forwarding an error
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
eb63e796a8 e2e: deleted secrets from Vault are available, but empty
Once the Vault API removed a secret, the contents will have been wiped.
The key is still available, until it gets destroyed. This causes the e2e
test to detect an empty secret, and assume that it has not been deleted
yet.

By requesting the `data` field from the secret, an error is thrown in
case the secret has been wiped. This makes it possible for the e2e test
to detect that the secret has been removed and scheduled for destroying.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
5fba89f783 cleanup: use libopenstorage/secrets for Vault access
Instead of the hand-rolled Vault usage, use the libopenstorage/secrets
package that provides a nice API. The support for Vault becomes much
simpler and maintainable that way.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
db6d376434 deploy: add sys/mounts to Vault policy
Add "sys/mounts" so that VaultBackendKey does not need to be set. The
libopenstorage API detects the version for the key-value store in Vault
by reading "sys/mounts". Without permissions to read this endpoint, the
VaultBackendKey is required to be configured.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Niels de Vos
04586dc733 deploy: add "list" operation to Vault policy
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-29 04:03:59 +00:00
Madhu Rajanna
fc9b2e5ac5 deploy: update deployment template for new controller
updated deployment template for the new controller and
also added `update` configmap RBAC for the controller
as the controller uses the configmap for the leader
election.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
6091490393 rbd: improve logging in getCloneDepth
earlier if the depth check fails the
complete vol struct was getting logged,
this commits logs only the pool and image
name.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
b3120926b9 rbd: remove extra Destory of parent volume
removed extra Destory of the parent volume.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
4fcd649fa7 e2e: add testing for controller
This commits adds an E2E testing
to verify the metadata created by controller,
We are not checking the generated omap data,
but we will be verify PVC resize and binding
pvc to application.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
0dd152928d e2e: add option to set retainpolicy for rbd storageclass
added an option to set retain policy for
rbd storageclasses.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
30af703a2f rbd: add controller to main
initialize and start the rbd controller when
we the driver type is controller.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
68bd44beba rbd: add new controller to regenerate omap data
In the case of Disaster Recovery failover, the
user expected to create the static PVC's. We have
planned not to go with the PVC name and namespace
for many reasons (as in kubernetes it's planned to
support PVC transfer to a new namespace with a
different name and with new features coming in
like data populator etc). For now, we are
planning to go with static PVC's to support
async mirroring.

During Async mirroring only the RBD images are
mirrored to the secondary site, and when the
user creates the static PVC's on the failover
we need to regenerate the omap data. The
volumeHandler in PV spec is an encoded string
which contains clusterID and poolID and image UUID,
The clusterID and poolID won't remain same on both
the clusters, for that cephcsi need to generate the
new volume handler and its to create a mapping
between new volume handler and old volume handler
with that whenever cephcsi gets csi requests it
check if the mapping exists it will pull the new
volume handler and continues other operations.

The new controller watches for the PVs created,
It checks if the omap exists if it doesn't it
will regenerate the entire omap data.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
5af3fe5deb rebase: add controller runtime dependency
this commits add the controller runtime
and its dependency to the vendor.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
14700b89d1 rbd: update inuse logic of a rbd image
in case of mirrored image, if the image is
primary a watcher will be added by the rbd
mirror deamon on the rbd image.
we have to consider 2 watcher to check image
is in use.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Madhu Rajanna
ba84f14241 journal: create object with provided UUID
incase of async mirroring the volume UUID is
retrieved from the volume name, instead of cephcsi
generating a new UUID it should reserve the passed
UUID it will be useful when we support both metro DR
and async mirroring on a kubernetes clusters.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-28 18:50:00 +00:00
Niels de Vos
1f18e876f0 e2e: use docker.io/library as prefix for official images
Docker Hub offers a way to pull official images without any project
prefix, like "docker.io/vault:latest". This does a redirect to the
images located under "docker.io/library".

By using the full qualified image name, a redirect gets removed while
pulling the images. This reduces the likelyhood of hittin Docker Hub
pull rate-limits.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-26 13:51:02 +00:00
Niels de Vos
954ac97d22 ci: add more logging during Rook deployment
It seems that the new log_errors() function does not get triggered when
the script hits `exit 1` conditions in functions. The functions should
return a non-0 value, not cause an exit of the script.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 18:12:29 +00:00
Niels de Vos
ed033153ea ci: gather logs when deploying Rook fails
Log a few commands that help troubleshooting Rook deployment issues.
This might need to get extended with more commands.

Updates: #1636
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 14:53:38 +00:00
Niels de Vos
eaeee8ac3d deploy: use docker.io for unqualified image names
Images that have an unqualified name (no explicit registry) come from
Docker Hub. This can be made explicit by adding docker.io as prefix. In
addition, the default :latest tag has been added too.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-24 10:27:33 +00:00
Niels de Vos
7a69c5e238 build: add ROOK_VERSION to build.env
The CentOS CI jobs use Rook v1.3.9, this version should be places in
build.env just like other versions that the CI jobs detect.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-23 11:15:23 +00:00
Niels de Vos
4fd0294eb7 e2e: pull centos image from registry.centos.org
The BlockVolume PVC tests consume the example files that refer to
"centos:latest" without registry. This means that the images will get
pulled from Docker Hub, which has rate limits preventing CI jobs from
pulling images.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-19 16:00:33 +00:00
Madhu Rajanna
7d229c2369 build: update imagepullpolicy for vault
this allows the image to be reused instead of pulling
it again.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-19 16:00:33 +00:00
Madhu Rajanna
168526a906 e2e: use centos as image for normal user validation
Reduce the number of images that get pulled from Docker Hub. Use the
official CentOS container registry instead.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-19 16:00:33 +00:00
Madhu Rajanna
8d86bac9b7 e2e: set imagePullPolicy to ifNotPresent
If the imagePullPolicy is not set and the image
tag is empty or latest the image is always pulled.
This commit sets the policy to pull image if not
present.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-18 13:43:52 +00:00
Madhu Rajanna
8d3a44d0c4 rbd: add minsnapshotsonimage flag
An rbd image can have a maximum number of
snapshots defined by maxsnapshotsonimage
On the limit is reached the cephcsi will
start flattening the older snapshots and
returns the ABORT error message, The Request
comes after this as to wait till all the
images are flattened (this will increase the
PVC creation time.  Instead of waiting till
the maximum snapshots on an RBD image, we can
have a soft limit, once the limit reached
cephcsi will start flattening the task to
break the chain. With this PVC  creation time
will only be affected when the hard limit
(minsnapshotsonimage) reached.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-18 05:59:20 +00:00
Niels de Vos
880b5bb427 ci: use the Fedora container registry for cephcsi:test
This reduces the dependency on Docker, where image pull rate limits are
seen in the CI.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-17 09:28:02 +00:00
Prasanna Kumar Kalever
817edfd1c7 cleanup: remove the use of text in markdown
We do not have `text` in the new section of the MarkDown Rules. Hence
dropping them.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-11-11 13:18:05 +00:00
Prasanna Kumar Kalever
8475a3b97e doc: update about a markdown rule in coding guide
Update the coding guide about MD014, i.e.
Dollar signs used before commands without showing output

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-11-11 13:18:05 +00:00
Prasanna Kumar Kalever
2945f7b669 cleanup: stick to standards when using dollar-sign in md
MD014 - Dollar signs used before commands without showing output
The dollar signs are unnecessary, it is easier to copy and paste and
less noisy if the dollar signs are omitted. Especially when the
command doesn't list the output, but if the command follows output
we can use `$ ` (dollar+space) mainly to differentiate between
command and its ouput.

scenario 1: when command doesn't follow output
```console
cd ~/work
```

scenario 2: when command follow output (use dollar+space)
```console
$ ls ~/work
file1 file2 dir1 dir2 ...
```

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-11-11 13:18:05 +00:00
Prasanna Kumar Kalever
fcaa332921 doc: improve e2e tests guide
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-11-11 13:18:05 +00:00