These deployment files are heavily based on the CephFS deployment.
Deploying an environment with these files work for me in minikube. This
should make it possible to add e2e testing as well.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
The API is extended for generation of the NFS CSIDriver object. The
YAML file under deploy/ was created by `yamlgen`.
The contents of the csidriver.yaml file is heavily based on the upstream
CSIDriver from the Kubernetes csi-driver-nfs project.
Because ./tools/yamlgen uses the API, it gets copied under vendor/ .
This causes two copies of the API to be included in the repository, but
that can not be prevented, it seems.
See-also: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/deploy/csi-nfs-driverinfo.yaml
Signed-off-by: Niels de Vos <ndevos@redhat.com>
OIDC token file path has been modified from
`/var/run/secrets/token` to `/run/secrets/tokens`.
This has been done to ensure compliance with
FHS 3.0.
refer:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s13.html
Signed-off-by: Rakshith R <rar@redhat.com>
With Amazon STS and kubernetes cluster is configured with
OIDC identity provider, credentials to access Amazon KMS
can be fetched using oidc-token(serviceaccount token).
Each tenant/namespace needs to create a secret with aws region,
role and CMK ARN.
Ceph-CSI will assume the given role with oidc token and access
aws KMS, with given CMK to encrypt/decrypt DEK which will stored
in the image metdata.
Refer: https://docs.aws.amazon.com/STS/latest/APIReference/welcome.htmlResolves: #2879
Signed-off-by: Rakshith R <rar@redhat.com>
Mounts managed by ceph-fuse may get corrupted by e.g. the ceph-fuse process
exiting abruptly, or its parent container being terminated, taking down its
child processes with it.
This commit adds checks to NodeStageVolume and NodePublishVolume procedures
to detect whether a mountpoint in staging_target_path and/or target_path is
corrupted, and remount is performed if corruption is detected.
Signed-off-by: Robert Vasek <robert.vasek@cern.ch>
The CentOS 8 repository for Apache Arrow has been removed. This causes
container-image builds fail with the following error:
Errors during downloading metadata for repository 'apache-arrow-centos':
- Status code: 404 for https://apache.jfrog.io/artifactory/arrow/centos/8/x86_64/repodata/repomd.xml (IP: 54.190.66.70)
Error: Failed to download metadata for repo 'apache-arrow-centos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
The Ceph base image has `arrow/centos/8` configured, maybe Apache Arrow
offers a CentOS Stream 8 repository now? Once the Ceph container-image
has been updated, the repository can be enabled again.
Ceph-CSI does not depend on Apache Arrow, so there is no functional
change by disabling the repository.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
updating external resizer image version
from 1.3.0 to latest available release i.e
1.4.0
1.4.0 changelog link
https://github.com/kubernetes-csi/
external-resizer/blob/master/CHANGELOG/CHANGELOG-1.4.md
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
This commit updates sidecars to the latest available version
which is compatible with kubernetes 1.23 and csi spec 1.5
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Deployments place all sockets for communicating with CSI components in
the shared `/csi` directory. The CSI-Addons socket was introduced
recently, but not configured to be in the same location (by default
placed in `/tmp`).
Signed-off-by: Niels de Vos <ndevos@redhat.com>
The generated files under the deploy/ directory contain an empty YAML
document that may cause confusion for some versions of kubectl. Dropping
the unneeded `---` start of the file for the header should make parsing
of the deployment artifacts a little less error prone.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
This initial version of yamlgen generates deploy/scc.yaml based on the
deployment artifact that is provided by the new api/deploy/ocp package.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
cephfs deployment doesnot need extra permission like
privileged,Capabilities and remove unwanted volumes.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
rbd deployment doesnot need extra permission like
privileged,Capabilities and remove unwanted volumes.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
we dont need securityContext for the rbd provisioner
pod as its not doing any special operations like map
,unmap selinux etc.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
we dont need securityContext for the cephfs provisioner
pod as its not doing any special operations like mounts,
selinux etc.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Problem:
--------
1. rbd-nbd by default logs to /var/log/ceph/ceph-client.admin.log,
Unfortunately, container doesn't have /var/log/ceph directory hence
rbd-nbd is not logging now.
2. Rbd-nbd logs are not persistent across nodeplugin restarts.
Solution:
--------
Provide a host path so that log directory is made available, and the
logs persist on the hostnode across container restarts.
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
- mount host's /etc/selinux in node plugins
- process mount options in all code paths for cephfs volume options
Signed-off-by: Alexandre Lossent <alexandre.lossent@cern.ch>
This change resolves a typo for installing the CSIDriver
resource in Kubernetes clusters before 1.18,
where the apiVersion is incorrect.
See also:
https://kubernetes-csi.github.io/docs/csi-driver-object.html
[ndevos: replace v1betav1 in examples with v1beta1]
Signed-off-by: Thomas Kooi <t.j.kooi@avisi.nl>
Problem:
-------
For rbd nbd userspace mounter backends, after a restart of the nodeplugin
all the mounts will start seeing IO errors. This is because, for rbd-nbd
backends there will be a userspace mount daemon running per volume, post
restart of the nodeplugin pod, there is no way to restore the daemons
back to life.
Solution:
--------
The volume healer is a one-time activity that is triggered at the startup
time of the rbd nodeplugin. It navigates through the list of volume
attachments on the node and acts accordingly.
For now, it is limited to nbd type storage only, but it is flexible and
can be extended in the future for other backend types as needed.
From a few feets above:
This solves a severe problem for nbd backed csi volumes. The healer while
going through the list of volume attachments on the node, if finds the
volume is in attached state and is of type nbd, then it will attempt to
fix the rbd-nbd volumes by sending a NodeStageVolume request with the
required volume attributes like secrets, device name, image attributes,
and etc.. which will finally help start the required rbd-nbd daemons in
the nodeplugin csi-rbdplugin container. This will allow reattaching the
backend images with the right nbd device, thus allowing the applications
to perform IO without any interruptions even after a nodeplugin restart.
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
Nodeplugin needs below cluster roles:
persistentvolumes: get
volumeattachments: list, get
These additional permissions are needed by the volume healer. Volume healer
aims at fixing the volume health issues at the very startup time of the
nodeplugin. As part of its operations, volume healer has to run through
the list of volume attachments and understand details about each
persistentvolume.
The later commits will use these additional cluster roles.
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
The provisioner and node-plugin have the capability to connect to
Hashicorp Vault with a ServiceAccount from the Namespace where the PVC
is created. This requires permissions to read the contents of the
ServiceAccount from an other Namespace than where Ceph-CSI is deployed.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
csidriver object can be created on the kubernetes
for below reason.
If a CSI driver creates a CSIDriver object,
Kubernetes users can easily discover the CSI
Drivers installed on their cluster
(simply by issuing kubectl get CSIDriver)
Ref: https://kubernetes-csi.github.io/docs/csi-driver-object.html#what-is-the-csidriver-object
attachRequired is always required to be set to
true to avoid issue on RWO PVC.
more details about it at https://github.com/rook/rook/pull/4332
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
use the latest version of csi-snapshotter sidecar image at the
provisioner templates
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
set system-cluster-critical priorityclass on
provisioner pods. the system-cluster-critical is
having lowest priority compared to node-critical.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
set system-node-critical priority on the plugin
pods, as its the highest priority and this need to
be applied on plugin pods as its critical for
storage in cluster.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
as provisioner need to get the configmap from
different namespace to check tenant configuration.
added the clusterrole get access for the same.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Tenants can have their own ConfigMap that contains connection parameters
to the Vault Service where the PV encyption keys are located. It is
possible for a Tenant to use a different Vault Service than the one
configured by the Storage Admin who deployed Ceph-CSI.
For this, the node-plugin needs to be able to read the ConfigMap from
the Tenants namespace.
See-also: docs/design/proposals/encryption-with-vault-tokens.md
Signed-off-by: Niels de Vos <ndevos@redhat.com>
if the kms encryption configmap is not mounted
as a volume to the CSI pods, add the code to
read the configuration from the kubernetes. Later
the code to fetch the configmap will be moved to
the new sidecar which is will talk to respective
CO to fetch the encryption configurations.
The k8s configmap uses the standard vault spefic
names to add the configurations. this will be converted
back to the CSI configurations.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
In order to fetch the Kubernetes Secret with the Vault Token for a
Tenant, the ClusterRole needs to allow reading Secrets from all
Kubernetes Namespaces (each Tenant has their own Namespace).
Signed-off-by: Niels de Vos <ndevos@redhat.com>
This argument in csi-provisioner sidecar allows us to receive pv/pvc
name/namespace metadata in the createVolume() request.
For ex:
csi.storage.k8s.io/pvc/name
csi.storage.k8s.io/pvc/namespace
csi.storage.k8s.io/pv/name
This is a useful information which can be used depend on the use case we
have at our driver. The features like vault token enablement for multi
tenancy, RBD mirroring ..etc can consume this based on the need.
Refer: #1305
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
external-provisioner is exposing a new argument
to set the default fstype while starting the provisioner
sidecar, if the fstype is not specified in the storageclass
the default fstype will be applied for the pvc created from
the storageclass.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
with csi-provisioner v2.x the topology based
provisioning will not have any backward compatibility
with older version of kubernetes, if the nodes are
not labeled with topology keys, the pvc creation
is going to get fail with error `accessibility
requirements: no available topology found`, disabling
the topology based provisioning by default, if user want
to use it he can always enable it.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
This PR makes the changes in csi templates and
upgrade documentation required for updating
csi sidecar images.
Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
updated deployment template for the new controller and
also added `update` configmap RBAC for the controller
as the controller uses the configmap for the leader
election.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
When running a simple build with only the required arguments, the
following warning are reported:
$ buildah bud --build-arg=BASE_IMAGE=ceph/ceph:v15 --build-arg=GO_ARCH=amd64 -f ./deploy/cephcsi/image/Dockerfile .
...
STEP 15: COPY . ${SRC_DIR}
STEP 16: RUN make cephcsi
cephcsi image settings: quay.io/cephcsi/cephcsi version canary
make: git: Command not found
make: git: Command not found
if [ ! -d ./vendor ]; then (go mod tidy && go mod vendor); fi
make: git: Command not found
...
STEP 23: COMMIT
Getting image source signatures
...
Writing manifest to image destination
Storing signatures
--> 239b19c4049
git is used to detect the current commit, and store it in the binary
that is built. Without the commit, the "Git Commit:" in the output is
empty, making it impossible to get the exact version:
$ podman run --rm 239b19c4049 --version
Cephcsi Version: canary
Git Commit:
Go Version: go1.15
Compiler: gc
Platform: linux/amd64
Kernel: 5.8.4-200.fc32.x86_64
Signed-off-by: Niels de Vos <ndevos@redhat.com>
The added anti-affinity rules prevent provisioner operators from scheduling on
the same nodes. The kubernetes scheduler will spread the pods across nodes to
improve availability during node failures.
Signed-off-by: Nico Berlee <nico.berlee@on2it.net>
The lifecycle preStop hook fails on container stop / exit
because /bin/sh is not present in the driver registrar container
image.
the driver-registrar will remove the socket file
before stopping. we dont need to have any preStop hook
to remove the socket as it was not working as expected
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The aggregate clusterrole were designed for the scenario where
the rules are not completely owned by one component.
the aggregate rules can be removed and simplify
certain issues around upgrades.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
as v1.0.0 is deprecated we need to remove the support
for it in the Next coming (v3.0.0) release. This PR
removes the support for the same.
closes#882
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
`make image-cephcsi` will fail when Golang is not installed. There is no
strict requirement for Golang to be available, it is only used to gather
the architecture of the OS where the image is built. It is possible to
build the image successfully with `make image-cephcsi GOARCH=amd64`.
In case Golang is not installed, GOARCH can not be detected
automatically. This will cause a failure while installing Golang in the
container image. Because the failure is not very clear, display a
warning in the case the GO_ARCH (from ${GOARCH} in the Makefile) is not
set.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
added Hardlimit and Softlimit flags for cephcsi
arguments. When the Softlimit is reached cephcsi
will start a background task to flatten the rbd
image and return success and if the hardlimit
is reached it will start a background task
to flatten the rbd image and return ready
to use as false to make sure that the image
will not be used until it is flatten.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
For building a arm64 container image on amd64, it is needed to configure
the system specifically for that. In order to prevent including a amd64
executable in a arm64 image, a check has been added.
When running an arm64 executable on a amd64 system, an error should
occur when cross architecture containers are not supported. This can be
triggered when running `make image-cephcsi GO_ARCH=arm64` on a amd64
system.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Considering this parameter is available for other sidecars we should
have a parity between the sidecars. Adding it for the same reason
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
When building with go-ceph, there are several dynamically linked
libraries used directly (libcephfs, librados and librbd), and many more
indirectly.
By adding an additional RUN statement to check if all libraries are
available in the final image, problems related to missing libraries
should be caught before publishing/consuming the image.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
unlike other containers, image pull policy of
csi-snapshotter was set to "Always", which can
be changed to pull only if not present.
Signed-off-by: Yug Gupta <ygupta@redhat.com>
with the help of qemu-user-static we can run
different architecute contains.
more info at https://github.com/multiarch/qemu-user-static
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Add support for multi architecture build
for cephcsi. with multistage docker build
we we build cephcsi binary for both arm64
and amd64 architecture.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Recently resizer 0.5.0 has been released.
This PR updated the resizer container from
v0.4.0 to v0.5.0
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
As kubernetes CSI sidecar is exposing the
GRPC mertics we can make use of the same in
ceph-csi we dont need to expose our own.
update: #881
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
As we have the octopus as the latest
release base image,this PR updates the
base image in Dockerfile
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
There are currently unwanted RBAC permission
is given for ceph-csi, This PR reduces removes
such unwanted RBAC resources.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
PR #282 introduces the mount cache to
solve cephfs fuse mount issue when cephfs plugin pod
restarts .This is not working as intended. This PR removes
the code for maintainability.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
- adds proposal document for PVC encryption from PR448
- adds per-volume encription by generating encryption passphrase
for each volume and storing it in a KMS
- adds HashiCorp Vault integration as a KMS for encryption passphrases
- avoids encrypting volume second time if it was already encrypted but
no file system created
- avoids unnecessary checks if volume is a mapped device when encryption
was not requested
- prevents resizing encrypted volumes (it is not currently supported)
- prevents creating snapshots from encrypted volumes to prevent attack
on encryption key (security guard until re-encryption of volumes
implemented)
Signed-off-by: Vasyl Purchel vasyl.purchel@workday.com
Signed-off-by: Andrea Baglioni andrea.baglioni@workday.comFixes#420Fixes#744
`/run/mount` need to be share between host and
csi-plugin containers for `/run/mount/utab`
this is required to ensures that the network
is not stopped prior to unmounting the network devices.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
On systems with SELinux enabled, non-privileged containers
can't access data of privileged containers. Since the socket
is exposed by privileged containers, all sidecars must be
privileged too. This is needed only for containers running
in daemonset as we are using bidirectional mounts in daemonset
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
currently, we are making use of host path directory
to store the provisioner socket, as this
the socket is not needed by anyone else other than
containers inside the provisioner pod using the
empty directory to store this socket is the best option.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Update CI merge job to build and push Arm64 image to
quay.io/cephcsi/cephcsi:version-arm64.
Add CI PR job running on Travis Arm64 nodes to make sure cephcsi
compiles successfully on Arm64.
No CI test job is availabe for Arm64 now due to below issues
- k8s-csi sidecar images for Arm64 are not available
- Travis Arm64 CI job runs inside unprivileged LXD which blocks
launching minikube test environment
Signed-off-by: Yibo Cai <yibo.cai@arm.com>
