mirror of
https://github.com/ceph/ceph-csi.git
synced 2024-12-26 23:10:22 +00:00
539686329f
This commit address the issue- https://github.com/ceph/ceph-csi/issues/3448. Signed-off-by: riya-singhal31 <rsinghal@redhat.com>
285 lines
11 KiB
Markdown
285 lines
11 KiB
Markdown
# Ceph Filesystem fscrypt Support
|
|
|
|
## Problem Description
|
|
|
|
As a Ceph Container Storage Interface (Ceph CSI) user, I want a cloud native way
|
|
to manage keys and enable encryption on Ceph Filesystem (CephFS) volumes.
|
|
|
|
In order to access encrypted volumes without Ceph CSI, this can be done by unlocking
|
|
volumes with user space tools.
|
|
|
|
## Background
|
|
|
|
*fscrypt* or *FSCrypt* is a Linux Kernel feature that allows the filesystem
|
|
to support the transparent encryption of files and directories. Local
|
|
filesystems like ext4 and F2FS (Flash-Friendly File System) support this feature
|
|
already.
|
|
|
|
Work is in progress to add fscrypt support to CephFS for filesystem-level encryption.
|
|
|
|
- [FSCrypt Kernel Documentation](https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html)
|
|
- Management Tools
|
|
- [`fscrypt`](https://github.com/google/fscrypt)
|
|
- [`fscryptctl`](https://github.com/google/fscryptctl)
|
|
- [Ceph Feature Tracker: "Add fscrypt support to the kernel CephFS client"](https://tracker.ceph.com/issues/46690)
|
|
- [`fscrypt` design document](https://goo.gl/55cCrI)
|
|
|
|
**NOTE:** In this document, fscrypt refers to the filesystem-level encryption
|
|
feature, while `fscrypt` specifically refers to the user space tool.
|
|
|
|
## Terminology
|
|
|
|
- *FSCrypt*, *fscrypt* - Linux Kernel filesystem-level encryption feature
|
|
- `fscrypt` (code formatted) - User space tool manage keys and encryption policies
|
|
- `fscryptctl` (code formatted) - Low-level user space tool manage
|
|
keys and encryption policies
|
|
- *subvolume* - CephFS subvolume
|
|
- *unlocking* - Using a key to make an encrypted filesystem
|
|
accessible in plain text
|
|
- *protector* (fscrypt) - A single method or secret plus data used to
|
|
derive a *protector key*. Example: user login passphrase
|
|
- *protector key* (fscrypt) - A symmetric key derived from an external
|
|
source. Used by a *policy* to unwrap a *policy key*
|
|
- *policy key* (fscrypt) - An encryption key passed to the kernel to
|
|
unlock a directory
|
|
- *policy* (fscrypt) - A collection of directories protected and unlocked as a unit
|
|
- *KMS* - Key management system
|
|
|
|
## User Visible Change
|
|
|
|
Similar to the existing RADOS Block Device (RBD) encryption support,
|
|
we propose adding encryption support in the configuration and Key
|
|
Management Service (KMS) integration.
|
|
|
|
In this example, a user may enable encryption using storage class keys similar
|
|
to RBD. Ceph CSI then configures and unlocks the persistent volumes and CephFS
|
|
subvolumes.
|
|
|
|
Due to the way `fscrypt` stores metadata, subvolumes have a regular root
|
|
directory containing a `/.fscrypt` directory and a
|
|
`/ceph-csi-encrypted` directory. The first contains `fscrypt` metadata; the
|
|
latter is the fscrypt-enabled directory made that is accessible to pods.
|
|
|
|
Example configuration using a secrets-based KMS:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: cephfs-storage-encryption-secret
|
|
stringData:
|
|
encryptionPassphrase: verysecretpassword
|
|
---
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: csi-cephfs-sc-encrypted
|
|
provisioner: cephfs.csi.ceph.com
|
|
parameters:
|
|
clusterID: <cluster-id>
|
|
fsName: cephfs
|
|
|
|
encrypted: "true"
|
|
encryptionKMSID: "user-ns-secrets-metadata"
|
|
|
|
csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
|
|
csi.storage.k8s.io/provisioner-secret-namespace: default
|
|
csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
|
|
csi.storage.k8s.io/controller-expand-secret-namespace: default
|
|
csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
|
|
csi.storage.k8s.io/node-stage-secret-namespace: default
|
|
reclaimPolicy: Delete
|
|
allowVolumeExpansion: true
|
|
mountOptions:
|
|
- debug
|
|
```
|
|
|
|
The change will leverage the existing Ceph CSI KMS and support any
|
|
integration now available to RBD encryption
|
|
|
|
## Implementation
|
|
|
|
We suggest to leverage the encryption features in Ceph CSI and integrate
|
|
that with `fscrypt`, a Go tool, and the library for key management and configuration
|
|
of the fscrypt kernel feature.
|
|
|
|
Ceph CSI and `fscrypt` have a lot of overlap between their key management
|
|
features. The Key Management section will go into detail on how and where keys
|
|
are managed.
|
|
|
|
- Ceph CSI provides the user facing configuration and access to key
|
|
management systems
|
|
- `fscrypt` handles key derivation, storage of wrapped keys and metadata
|
|
|
|
The current CephFS subvolume root will remain untouched with the exception that
|
|
the subvolume root is not bind mounted into the pod, but rather a well-known
|
|
subdirectory. The root will contain a `/.fscrypt` directory managed by `fscrypt`.
|
|
|
|
`fscrypt` requires access to a mounted filesystem and therefore the encryption setup
|
|
must take place in the `NodeStageVolume` request handler instead of `CreateVolume`.
|
|
This is the same case for RBD. The set up will take place right between
|
|
subvolume mount and bind mount to the container namespace.
|
|
|
|
Additional checks after unlocking will ensure that a container operates on an
|
|
unlocked encrypted directory and never on directory that has fscrypt enabled.
|
|
|
|
### Key Management
|
|
|
|
```mermaid
|
|
graph LR
|
|
vault[Vault, Default] -->|data encryption key| csi_kms_integrated
|
|
secrets[Passphrase<br>K8s secrets, AWS, IBM Key Protect] -->|passphrase| csi_kms_metadata
|
|
|
|
|
|
subgraph ceph_csi_kms[Ceph CSI KMS]
|
|
csi_kms_metadata['metadata' type DEK store]
|
|
csi_kms_integrated['integrated' type DEK store]
|
|
|
|
end
|
|
|
|
|
|
subgraph protector[fscrypt protector]
|
|
custom_passphrase[CustomPasswordSource];
|
|
raw_passphrase[RawKeySource];
|
|
|
|
protector_metadata[(/.fscrypt/protectors)];
|
|
|
|
protector_metadata -->|wrapped key| custom_passphrase;
|
|
protector_metadata -->|wrapped key| raw_passphrase;
|
|
|
|
end
|
|
|
|
subgraph policy[fscrypt policy]
|
|
policy_unwrap
|
|
policy_metadata[(/.fscrypt/policies)]
|
|
|
|
policy_metadata -->|wrapped key| policy_unwrap
|
|
end
|
|
|
|
|
|
csi_kms_metadata -->|DEK| custom_passphrase
|
|
csi_kms_integrated -->|DEK| raw_passphrase
|
|
|
|
custom_passphrase -->|protector key| policy_unwrap
|
|
raw_passphrase -->|protector key| policy_unwrap
|
|
|
|
policy_unwrap -->|policy key| kernel[Kernel API]
|
|
|
|
```
|
|
|
|
The diagram shows the keys flowing from Ceph CSI to the Kernel API unlocking
|
|
a directory. On the way, key material from Ceph CSI passes two key derivation
|
|
steps in fscrypt:
|
|
|
|
- *protectors* and
|
|
- *policies*
|
|
|
|
`fscrypt` supports multiple *protectors*. These may source secrets from login
|
|
passwords, custom passwords or soon Ceph CSI. Unlocking a protector
|
|
yields a *protector key* that is then used to unlock a *policy*.
|
|
|
|
A *policy* may unlock multiple directories. In our case there will be
|
|
only a single policy for a single well-known directory on the subvolume root.
|
|
A policy is used to derive a *policy key*, which is passed to the Kernel API
|
|
along with other settings, such as the desired encryption algorithm.
|
|
|
|
Going back to the beginning of the diagram and looking at the interface between
|
|
`fscrypt` and Ceph CSI one can see that the two data encryption key (DEK)
|
|
styles (*metadata* and *integrated*) map to different `fscrypt` protectors.
|
|
|
|
The `fscrypt` protector of key sources `CustomPasswordSource` and `RawKeySource`
|
|
differ in how they derive a key from a source. Refer to the `fscrypt` design
|
|
doc for details.
|
|
|
|
Metadata DEKs: In the RBD case, Ceph CSI stores a wrapped key in the
|
|
RBD volume metadata and then a user configured secret (for example, a Kubernetes
|
|
secret) is passed to a key derivation function (KDF) to then unwrap the
|
|
key. The resulting key unlocks the volume.
|
|
|
|
Since `fscrypt` already stores wrapped keys there is no need for an
|
|
extra layer of wrapping. We can also skip the KDF and use a
|
|
`CustomPasswordSource` to pass the Ceph CSI secret directly to
|
|
`fscrypt`.
|
|
|
|
With integrated DEKs (for example, Vault) Ceph CSI uses a key from a KMS
|
|
directly. To integrate this with `fscrypt` we use a
|
|
`RawKeySource`, that is similar to a `CustomPasswordSource`, but skips
|
|
the KDF.
|
|
|
|
As the diagram shows, both policies and protectors require a metadata
|
|
store. The default `fscrypt` data store is in a `/.fscrypt`
|
|
directory under a filesystem root. The `fscrypt` design doc details
|
|
alternatives and explains what data is stored.
|
|
|
|
To be compatible with `fscrypt`, this directory requires support as
|
|
well. The downside of this is that we lose the CephFS subvolume root to
|
|
metadata and encrypted data will reside under a well-known
|
|
subdirectory (for example, `/ceph-csi-encrypted`).
|
|
|
|
## Dependencies
|
|
|
|
The proposed change is tailored to CephFS and requires CephFS support
|
|
to work *with* CephFS. The kernel APIs however are not specific to
|
|
CephFS and are unlikely to change as they only deal with configuration
|
|
and key management. There is no direct dependency on CephFS. Using the
|
|
proposed features will simply fail at runtime, when neither Ceph nor
|
|
the Kernel have the appropriate support. At build time this feature
|
|
does not require CephFS fscrypt support.
|
|
|
|
Runtime dependencies:
|
|
|
|
- Kernel >= v5.4 with `CONFIG_FS_ENCRYPTION=y`
|
|
- CephFS kernel client fscrypt support [Ceph Feature Tracker](https://tracker.ceph.com/issues/46690)
|
|
|
|
Build dependencies:
|
|
|
|
- `google/fscrypt` library, which has minimal build dependencies
|
|
([fscrypt doc](https://github.com/google/fscrypt#building-and-installing))
|
|
|
|
## Alternatives
|
|
|
|
### [ceph-csi-kms] Key Management: Policy Key Directly From Ceph CSI
|
|
|
|
A simpler approach to the one proposed above, but incompatible with
|
|
`fscrypt`. To unlock a subvolume, the user would have to use Ceph CSI.
|
|
The implementation is similar to the RBD encryption feature.
|
|
It uses the low-level `fscryptctl` tool to set a policy key
|
|
from a Ceph CSI data encryption key.
|
|
|
|
Ceph CSI KMS requires metadata data encryption key storage that can use
|
|
xattrs on a mounted CephFS filesystem.
|
|
|
|
A prototype showing this approach is available: [repository](https://github.com/irq0/ceph-csi/tree/wip/fscrypt)
|
|
|
|
Benefits:
|
|
|
|
- Simpler key wrapping
|
|
- No `/.fscrypt` on the subvolume root
|
|
|
|
Drawbacks:
|
|
|
|
- `fscryptctl` is a C tool and does not lend itself to be integrated into Ceph CSI
|
|
- Incompatible with `fscrypt`
|
|
- Does not support unlocking with any of possibly multiple keys
|
|
configured (`fscrypt` protectors feature)
|
|
|
|
### [manual] Manual Setup
|
|
|
|
For completeness, a user may set up FSCrypt without any support in
|
|
Ceph CSI. Both `fscrypt` and `fscryptctl` work in containers
|
|
and may even be used with the proposed change or alternative [ceph-csi-kms].
|
|
|
|
The following links provide examples from the documentation that also apply to
|
|
CephFS:
|
|
|
|
- [fscrypt example usage](https://github.com/google/fscrypt#example-usage)
|
|
- [fscryptctl example usage](https://github.com/google/fscryptctl#example-usage)
|
|
|
|
### [subdirs] Support Unlocking Arbitrary subdirs (instead of subvolume basis)
|
|
|
|
An extension to the proposal: As mentioned in the implementation section, a
|
|
`fscrypt` policy may apply to multiple directories and from a set of protectors
|
|
any suffices to unlock a policy. A user may configure a complex mapping of
|
|
subdirectories and Ceph CSI secret sources to unlock different parts of a CephFS
|
|
subvolume with different keys.
|