From b7ec0b2d6688bfa621a8cb03d8b0b4f107f79cc4 Mon Sep 17 00:00:00 2001 From: Marcel Lauhoff Date: Tue, 1 Mar 2022 15:21:46 +0100 Subject: [PATCH] doc: add proposal for CephFS fscrypt integration Add proposal document covering key management integration of Ceph CSI and https://github.com/google/fscrypt Updates: #1563 Signed-off-by: Marcel Lauhoff --- docs/design/proposals/cephfs-fscrypt.md | 284 ++++++++++++++++++++++++ 1 file changed, 284 insertions(+) create mode 100644 docs/design/proposals/cephfs-fscrypt.md diff --git a/docs/design/proposals/cephfs-fscrypt.md b/docs/design/proposals/cephfs-fscrypt.md new file mode 100644 index 000000000..b96d94fe3 --- /dev/null +++ b/docs/design/proposals/cephfs-fscrypt.md @@ -0,0 +1,284 @@ +# Ceph Filesystem fscrypt Support + +## Problem Description + +As a Ceph Container Storage Interface (Ceph CSI) user, I want a cloud native way +to manage keys and enable encryption on Ceph Filesystem (CephFS) volumes. + +In order to access encrypted volumes without Ceph CSI, this can be done by unlocking +volumes with user space tools. + +## Background + +*fscrypt* or *FSCrypt* is a Linux Kernel feature that allows the filesystem +to support the transparent encryption of files and directories. Local +filesystems like ext4 and F2FS (Flash-Friendly File System) support this feature +already. + +Work is in progress to add fscrypt support to CephFS for filesystem-level encryption. + +- [FSCrypt Kernel Documentation](https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html) +- Management Tools + - [`fscrypt`](https://github.com/google/fscrypt) + - [`fscryptctl`](https://github.com/google/fscryptctl) +- [Ceph Feature Tracker: "Add fscrypt support to the kernel CephFS client"](https://tracker.ceph.com/issues/46690) +- [`fscrypt` design document](https://goo.gl/55cCrI) + +**NOTE:** In this document, fscrypt refers to the filesystem-level encryption +feature, while `fscrypt` specifically refers to the user space tool. + +## Terminology + +- *FSCrypt*, *fscrypt* - Linux Kernel filesystem-level encryption feature +- `fscrypt` (code formatted) - User space tool manage keys and encryption policies +- `fscryptctl` (code formatted) - Low-level user space tool manage + keys and encryption policies +- *subvolume* - CephFS subvolume +- *unlocking* - Using a key to make an encrypted filesystem + accessible in plain text +- *protector* (fscrypt) - A single method or secret plus data used to + derive a *protector key*. Example: user login passphrase +- *protector key* (fscrypt) - A symmetric key derived from an external + source. Used by a *policy* to unwrap a *policy key* +- *policy key* (fscrypt) - An encryption key passed to the kernel to + unlock a directory +- *policy* (fscrypt) - A collection of directories protected and unlocked as a unit +- *KMS* - Key management system + +## User Visible Change + +Similar to the existing RADOS Block Device (RBD) encryption support, +we propose adding encryption support in the configuration and Key +Management Service (KMS) integration. + +In this example, a user may enable encryption using storage class keys similar +to RBD. Ceph CSI then configures and unlocks the persistent volumes and CephFS +subvolumes. + +Due to the way `fscrypt` stores metadata, subvolumes have a regular root +directory containing a `/.fscrypt` directory and a +`/ceph-csi-encrypted` directory. The first contains `fscrypt` metadata; the +latter is the fscrypt-enabled directory made that is accessible to pods. + +Example configuration using a secrets-based KMS: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: cephfs-storage-encryption-secret +stringData: + encryptionPassphrase: verysecretpassword +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: csi-cephfs-sc-encrypted +provisioner: cephfs.csi.ceph.com +parameters: + clusterID: + fsName: cephfs + + encrypted: "true" + encryptionKMSID: "user-ns-secrets-metadata" + + csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret + csi.storage.k8s.io/provisioner-secret-namespace: default + csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret + csi.storage.k8s.io/controller-expand-secret-namespace: default + csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret + csi.storage.k8s.io/node-stage-secret-namespace: default +reclaimPolicy: Delete +allowVolumeExpansion: true +mountOptions: + - debug +``` + +The change will leverage the existing Ceph CSI KMS and support any +integration now available to RBD encryption + +## Implementation + +We suggest to leverage the encryption features in Ceph CSI and integrate +that with `fscrypt`, a Go tool, and the library for key management and configuration +of the fscrypt kernel feature. + +Ceph CSI and `fscrypt` have a lot of overlap between their key management +features. The Key Management section will go into detail on how and where keys +are managed. + +- Ceph CSI provides the user facing configuration and access to key + management systems +- `fscrypt` handles key derivation, storage of wrapped keys and metadata + +The current CephFS subvolume root will remain untouched with the exception that +the subvolume root is not bind mounted into the pod, but rather a well-known +subdirectory. The root will contain a `/.fscrypt` directory managed by `fscrypt`. + +`fscrypt` requires access to a mounted filesystem and therefore the encryption setup +must take place in the `NodeStageVolume` request handler instead of `CreateVolume`. +This is the same case for RBD. The set up will take place right between +subvolume mount and bind mount to the container namespace. + +Additional checks after unlocking will ensure that a container operates on an +unlocked encrypted directory and never on directory that has fscrypt enabled. + +### Key Management + +```mermaid +graph LR + vault[Vault, Default] -->|data encryption key| csi_kms_integrated + secrets[Passphrase
K8s secrets, AWS, IBM Key Protect] -->|passphrase| csi_kms_metadata + + + subgraph ceph_csi_kms[Ceph CSI KMS] + csi_kms_metadata['metadata' type DEK store] + csi_kms_integrated['integrated' type DEK store] + + end + + + subgraph protector[fscrypt protector] + custom_passphrase[CustomPasswordSource]; + raw_passphrase[RawKeySource]; + + protector_metadata[(/.fscrypt/protectors)]; + + protector_metadata -->|wrapped key| custom_passphrase; + protector_metadata -->|wrapped key| raw_passphrase; + + end + + subgraph policy[fscrypt policy] + policy_unwrap + policy_metadata[(/.fscrypt/policies)] + + policy_metadata -->|wrapped key| policy_unwrap + end + + + csi_kms_metadata -->|DEK| custom_passphrase + csi_kms_integrated -->|DEK| raw_passphrase + + custom_passphrase -->|protector key| policy_unwrap + raw_passphrase -->|protector key| policy_unwrap + + policy_unwrap -->|policy key| kernel[Kernel API] + +``` + +The diagram shows the keys flowing from Ceph CSI to the Kernel API unlocking +a directory. On the way, key material from Ceph CSI passes two key derivation +steps in fscrypt: + +- *protectors* and +- *policies* + +`fscrypt` supports multiple *protectors*. These may source secrets from login +passwords, custom passwords or soon Ceph CSI. Unlocking a protector +yields a *protector key* that is then used to unlock a *policy*. + +A *policy* may unlock multiple directories. In our case there will be +only a single policy for a single well-known directory on the subvolume root. +A policy is used to derive a *policy key*, which is passed to the Kernel API +along with other settings, such as the desired encryption algorithm. + +Going back to the beginning of the diagram and looking at the interface between +`fscrypt` and Ceph CSI one can see that the two data encryption key (DEK) +styles (*metadata* and *integrated*) map to different `fscrypt` protectors. + +The `fscrypt` protector of key sources `CustomPasswordSource` and `RawKeySource` +differ in how they derive a key from a source. Refer to the `fscrypt` design +doc for details. + +Metadata DEKs: In the RBD case, Ceph CSI stores a wrapped key in the +RBD volume metadata and then a user configured secret (for example, a Kubernetes +secret) is passed to a key derivation function (KDF) to then unwrap the +key. The resulting key unlocks the volume. + +Since `fscrypt` already stores wrapped keys there is no need for an +extra layer of wrapping. We can also skip the KDF and use a +`CustomPasswordSource` to pass the Ceph CSI secret directly to +`fscrypt`. + +With integrated DEKs (for example, Vault) Ceph CSI uses a key from a KMS +directly. To integrate this with `fscrypt` we use a +`RawKeySource`, that is similar to a `CustomPasswordSource`, but skips +the KDF. + +As the diagram shows, both policies and protectors require a metadata +store. The default `fscrypt` data store is in a `/.fscrypt` +directory under a filesystem root. The `fscrypt` design doc details +alternatives and explains what data is stored. + +To be compatible with `fscrypt`, this directory requires support as +well. The downside of this is that we lose the CephFS subvolume root to +metadata and encrypted data will reside under a well-known +subdirectory (for example, `/ceph-csi-encrypted`). + +## Dependencies + +The proposed change is tailored to CephFS and requires CephFS support +to work *with* CephFS. The kernel APIs however are not specific to +CephFS and are unlikely to change as they only deal with configuration +and key management. There is no direct dependency on CephFS. Using the +proposed features will simply fail at runtime, when neither Ceph nor +the Kernel have the appropriate support. At build time this feature +does not require CephFS fscrypt support. + +Runtime dependencies: + +- Kernel >= v5.4 with `CONFIG_FS_ENCRYPTION=y` +- CephFS kernel client fscrypt support [Ceph Feature Tracker](https://tracker.ceph.com/issues/46690) + +Build dependencies: + +- `google/fscrypt` library, which has minimal build dependencies + ([fscrypt doc](https://github.com/google/fscrypt#building-and-installing)) + +## Alternatives + +### [ceph-csi-kms] Key Management: Policy Key Directly From Ceph CSI + +A simpler approach to the one proposed above, but incompatible with +`fscrypt`. To unlock a subvolume, the user would have to use Ceph CSI. +The implementation is similar to the RBD encryption feature. +It uses the low-level `fscryptctl` tool to set a policy key +from a Ceph CSI data encryption key. + +Ceph CSI KMS requires metadata data encryption key storage that can use +xattrs on a mounted CephFS filesystem. + +A prototype showing this approach is available: [repository](https://github.com/irq0/ceph-csi/tree/wip/fscrypt) + +Benefits: + +- Simpler key wrapping +- No `/.fscrypt` on the subvolume root + +Drawbacks: + +- `fscryptctl` is a C tool and does not lend itself to be integrated into Ceph CSI +- Incompatible with `fscrypt` +- Does not support unlocking with any of possibly multiple keys + configured (`fscrypt` protectors feature) + +### [manual] Manual Setup + +For completeness, a user may set up FSCrypt without any support in +Ceph CSI. Both `fscrypt` and `fscryptctl` work in containers +and may even be used with the proposed change or alternative [ceph-csi-kms]. + +The following links provide examples from the documentation that also apply to +CephFS: + +- [fscrypt example usage](https://github.com/google/fscrypt#example-usage) +- [fscryptctl example usage](https://github.com/google/fscryptctl#example-usage) + +### [subdirs] Support Unlocking Arbitrary subdirs (instead of subvolume basis) + +An extension to the proposal: As mentioned in the implementation section, a +`fscrypt` policy may apply to multiple directories and from a set of protectors +any suffices to unlock a policy. A user may configure a complex mapping of +subdirectories and Ceph CSI secret sources to unlock different parts of a CephFS +subvolume with different keys.