mirror of
https://github.com/ceph/ceph-csi.git
synced 2024-11-17 20:00:23 +00:00
doc: add proposal for CephFS fscrypt integration
Add proposal document covering key management integration of Ceph CSI and https://github.com/google/fscrypt Updates: #1563 Signed-off-by: Marcel Lauhoff <marcel.lauhoff@suse.com>
This commit is contained in:
Marcel Lauhoff
mergify[bot]
parent
60281fab40
commit
b7ec0b2d66
284
docs/design/proposals/cephfs-fscrypt.md
Normal file
284
docs/design/proposals/cephfs-fscrypt.md
Normal file
@ -0,0 +1,284 @@
|
||||
# Ceph Filesystem fscrypt Support
|
||||
|
||||
## Problem Description
|
||||
|
||||
As a Ceph Container Storage Interface (Ceph CSI) user, I want a cloud native way
|
||||
to manage keys and enable encryption on Ceph Filesystem (CephFS) volumes.
|
||||
|
||||
In order to access encrypted volumes without Ceph CSI, this can be done by unlocking
|
||||
volumes with user space tools.
|
||||
|
||||
## Background
|
||||
|
||||
*fscrypt* or *FSCrypt* is a Linux Kernel feature that allows the filesystem
|
||||
to support the transparent encryption of files and directories. Local
|
||||
filesystems like ext4 and F2FS (Flash-Friendly File System) support this feature
|
||||
already.
|
||||
|
||||
Work is in progress to add fscrypt support to CephFS for filesystem-level encryption.
|
||||
|
||||
- [FSCrypt Kernel Documentation](https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html)
|
||||
- Management Tools
|
||||
- [`fscrypt`](https://github.com/google/fscrypt)
|
||||
- [`fscryptctl`](https://github.com/google/fscryptctl)
|
||||
- [Ceph Feature Tracker: "Add fscrypt support to the kernel CephFS client"](https://tracker.ceph.com/issues/46690)
|
||||
- [`fscrypt` design document](https://goo.gl/55cCrI)
|
||||
|
||||
**NOTE:** In this document, fscrypt refers to the filesystem-level encryption
|
||||
feature, while `fscrypt` specifically refers to the user space tool.
|
||||
|
||||
## Terminology
|
||||
|
||||
- *FSCrypt*, *fscrypt* - Linux Kernel filesystem-level encryption feature
|
||||
- `fscrypt` (code formatted) - User space tool manage keys and encryption policies
|
||||
- `fscryptctl` (code formatted) - Low-level user space tool manage
|
||||
keys and encryption policies
|
||||
- *subvolume* - CephFS subvolume
|
||||
- *unlocking* - Using a key to make an encrypted filesystem
|
||||
accessible in plain text
|
||||
- *protector* (fscrypt) - A single method or secret plus data used to
|
||||
derive a *protector key*. Example: user login passphrase
|
||||
- *protector key* (fscrypt) - A symmetric key derived from an external
|
||||
source. Used by a *policy* to unwrap a *policy key*
|
||||
- *policy key* (fscrypt) - An encryption key passed to the kernel to
|
||||
unlock a directory
|
||||
- *policy* (fscrypt) - A collection of directories protected and unlocked as a unit
|
||||
- *KMS* - Key management system
|
||||
|
||||
## User Visible Change
|
||||
|
||||
Similar to the existing RADOS Block Device (RBD) encryption support,
|
||||
we propose adding encryption support in the configuration and Key
|
||||
Management Service (KMS) integration.
|
||||
|
||||
In this example, a user may enable encryption using storage class keys similar
|
||||
to RBD. Ceph CSI then configures and unlocks the persistent volumes and CephFS
|
||||
subvolumes.
|
||||
|
||||
Due to the way `fscrypt` stores metadata, subvolumes have a regular root
|
||||
directory containing a `/.fscrypt` directory and a
|
||||
`/ceph-csi-encrypted` directory. The first contains `fscrypt` metadata; the
|
||||
latter is the fscrypt-enabled directory made that is accessible to pods.
|
||||
|
||||
Example configuration using a secrets-based KMS:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cephfs-storage-encryption-secret
|
||||
stringData:
|
||||
encryptionPassphrase: verysecretpassword
|
||||
---
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: csi-cephfs-sc-encrypted
|
||||
provisioner: cephfs.csi.ceph.com
|
||||
parameters:
|
||||
clusterID: <cluster-id>
|
||||
fsName: cephfs
|
||||
|
||||
encrypted: "true"
|
||||
encryptionKMSID: "user-ns-secrets-metadata"
|
||||
|
||||
csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: default
|
||||
csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: default
|
||||
csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
|
||||
csi.storage.k8s.io/node-stage-secret-namespace: default
|
||||
reclaimPolicy: Delete
|
||||
allowVolumeExpansion: true
|
||||
mountOptions:
|
||||
- debug
|
||||
```
|
||||
|
||||
The change will leverage the existing Ceph CSI KMS and support any
|
||||
integration now available to RBD encryption
|
||||
|
||||
## Implementation
|
||||
|
||||
We suggest to leverage the encryption features in Ceph CSI and integrate
|
||||
that with `fscrypt`, a Go tool, and the library for key management and configuration
|
||||
of the fscrypt kernel feature.
|
||||
|
||||
Ceph CSI and `fscrypt` have a lot of overlap between their key management
|
||||
features. The Key Management section will go into detail on how and where keys
|
||||
are managed.
|
||||
|
||||
- Ceph CSI provides the user facing configuration and access to key
|
||||
management systems
|
||||
- `fscrypt` handles key derivation, storage of wrapped keys and metadata
|
||||
|
||||
The current CephFS subvolume root will remain untouched with the exception that
|
||||
the subvolume root is not bind mounted into the pod, but rather a well-known
|
||||
subdirectory. The root will contain a `/.fscrypt` directory managed by `fscrypt`.
|
||||
|
||||
`fscrypt` requires access to a mounted filesystem and therefore the encryption setup
|
||||
must take place in the `NodeStageVolume` request handler instead of `CreateVolume`.
|
||||
This is the same case for RBD. The set up will take place right between
|
||||
subvolume mount and bind mount to the container namespace.
|
||||
|
||||
Additional checks after unlocking will ensure that a container operates on an
|
||||
unlocked encrypted directory and never on directory that has fscrypt enabled.
|
||||
|
||||
### Key Management
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
vault[Vault, Default] -->|data encryption key| csi_kms_integrated
|
||||
secrets[Passphrase<br>K8s secrets, AWS, IBM Key Protect] -->|passphrase| csi_kms_metadata
|
||||
|
||||
|
||||
subgraph ceph_csi_kms[Ceph CSI KMS]
|
||||
csi_kms_metadata['metadata' type DEK store]
|
||||
csi_kms_integrated['integrated' type DEK store]
|
||||
|
||||
end
|
||||
|
||||
|
||||
subgraph protector[fscrypt protector]
|
||||
custom_passphrase[CustomPasswordSource];
|
||||
raw_passphrase[RawKeySource];
|
||||
|
||||
protector_metadata[(/.fscrypt/protectors)];
|
||||
|
||||
protector_metadata -->|wrapped key| custom_passphrase;
|
||||
protector_metadata -->|wrapped key| raw_passphrase;
|
||||
|
||||
end
|
||||
|
||||
subgraph policy[fscrypt policy]
|
||||
policy_unwrap
|
||||
policy_metadata[(/.fscrypt/policies)]
|
||||
|
||||
policy_metadata -->|wrapped key| policy_unwrap
|
||||
end
|
||||
|
||||
|
||||
csi_kms_metadata -->|DEK| custom_passphrase
|
||||
csi_kms_integrated -->|DEK| raw_passphrase
|
||||
|
||||
custom_passphrase -->|protector key| policy_unwrap
|
||||
raw_passphrase -->|protector key| policy_unwrap
|
||||
|
||||
policy_unwrap -->|policy key| kernel[Kernel API]
|
||||
|
||||
```
|
||||
|
||||
The diagram shows the keys flowing from Ceph CSI to the Kernel API unlocking
|
||||
a directory. On the way, key material from Ceph CSI passes two key derivation
|
||||
steps in fscrypt:
|
||||
|
||||
- *protectors* and
|
||||
- *policies*
|
||||
|
||||
`fscrypt` supports multiple *protectors*. These may source secrets from login
|
||||
passwords, custom passwords or soon Ceph CSI. Unlocking a protector
|
||||
yields a *protector key* that is then used to unlock a *policy*.
|
||||
|
||||
A *policy* may unlock multiple directories. In our case there will be
|
||||
only a single policy for a single well-known directory on the subvolume root.
|
||||
A policy is used to derive a *policy key*, which is passed to the Kernel API
|
||||
along with other settings, such as the desired encryption algorithm.
|
||||
|
||||
Going back to the beginning of the diagram and looking at the interface between
|
||||
`fscrypt` and Ceph CSI one can see that the two data encryption key (DEK)
|
||||
styles (*metadata* and *integrated*) map to different `fscrypt` protectors.
|
||||
|
||||
The `fscrypt` protector of key sources `CustomPasswordSource` and `RawKeySource`
|
||||
differ in how they derive a key from a source. Refer to the `fscrypt` design
|
||||
doc for details.
|
||||
|
||||
Metadata DEKs: In the RBD case, Ceph CSI stores a wrapped key in the
|
||||
RBD volume metadata and then a user configured secret (for example, a Kubernetes
|
||||
secret) is passed to a key derivation function (KDF) to then unwrap the
|
||||
key. The resulting key unlocks the volume.
|
||||
|
||||
Since `fscrypt` already stores wrapped keys there is no need for an
|
||||
extra layer of wrapping. We can also skip the KDF and use a
|
||||
`CustomPasswordSource` to pass the Ceph CSI secret directly to
|
||||
`fscrypt`.
|
||||
|
||||
With integrated DEKs (for example, Vault) Ceph CSI uses a key from a KMS
|
||||
directly. To integrate this with `fscrypt` we use a
|
||||
`RawKeySource`, that is similar to a `CustomPasswordSource`, but skips
|
||||
the KDF.
|
||||
|
||||
As the diagram shows, both policies and protectors require a metadata
|
||||
store. The default `fscrypt` data store is in a `/.fscrypt`
|
||||
directory under a filesystem root. The `fscrypt` design doc details
|
||||
alternatives and explains what data is stored.
|
||||
|
||||
To be compatible with `fscrypt`, this directory requires support as
|
||||
well. The downside of this is that we lose the CephFS subvolume root to
|
||||
metadata and encrypted data will reside under a well-known
|
||||
subdirectory (for example, `/ceph-csi-encrypted`).
|
||||
|
||||
## Dependencies
|
||||
|
||||
The proposed change is tailored to CephFS and requires CephFS support
|
||||
to work *with* CephFS. The kernel APIs however are not specific to
|
||||
CephFS and are unlikely to change as they only deal with configuration
|
||||
and key management. There is no direct dependency on CephFS. Using the
|
||||
proposed features will simply fail at runtime, when neither Ceph nor
|
||||
the Kernel have the appropriate support. At build time this feature
|
||||
does not require CephFS fscrypt support.
|
||||
|
||||
Runtime dependencies:
|
||||
|
||||
- Kernel >= v5.4 with `CONFIG_FS_ENCRYPTION=y`
|
||||
- CephFS kernel client fscrypt support [Ceph Feature Tracker](https://tracker.ceph.com/issues/46690)
|
||||
|
||||
Build dependencies:
|
||||
|
||||
- `google/fscrypt` library, which has minimal build dependencies
|
||||
([fscrypt doc](https://github.com/google/fscrypt#building-and-installing))
|
||||
|
||||
## Alternatives
|
||||
|
||||
### [ceph-csi-kms] Key Management: Policy Key Directly From Ceph CSI
|
||||
|
||||
A simpler approach to the one proposed above, but incompatible with
|
||||
`fscrypt`. To unlock a subvolume, the user would have to use Ceph CSI.
|
||||
The implementation is similar to the RBD encryption feature.
|
||||
It uses the low-level `fscryptctl` tool to set a policy key
|
||||
from a Ceph CSI data encryption key.
|
||||
|
||||
Ceph CSI KMS requires metadata data encryption key storage that can use
|
||||
xattrs on a mounted CephFS filesystem.
|
||||
|
||||
A prototype showing this approach is available: [repository](https://github.com/irq0/ceph-csi/tree/wip/fscrypt)
|
||||
|
||||
Benefits:
|
||||
|
||||
- Simpler key wrapping
|
||||
- No `/.fscrypt` on the subvolume root
|
||||
|
||||
Drawbacks:
|
||||
|
||||
- `fscryptctl` is a C tool and does not lend itself to be integrated into Ceph CSI
|
||||
- Incompatible with `fscrypt`
|
||||
- Does not support unlocking with any of possibly multiple keys
|
||||
configured (`fscrypt` protectors feature)
|
||||
|
||||
### [manual] Manual Setup
|
||||
|
||||
For completeness, a user may set up FSCrypt without any support in
|
||||
Ceph CSI. Both `fscrypt` and `fscryptctl` work in containers
|
||||
and may even be used with the proposed change or alternative [ceph-csi-kms].
|
||||
|
||||
The following links provide examples from the documentation that also apply to
|
||||
CephFS:
|
||||
|
||||
- [fscrypt example usage](https://github.com/google/fscrypt#example-usage)
|
||||
- [fscryptctl example usage](https://github.com/google/fscryptctl#example-usage)
|
||||
|
||||
### [subdirs] Support Unlocking Arbitrary subdirs (instead of subvolume basis)
|
||||
|
||||
An extension to the proposal: As mentioned in the implementation section, a
|
||||
`fscrypt` policy may apply to multiple directories and from a set of protectors
|
||||
any suffices to unlock a policy. A user may configure a complex mapping of
|
||||
subdirectories and Ceph CSI secret sources to unlock different parts of a CephFS
|
||||
subvolume with different keys.
|
||||
Loading…
Reference in New Issue
Block a user