ceph-csi/docs/design/proposals/cephfs-fscrypt.md

Ceph Filesystem fscrypt Support

Problem Description

As a Ceph Container Storage Interface (Ceph CSI) user, I want a cloud native way to manage keys and enable encryption on Ceph Filesystem (CephFS) volumes.

In order to access encrypted volumes without Ceph CSI, this can be done by unlocking volumes with user space tools.

Background

fscrypt or FSCrypt is a Linux Kernel feature that allows the filesystem to support the transparent encryption of files and directories. Local filesystems like ext4 and F2FS (Flash-Friendly File System) support this feature already.

Work is in progress to add fscrypt support to CephFS for filesystem-level encryption.

• FSCrypt Kernel Documentation
• Management Tools
  • fscrypt
  • fscryptctl
• Ceph Feature Tracker: "Add fscrypt support to the kernel CephFS client"
• fscrypt design document

NOTE: In this document, fscrypt refers to the filesystem-level encryption feature, while fscrypt specifically refers to the user space tool.

Terminology

• FSCrypt, fscrypt - Linux Kernel filesystem-level encryption feature
• fscrypt (code formatted) - User space tool manage keys and encryption policies
• fscryptctl (code formatted) - Low-level user space tool manage keys and encryption policies
• subvolume - CephFS subvolume
• unlocking - Using a key to make an encrypted filesystem accessible in plain text
• protector (fscrypt) - A single method or secret plus data used to derive a protector key. Example: user login passphrase
• protector key (fscrypt) - A symmetric key derived from an external source. Used by a policy to unwrap a policy key
• policy key (fscrypt) - An encryption key passed to the kernel to unlock a directory
• policy (fscrypt) - A collection of directories protected and unlocked as a unit
• KMS - Key management system

User Visible Change

Similar to the existing RADOS Block Device (RBD) encryption support, we propose adding encryption support in the configuration and Key Management Service (KMS) integration.

In this example, a user may enable encryption using storage class keys similar to RBD. Ceph CSI then configures and unlocks the persistent volumes and CephFS subvolumes.

Due to the way fscrypt stores metadata, subvolumes have a regular root directory containing a /.fscrypt directory and a /ceph-csi-encrypted directory. The first contains fscrypt metadata; the latter is the fscrypt-enabled directory made that is accessible to pods.

Example configuration using a secrets-based KMS:

apiVersion: v1
kind: Secret
metadata:
  name: cephfs-storage-encryption-secret
stringData:
  encryptionPassphrase: verysecretpassword
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc-encrypted
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>
  fsName: cephfs

  encrypted: "true"
  encryptionKMSID: "user-ns-secrets-metadata"

  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug

The change will leverage the existing Ceph CSI KMS and support any integration now available to RBD encryption

Implementation

We suggest to leverage the encryption features in Ceph CSI and integrate that with fscrypt, a Go tool, and the library for key management and configuration of the fscrypt kernel feature.

Ceph CSI and fscrypt have a lot of overlap between their key management features. The Key Management section will go into detail on how and where keys are managed.

• Ceph CSI provides the user facing configuration and access to key management systems
• fscrypt handles key derivation, storage of wrapped keys and metadata

The current CephFS subvolume root will remain untouched with the exception that the subvolume root is not bind mounted into the pod, but rather a well-known subdirectory. The root will contain a /.fscrypt directory managed by fscrypt.

fscrypt requires access to a mounted filesystem and therefore the encryption setup must take place in the NodeStageVolume request handler instead of CreateVolume. This is the same case for RBD. The set up will take place right between subvolume mount and bind mount to the container namespace.

Additional checks after unlocking will ensure that a container operates on an unlocked encrypted directory and never on directory that has fscrypt enabled.

Key Management

graph LR
    vault[Vault, Default] -->|data encryption key| csi_kms_integrated
    secrets[Passphrase<br>K8s secrets, AWS, IBM Key Protect] -->|passphrase| csi_kms_metadata


    subgraph ceph_csi_kms[Ceph CSI KMS]
        csi_kms_metadata['metadata' type DEK store]
        csi_kms_integrated['integrated' type DEK store]

    end


    subgraph protector[fscrypt protector]
        custom_passphrase[CustomPasswordSource];
        raw_passphrase[RawKeySource];

        protector_metadata[(/.fscrypt/protectors)];

        protector_metadata -->|wrapped key| custom_passphrase;
        protector_metadata -->|wrapped key| raw_passphrase;

    end

    subgraph policy[fscrypt policy]
        policy_unwrap
        policy_metadata[(/.fscrypt/policies)]

        policy_metadata -->|wrapped key| policy_unwrap
    end


    csi_kms_metadata -->|DEK| custom_passphrase
    csi_kms_integrated -->|DEK| raw_passphrase

    custom_passphrase -->|protector key| policy_unwrap
    raw_passphrase -->|protector key| policy_unwrap

    policy_unwrap -->|policy key| kernel[Kernel API]

The diagram shows the keys flowing from Ceph CSI to the Kernel API unlocking a directory. On the way, key material from Ceph CSI passes two key derivation steps in fscrypt:

• protectors and
• policies

fscrypt supports multiple protectors. These may source secrets from login passwords, custom passwords or soon Ceph CSI. Unlocking a protector yields a protector key that is then used to unlock a policy.

A policy may unlock multiple directories. In our case there will be only a single policy for a single well-known directory on the subvolume root. A policy is used to derive a policy key, which is passed to the Kernel API along with other settings, such as the desired encryption algorithm.

Going back to the beginning of the diagram and looking at the interface between fscrypt and Ceph CSI one can see that the two data encryption key (DEK) styles (metadata and integrated) map to different fscrypt protectors.

The fscrypt protector of key sources CustomPasswordSource and RawKeySource differ in how they derive a key from a source. Refer to the fscrypt design doc for details.

Metadata DEKs: In the RBD case, Ceph CSI stores a wrapped key in the RBD volume metadata and then a user configured secret (for example, a Kubernetes secret) is passed to a key derivation function (KDF) to then unwrap the key. The resulting key unlocks the volume.

Since fscrypt already stores wrapped keys there is no need for an extra layer of wrapping. We can also skip the KDF and use a CustomPasswordSource to pass the Ceph CSI secret directly to fscrypt.

With integrated DEKs (for example, Vault) Ceph CSI uses a key from a KMS directly. To integrate this with fscrypt we use a RawKeySource, that is similar to a CustomPasswordSource, but skips the KDF.

As the diagram shows, both policies and protectors require a metadata store. The default fscrypt data store is in a /.fscrypt directory under a filesystem root. The fscrypt design doc details alternatives and explains what data is stored.

To be compatible with fscrypt, this directory requires support as well. The downside of this is that we lose the CephFS subvolume root to metadata and encrypted data will reside under a well-known subdirectory (for example, /ceph-csi-encrypted).

Dependencies

The proposed change is tailored to CephFS and requires CephFS support to work with CephFS. The kernel APIs however are not specific to CephFS and are unlikely to change as they only deal with configuration and key management. There is no direct dependency on CephFS. Using the proposed features will simply fail at runtime, when neither Ceph nor the Kernel have the appropriate support. At build time this feature does not require CephFS fscrypt support.

Runtime dependencies:

• Kernel >= v5.4 with CONFIG_FS_ENCRYPTION=y
• CephFS kernel client fscrypt support Ceph Feature Tracker

Build dependencies:

• google/fscrypt library, which has minimal build dependencies (fscrypt doc)

Alternatives

[ceph-csi-kms] Key Management: Policy Key Directly From Ceph CSI

A simpler approach to the one proposed above, but incompatible with fscrypt. To unlock a subvolume, the user would have to use Ceph CSI. The implementation is similar to the RBD encryption feature. It uses the low-level fscryptctl tool to set a policy key from a Ceph CSI data encryption key.

Ceph CSI KMS requires metadata data encryption key storage that can use xattrs on a mounted CephFS filesystem.

A prototype showing this approach is available: repository

Benefits:

• Simpler key wrapping
• No /.fscrypt on the subvolume root

Drawbacks:

• fscryptctl is a C tool and does not lend itself to be integrated into Ceph CSI
• Incompatible with fscrypt
• Does not support unlocking with any of possibly multiple keys configured (fscrypt protectors feature)

[manual] Manual Setup

For completeness, a user may set up FSCrypt without any support in Ceph CSI. Both fscrypt and fscryptctl work in containers and may even be used with the proposed change or alternative [ceph-csi-kms].

The following links provide examples from the documentation that also apply to CephFS:

• fscrypt example usage
• fscryptctl example usage

[subdirs] Support Unlocking Arbitrary subdirs (instead of subvolume basis)

An extension to the proposal: As mentioned in the implementation section, a fscrypt policy may apply to multiple directories and from a set of protectors any suffices to unlock a policy. A user may configure a complex mapping of subdirectories and Ceph CSI secret sources to unlock different parts of a CephFS subvolume with different keys.