# Ceph Filesystem fscrypt Support

## Problem Description

As a Ceph Container Storage Interface (Ceph CSI) user, I want a cloud native way
to manage keys and enable encryption on Ceph Filesystem (CephFS) volumes.

In order to access encrypted volumes without Ceph CSI, this can be done by unlocking
volumes with user space tools.

## Background

*fscrypt* or *FSCrypt* is a Linux Kernel feature that allows the filesystem
to support the transparent encryption of files and directories. Local
filesystems like ext4 and F2FS (Flash-Friendly File System) support this feature
already.

Work is in progress to add fscrypt support to CephFS for filesystem-level encryption.

- [FSCrypt Kernel Documentation](https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html)
- Management Tools
   - [`fscrypt`](https://github.com/google/fscrypt)
   - [`fscryptctl`](https://github.com/google/fscryptctl)
- [Ceph Feature Tracker: "Add fscrypt support to the kernel CephFS client"](https://tracker.ceph.com/issues/46690)
- [`fscrypt` design document](https://goo.gl/55cCrI)

**NOTE:** In this document, fscrypt refers to the filesystem-level encryption
feature, while `fscrypt` specifically refers to the user space tool.

## Terminology

- *FSCrypt*, *fscrypt* - Linux Kernel filesystem-level encryption feature
- `fscrypt` (code formatted) - User space tool manage keys and encryption policies
- `fscryptctl` (code formatted) - Low-level user space tool manage
  keys and encryption policies
- *subvolume* - CephFS subvolume
- *unlocking* - Using a key to make an encrypted filesystem
  accessible in plain text
- *protector* (fscrypt) - A single method or secret plus data used to
  derive a *protector key*. Example: user login passphrase
- *protector key* (fscrypt) - A symmetric key derived from an external
  source. Used by a *policy* to unwrap a *policy key*
- *policy key* (fscrypt) - An encryption key passed to the kernel to
  unlock a directory
- *policy* (fscrypt) - A collection of directories protected and unlocked as a unit
- *KMS* - Key management system

## User Visible Change

Similar to the existing RADOS Block Device (RBD) encryption support,
we propose adding encryption support in the configuration and Key
Management Service (KMS) integration.

In this example, a user may enable encryption using storage class keys similar
to RBD. Ceph CSI then configures and unlocks the persistent volumes and CephFS
subvolumes.

Due to the way `fscrypt` stores metadata, subvolumes have a regular root
directory containing  a `/.fscrypt` directory and a
`/ceph-csi-encrypted` directory. The first contains `fscrypt` metadata; the
latter is the fscrypt-enabled directory made that is accessible to pods.

Example configuration using a secrets-based KMS:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: cephfs-storage-encryption-secret
stringData:
  encryptionPassphrase: verysecretpassword
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc-encrypted
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>
  fsName: cephfs

  encrypted: "true"
  encryptionKMSID: "user-ns-secrets-metadata"

  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug
```

The change will leverage the existing Ceph CSI KMS and support any
integration now available to RBD encryption

## Implementation

We suggest to leverage the encryption features in Ceph CSI and integrate
that with `fscrypt`, a Go tool, and the library for key management and configuration
of the fscrypt kernel feature.

Ceph CSI and `fscrypt` have a lot of overlap between their key management
features. The Key Management section will go into detail on how and where keys
are managed.

- Ceph CSI provides the user facing configuration and access to key
  management systems
- `fscrypt` handles key derivation, storage of wrapped keys and metadata

The current CephFS subvolume root will remain untouched with the exception that
the subvolume root is not bind mounted into the pod, but rather a well-known
subdirectory. The root will contain a `/.fscrypt` directory managed by `fscrypt`.

`fscrypt` requires access to a mounted filesystem and therefore the encryption setup
must take place in the `NodeStageVolume` request handler instead of `CreateVolume`.
This is the same case for RBD. The set up will take place right between
subvolume mount and bind mount to the container namespace.

Additional checks after unlocking will ensure that a container operates on an
unlocked encrypted directory and never on directory that has fscrypt enabled.

### Key Management

```mermaid
graph LR
    vault[Vault, Default] -->|data encryption key| csi_kms_integrated
    secrets[Passphrase<br>K8s secrets, AWS, IBM Key Protect] -->|passphrase| csi_kms_metadata


    subgraph ceph_csi_kms[Ceph CSI KMS]
        csi_kms_metadata['metadata' type DEK store]
        csi_kms_integrated['integrated' type DEK store]

    end


    subgraph protector[fscrypt protector]
        custom_passphrase[CustomPasswordSource];
        raw_passphrase[RawKeySource];

        protector_metadata[(/.fscrypt/protectors)];

        protector_metadata -->|wrapped key| custom_passphrase;
        protector_metadata -->|wrapped key| raw_passphrase;

    end

    subgraph policy[fscrypt policy]
        policy_unwrap
        policy_metadata[(/.fscrypt/policies)]

        policy_metadata -->|wrapped key| policy_unwrap
    end


    csi_kms_metadata -->|DEK| custom_passphrase
    csi_kms_integrated -->|DEK| raw_passphrase

    custom_passphrase -->|protector key| policy_unwrap
    raw_passphrase -->|protector key| policy_unwrap

    policy_unwrap -->|policy key| kernel[Kernel API]

```

The diagram shows the keys flowing from Ceph CSI to the Kernel API unlocking
a directory. On the way, key material from Ceph CSI passes two key derivation
steps in fscrypt:

- *protectors* and
- *policies*

`fscrypt` supports multiple *protectors*. These may source secrets from login
passwords, custom passwords or soon Ceph CSI. Unlocking a protector
yields a *protector key* that is then used to unlock a *policy*.

A *policy* may unlock multiple directories. In our case there will be
only a single policy for a single well-known directory on the subvolume root.
A policy is used to derive a *policy key*, which is passed to the Kernel API
along with other settings, such as the desired encryption algorithm.

Going back to the beginning of the diagram and looking at the interface between
`fscrypt` and Ceph CSI one can see that the two data encryption key (DEK)
styles (*metadata* and *integrated*) map to different `fscrypt` protectors.

The `fscrypt` protector of key sources `CustomPasswordSource` and `RawKeySource`
differ in how they derive a key from a source. Refer to the `fscrypt` design
doc for details.

Metadata DEKs: In the RBD case, Ceph CSI stores a wrapped key in the
RBD volume metadata and then a user configured secret (for example, a Kubernetes
secret) is passed to a key derivation function (KDF) to then unwrap the
key. The resulting key unlocks the volume.

Since `fscrypt` already stores wrapped keys there is no need for an
extra layer of wrapping. We can also skip the KDF and use a
`CustomPasswordSource` to pass the Ceph CSI secret directly to
`fscrypt`.

With integrated DEKs (for example, Vault) Ceph CSI uses a key from a KMS
directly. To integrate this with `fscrypt` we use a
`RawKeySource`, that is similar to a `CustomPasswordSource`, but skips
the KDF.

As the diagram shows, both policies and protectors require a metadata
store. The default `fscrypt` data store is in a `/.fscrypt`
directory under a filesystem root. The `fscrypt` design doc details
alternatives and explains what data is stored.

To be compatible with `fscrypt`, this directory requires support as
well. The downside of this is that we lose the CephFS subvolume root to
metadata and encrypted data will reside under a well-known
subdirectory (for example, `/ceph-csi-encrypted`).

## Dependencies

The proposed change is tailored to CephFS and requires CephFS support
to work *with* CephFS. The kernel APIs however are not specific to
CephFS and are unlikely to change as they only deal with configuration
and key management. There is no direct dependency on CephFS. Using the
proposed features will simply fail at runtime, when neither Ceph nor
the Kernel have the appropriate support. At build time this feature
does not require CephFS fscrypt support.

Runtime dependencies:

- Kernel >= v5.4 with `CONFIG_FS_ENCRYPTION=y`
- CephFS kernel client fscrypt support [Ceph Feature Tracker](https://tracker.ceph.com/issues/46690)

Build dependencies:

- `google/fscrypt` library, which has minimal build dependencies
  ([fscrypt doc](https://github.com/google/fscrypt#building-and-installing))

## Alternatives

### [ceph-csi-kms] Key Management: Policy Key Directly From Ceph CSI

A simpler approach to the one proposed above, but incompatible with
`fscrypt`. To unlock a subvolume, the user would have to use Ceph CSI.
The implementation is similar to the RBD encryption feature.
It uses the low-level `fscryptctl` tool to set a policy key
from a Ceph CSI data encryption key.

Ceph CSI KMS requires metadata data encryption key storage that can use
xattrs on a mounted CephFS filesystem.

A prototype showing this approach is available: [repository](https://github.com/irq0/ceph-csi/tree/wip/fscrypt)

Benefits:

- Simpler key wrapping
- No `/.fscrypt` on the subvolume root

Drawbacks:

- `fscryptctl` is a C tool and does not lend itself to be integrated into Ceph CSI
- Incompatible with `fscrypt`
- Does not support unlocking with any of possibly multiple keys
  configured (`fscrypt` protectors feature)

### [manual] Manual Setup

For completeness, a user may set up FSCrypt without any support in
Ceph CSI. Both `fscrypt` and `fscryptctl` work in containers
and may even be used with the proposed change or alternative [ceph-csi-kms].

The following links provide examples from the documentation that also apply to
CephFS:

- [fscrypt example usage](https://github.com/google/fscrypt#example-usage)
- [fscryptctl example usage](https://github.com/google/fscryptctl#example-usage)

### [subdirs] Support Unlocking Arbitrary subdirs (instead of subvolume basis)

An extension to the proposal: As mentioned in the implementation section, a
`fscrypt` policy may apply to multiple directories and from a set of protectors
any suffices to unlock a policy. A user may configure a complex mapping of
subdirectories and Ceph CSI secret sources to unlock different parts of a CephFS
subvolume with different keys.