diff --git a/docs/design/proposals/encryption-with-vault-tokens.md b/docs/design/proposals/encryption-with-vault-tokens.md new file mode 100644 index 000000000..e245b5a3d --- /dev/null +++ b/docs/design/proposals/encryption-with-vault-tokens.md @@ -0,0 +1,187 @@ +# Multi-tenancy with Vault Tokens + +## Current Feature: Vault access with single token + +The current feature to [support Hashicorp Vault as a KMS is +documented](./encrypted-pvc.md): + +- Tenants can create PVCs that are encrypted with a unique key (Data Encryption + Key, DEK) +- the key to encrypt/decrypt the PVC is stored in Hashicorp Vault +- there is a single JSON Web Token (JWT), the Kubernetes ServiceAccount, to + access Vault +- enabling is done in the StorageClass, pointing to a KMS configuration section + in a JSON configuration file (Kubernetes ConfigMap) + +## High-Level Requirements + +The new feature should support multi-tenancy, so that each Tenant can use their +own Vault Token to access the Key Encryption Key (KEK) and fetch the Data +Encryption Key (DEK) for PVC encryption: + +- each tenant can manage their Vault Token +- to get the KEK from Vault, each Tenant should provide their personal Vault token +- Ceph-CSI should use the Vault Token from the Tenant to store the unique key + (DEK) for the PVC +- a Tenant is the owner of a Kubernetes Namespace + +## Restrictions to consider + +- Tenants can only configure their Kubernetes namespace, their token to access + Vault should be located in their namespace (as a Kubernetes Secret) +- Ceph-CSI can not directly access Kubernetes namespaces, as CSI is an + abstraction layer +- Ceph-CSI needs to talk to a service (sidecar) to access the Vault Token from + a Tenant +- the KMS configuration is a ConfigMap with name + `ceph-csi-encryption-kms-config` in the namespace where the Ceph-CSI pods are + running +- the KMS configuration is available for the Ceph-CSI pods at + `/etc/ceph-csi-encryption-kms-config/config.json` +- [example of the + configuration](https://github.com/ceph/ceph-csi/blob/master/examples/kms/vault/kms-config.yaml) + +## Dependencies + +- the name of the Kubernetes Secret needs to be known (configured in the KMS + config file) +- each Tenant (Kubernetes Namespace) should use the same name for the Secret + that contains the Vault token +- the CSI-provisioner sidecar needs to provide the name of the Tenant together + with the metadata of the PVC to the Ceph-CSI provisioner + +## Implementation Outline + +- when creating the PVC the Ceph-CSI provisioner needs to store the Kubernetes + Namespace of the PVC in its metadata + - stores the `csi.volume.owner` (name of Tenant) in the metadata of the + volume and sets it as `rbdVolume.Owner` +- the Ceph-CSI node-plugin needs to request the Vault Token in the NodeStage + CSI operation and create/get the key for the PVC +- the Ceph-CSI provisioner needs to request the Vault Token to delete the key + for the PVC in the VolumeDelete CSI operation + +### New sidecar: Fetching Tenant Tokens from Kubernetes + +CSI is an abstraction and should not communicate with Kubernetes or other +Container Orchestration systems directly. Therefore, it is needed to +communicate with a service that can provide the Vault Token from the Tenants. +This service can be provided by a sidecar, exposing an endpoint as a +UNIX-Domain-Socket to communicate through. + +This sidecar does not exist at the moment. There are other features within +Ceph-CSI that will benefit from this sidecar, e.g. Topology support. Because +Ceph-CSI already uses the Kubernetes API to fetch details from Kubernetes, it +should be acceptable to fetch the Vault Token configuration for the Tenants the +same way. + +The feature for a sidecar that provides access to the required information from +Kubernetes and other Container Orchestration frameworks is tracked in +[#1782](https://github.com/ceph/ceph-csi/issues/1782). + +## Configuration Details + +- the current KMS configuration file needs extensions for Vault Token support +- introduce a new KMS-type: VaultTokenKMS (the current VaultKMS uses a + Kubernetes ServiceAccount) +- configuration of the VaultTokenKMS can be very similar to VaultKMS for common + settings +- the configuration can override the defaults for each Tenant separately + - Vault Service connection details (address, TLS options, ...) + - name of the Kubernetes Secret that can be looked up per tenant +- the configuration points to a Kubernetes Secret per Tenant that contains the + Vault Token +- the configuration points to an optional Kubernetes ConfigMap per Tenant that + contains alternative connection options for the VaultTokenKMS service + +### Example of the KMS configuration file for Vault Tokens + +The configuration is available in the Ceph-CSI containers as +`/etc/ceph-csi-encryption-kms-config/config.json`: + +```json +{ + "vault-with-tokens": { + "encryptionKMSType": "vaulttokens", + "vaultAddress": "http://vault.default.svc.cluster.local:8200", + "vaultBackendPath": "secret/", + "vaultTLSServerName": "vault.default.svc.cluster.local", + "vaultCAFromSecret": "vault-ca", + "vaultCAVerify": "false", + "tenantConfigName": "ceph-csi-kms-config", + "tenantTokenName": "ceph-csi-kms-token", + "tenants": { + "my-app": { + "vaultAddress": "https://vault.example.com", + "vaultCAVerify": "true" + }, + "an-other-app": { + "tenantTokenName": "storage-encryption-token" + } + } + } +} +``` + +In the Kubernetes StorageClass, the `kmsID` should be set to +`vault-with-tokens` in order to select the above configuration. + +**Required options**: + +- `encryptionKMSType`: should be set to `vaulttokens` +- `vaultAddress`: should be set to the URL of the Vault service + +**Optional options**: + +- `vaultBackendPath`: defaults to `secret/` +- `vaultTLSServerName`: not used when unset +- `vaultCAFromSecret`: not used when unset +- `vaultCAVerify`: defaults to `true` +- `tenantConfigName`: the name of the Kubernetes ConfigMap that contains the + Vault connection configuration (can be overridden per Tenant, defaults to + `ceph-csi-kms-config`) +- `tenantTokenName`: the name of the Kubernetes Secret that contains the Vault + Token (can be overridden per Tenant, defaults to `ceph-csi-kms-token`) +- `tenants`: list of Tenants (Kubernetes Namespaces) with their connection + configuration that differs from the global parameters + +### Configuration stored in the Tenants Kubernetes Namespace + +The Vault Token needs to be configured per Tenant. Each Tenant can create, +modify or delete their own personal Token. The Token is stored in a Kubernetes +Secret in the Kubernetes Namespace where the PVC is created. + +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: ceph-csi-kms-token +stringData: + token: "sample_root_token_id" +``` + +The name `ceph-csi-kms-token` is the default, but can be changed by setting +`tenantTokenName` in the `/etc/ceph-csi-encryption-kms-config/config.json` +configuration file. + +In addition to the Vault Token that can be configured per Tenant, the +connection parameters to the Vault Service can be stored in the Tenants +Kubernetes Namespace as well. + +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: ceph-csi-kms-config +data: + vaultAddress: "https://vault.infosec.example.org" + vaultBackendPath: "secret/ceph-csi-encryption/" + vaultTLSServerName: "vault.infosec.example.org" + vaultCAFromSecret: "vault-infosec-ca" + vaultCAVerify: "true" +``` + +Only parameters with the `vault`-prefix may be changed in the Kubernetes +ConfigMap of the Tenant.