doc: proposal for KMS with ServiceAccount per Tenant

A new KMS that supports Hashicorp Vault with the Kubernetes Auth backend
and ServiceAccounts per Tenant (Kubernetes Namespace).

Updates: #2222
Signed-off-by: Niels de Vos <ndevos@redhat.com>

docs/design/proposals/encryption-with-vault-sa.md

@@ -0,0 +1,83 @@
+# Encrypted Volumes with a ServiceAccount per Tenant
+
+## Requirements
+
+- Hashicorp Vault with Kubernetes Auth
+- each Tenant (=Namespace) has a ServiceAccount to connect to Vault
+- each Tenant can override Vault configuration options (like `BackendPath`)
+
+## Similarities with available KMS implementations
+
+Ceph-CSI provides several KMS implementations with different functionalities.
+Two implementations use Hashicorp Vault and could be used as a base for a new
+KMS implementation. Or, if changes would be minimal, a configuration option to
+one of the implementations can be added.
+
+Different KMS implementations and their configurable options can be found at
+[`csi-kms-connection-details.yaml`](../../../examples/kms/vault/csi-kms-connection-details.yaml).
+
+### VaultTokensKMS
+
+- global configuration in the Namespace where Ceph-CSI is deployed
+- optional configuration per Tenant, allows for overriding Vault options
+- fetches a Secret from the Tenants (volume owner) namespace
+
+An example of the per Tenant configuration options are in
+[`tenant-config.yaml`](../../../examples/kms/vault/tenant-config.yaml) and
+[`tenant-token.yaml`](../../../examples/kms/vault/tenant-token.yaml).
+
+Implementation is in [`vault_tokens.go`](../../../internal/util/vault_tokens.go).
+
+### Vault
+
+- uses Kubernetes Auth with single service account (aka storage admin account)
+
+Implementation is in [`vault.go`](../../../internal/util/vault.go).
+
+## Extension or New KMS implementation
+
+Normally ServiceAccounts are provided by Kubernetes in the containers
+filesystem. This only allows a single ServiceAccount and is static for the
+lifetime of the Pod. Ceph-CSI runs in the namespace of the storage
+administrator, and has access to the single ServiceAccount linked in the
+deployments of the Pods.
+
+For Ceph-CSI to dynamically use ServiceAccounts from tenants, the following
+steps need to be taken:
+
+- use the Namespace of the volume to identify the Tenant
+- identify the right ServiceAccount in the Tenants Namespace
+- read and parse the ServiceAccount contents
+- provide the ServiceAccount contents as a (temporary) directory structure
+- setup the Vault connection to use the contents from the ServiceAccount to
+  replace the default (`AuthKubernetesTokenPath:
+  /var/run/secrets/kubernetes.io/serviceaccount/token`)
+
+Currently the Ceph-CSI components may read Secrets and ConfigMaps from the
+Tenants namespace. These permissions need to be extended to allow Ceph-CSI to
+read the contents of the ServiceAccount(s) in the Tenants namespace.
+
+## High-Level Design
+
+### Global Configuration
+
+1. a StorageClass links to a KMS configuration by providing the `kmsID` parameter
+1. a ConfigMap in the namespace of the Ceph-CSI deployment contains the KMS
+   configuration for the `kmsID`
+   ([`csi-kms-connection-details.yaml`](../../../examples/kms/vault/csi-kms-connection-details.yaml))
+
+When Ceph-CSI receives a `CreateVolume` request, the parameters from the
+StorageClass and details about the requested Volume are available. The Ceph-CSI
+provisioner checks the `kmsID` parameter and fetches the matching KMS
+configuration from the ConfigMap.
+
+### Tenant Configuration
+
+1. needs ServiceAccount with a known name with permissions to connect to Vault
+1. optional ConfigMap with options for Vault that override default settings
+
+A `CreateVolume` request contains the owner (Namespace) of the Volume.
+The KMS configuration indicates that additional attributes need to be fetched
+from the Tenants namespace, so the provisioner will fetch these. The additional
+configuration and ServiceAccount are merged in the provisioners configuration
+for the KMS-implementation while creating the volume.