mirror of
https://github.com/ceph/ceph-csi.git
synced 2024-11-23 23:00:19 +00:00
253 lines
8.4 KiB
Markdown
253 lines
8.4 KiB
Markdown
# Multi-tenancy with Vault Tokens
|
|
|
|
## Current Feature: Vault access with single token
|
|
|
|
The current feature to [support Hashicorp Vault as a KMS is
|
|
documented](./encrypted-pvc.md):
|
|
|
|
- Tenants can create PVCs that are encrypted with a unique key (Data Encryption
|
|
Key, DEK)
|
|
- the key to encrypt/decrypt the PVC is stored in Hashicorp Vault
|
|
- there is a single JSON Web Token (JWT), the Kubernetes ServiceAccount, to
|
|
access Vault
|
|
- enabling is done in the StorageClass, pointing to a KMS configuration section
|
|
in a JSON configuration file (Kubernetes ConfigMap)
|
|
|
|
## High-Level Requirements
|
|
|
|
The new feature should support multi-tenancy, so that each Tenant can use their
|
|
own Vault Token to access the Key Encryption Key (KEK) and fetch the Data
|
|
Encryption Key (DEK) for PVC encryption:
|
|
|
|
- each tenant can manage their Vault Token
|
|
- to get the KEK from Vault, each Tenant should provide their personal Vault token
|
|
- Ceph-CSI should use the Vault Token from the Tenant to store the unique key
|
|
(DEK) for the PVC
|
|
- a Tenant is the owner of a Kubernetes Namespace
|
|
|
|
## Restrictions to consider
|
|
|
|
- Tenants can only configure their Kubernetes namespace, their token to access
|
|
Vault should be located in their namespace (as a Kubernetes Secret)
|
|
- Ceph-CSI can not directly access Kubernetes namespaces, as CSI is an
|
|
abstraction layer
|
|
- Ceph-CSI needs to talk to a service (sidecar) to access the Vault Token from
|
|
a Tenant
|
|
- the KMS configuration is a ConfigMap with name
|
|
`ceph-csi-encryption-kms-config` in the namespace where the Ceph-CSI pods are
|
|
running
|
|
- the KMS configuration is available for the Ceph-CSI pods at
|
|
`/etc/ceph-csi-encryption-kms-config/config.json`
|
|
- [example of the
|
|
configuration](https://github.com/ceph/ceph-csi/blob/devel/examples/kms/vault/kms-config.yaml)
|
|
|
|
## Dependencies
|
|
|
|
- the name of the Kubernetes Secret needs to be known (configured in the KMS
|
|
config file)
|
|
- each Tenant (Kubernetes Namespace) should use the same name for the Secret
|
|
that contains the Vault token
|
|
- the CSI-provisioner sidecar needs to provide the name of the Tenant together
|
|
with the metadata of the PVC to the Ceph-CSI provisioner
|
|
|
|
## Implementation Outline
|
|
|
|
- when creating the PVC the Ceph-CSI provisioner needs to store the Kubernetes
|
|
Namespace of the PVC in its metadata
|
|
- stores the `csi.volume.owner` (name of Tenant) in the metadata of the
|
|
volume and sets it as `rbdVolume.Owner`
|
|
- the Ceph-CSI node-plugin needs to request the Vault Token in the NodeStage
|
|
CSI operation and create/get the key for the PVC
|
|
- the Ceph-CSI provisioner needs to request the Vault Token to delete the key
|
|
for the PVC in the VolumeDelete CSI operation
|
|
|
|
### New sidecar: Fetching Tenant Tokens from Kubernetes
|
|
|
|
CSI is an abstraction and should not communicate with Kubernetes or other
|
|
Container Orchestration systems directly. Therefore, it is needed to
|
|
communicate with a service that can provide the Vault Token from the Tenants.
|
|
This service can be provided by a sidecar, exposing an endpoint as a
|
|
UNIX-Domain-Socket to communicate through.
|
|
|
|
This sidecar does not exist at the moment. There are other features within
|
|
Ceph-CSI that will benefit from this sidecar, e.g. Topology support. Because
|
|
Ceph-CSI already uses the Kubernetes API to fetch details from Kubernetes, it
|
|
should be acceptable to fetch the Vault Token configuration for the Tenants the
|
|
same way.
|
|
|
|
The feature for a sidecar that provides access to the required information from
|
|
Kubernetes and other Container Orchestration frameworks is tracked in
|
|
[#1782](https://github.com/ceph/ceph-csi/issues/1782).
|
|
|
|
## Configuration Details
|
|
|
|
- the current KMS configuration file needs extensions for Vault Token support
|
|
- introduce a new KMS-type: VaultTokenKMS (the current VaultKMS uses a
|
|
Kubernetes ServiceAccount)
|
|
- configuration of the VaultTokenKMS can be very similar to VaultKMS for common
|
|
settings
|
|
- the configuration can override the defaults for each Tenant separately
|
|
- Vault Service connection details (address, TLS options, ...)
|
|
- name of the Kubernetes Secret that can be looked up per tenant
|
|
- the configuration points to a Kubernetes Secret per Tenant that contains the
|
|
Vault Token
|
|
- the configuration points to an optional Kubernetes ConfigMap per Tenant that
|
|
contains alternative connection options for the VaultTokenKMS service
|
|
|
|
### Example of the KMS configuration file for Vault Tokens
|
|
|
|
The configuration is available in the Ceph-CSI containers as
|
|
`/etc/ceph-csi-encryption-kms-config/config.json`:
|
|
|
|
```json
|
|
{
|
|
"vault-with-tokens": {
|
|
"encryptionKMSType": "vaulttokens",
|
|
"vaultAddress": "http://vault.default.svc.cluster.local:8200",
|
|
"vaultBackendPath": "secret/",
|
|
"vaultTLSServerName": "vault.default.svc.cluster.local",
|
|
"vaultCAFromSecret": "vault-ca",
|
|
"vaultCAVerify": "false",
|
|
"tenantConfigName": "ceph-csi-kms-config",
|
|
"tenantTokenName": "ceph-csi-kms-token",
|
|
"tenants": {
|
|
"my-app": {
|
|
"vaultAddress": "https://vault.example.com",
|
|
"vaultCAVerify": "true"
|
|
},
|
|
"an-other-app": {
|
|
"tenantTokenName": "storage-encryption-token"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
In the Kubernetes StorageClass, the `kmsID` should be set to
|
|
`vault-with-tokens` in order to select the above configuration.
|
|
|
|
**Required options**:
|
|
|
|
- `encryptionKMSType`: should be set to `vaulttokens`
|
|
- `vaultAddress`: should be set to the URL of the Vault service
|
|
|
|
**Optional options**:
|
|
|
|
- `vaultBackendPath`: defaults to `secret/`
|
|
- `vaultTLSServerName`: not used when unset
|
|
- `vaultCAFromSecret`: not used when unset
|
|
- `vaultCAVerify`: defaults to `true`
|
|
- `tenantConfigName`: the name of the Kubernetes ConfigMap that contains the
|
|
Vault connection configuration (can be overridden per Tenant, defaults to
|
|
`ceph-csi-kms-config`)
|
|
- `tenantTokenName`: the name of the Kubernetes Secret that contains the Vault
|
|
Token (can be overridden per Tenant, defaults to `ceph-csi-kms-token`)
|
|
- `tenants`: list of Tenants (Kubernetes Namespaces) with their connection
|
|
configuration that differs from the global parameters
|
|
|
|
### Configuration stored in the Tenants Kubernetes Namespace
|
|
|
|
The Vault Token needs to be configured per Tenant. Each Tenant can create,
|
|
modify or delete their own personal Token. The Token is stored in a Kubernetes
|
|
Secret in the Kubernetes Namespace where the PVC is created.
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: ceph-csi-kms-token
|
|
stringData:
|
|
token: "sample_root_token_id"
|
|
```
|
|
|
|
The name `ceph-csi-kms-token` is the default, but can be changed by setting
|
|
`tenantTokenName` in the `/etc/ceph-csi-encryption-kms-config/config.json`
|
|
configuration file.
|
|
|
|
In addition to the Vault Token that can be configured per Tenant, the
|
|
connection parameters to the Vault Service can be stored in the Tenants
|
|
Kubernetes Namespace as well.
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: ceph-csi-kms-config
|
|
data:
|
|
vaultAddress: "https://vault.infosec.example.org"
|
|
vaultBackendPath: "secret/ceph-csi-encryption/"
|
|
vaultTLSServerName: "vault.infosec.example.org"
|
|
vaultCAFromSecret: "vault-infosec-ca"
|
|
vaultClientCertFromSecret: "vault-client-cert"
|
|
vaultClientCertKeyFromSecret: "vault-client-cert-key"
|
|
vaultCAVerify: "true"
|
|
```
|
|
|
|
Only parameters with the `vault`-prefix may be changed in the Kubernetes
|
|
ConfigMap of the Tenant.
|
|
|
|
### Certificates stored in the Tenants Kubernetes Namespace
|
|
|
|
The `vaultCAFromSecret` , `vaultClientCertFromSecret` and
|
|
`vaultClientCertKeyFromSecret` secrets should be created in the namespace where
|
|
Ceph-CSI is deployed. The sample of secrets for the CA and client Certificate.
|
|
|
|
#### CA Certificate to verify Vault server TLS certificate
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: secret
|
|
metadata:
|
|
name: vault-infosec-ca
|
|
stringData:
|
|
cert: |
|
|
MIIC2DCCAcCgAwIBAgIBATANBgkqh...
|
|
```
|
|
|
|
#### Client Certificate for Vault connection
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: secret
|
|
metadata:
|
|
name: vault-client-cert
|
|
stringData:
|
|
cert: |
|
|
BATANBgkqcCgAwIBAgIBATANBAwI...
|
|
```
|
|
|
|
#### Client Certificate key for Vault connection
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: secret
|
|
metadata:
|
|
name: vault-client-cert-key
|
|
stringData:
|
|
key: |
|
|
KNSC2DVVXcCgkqcCgAwIBAgIwewrvx...
|
|
```
|
|
|
|
Its also possible that a user can create a single secret for the certificates
|
|
and update the configuration to fetch certificates from a secret.
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: secret
|
|
metadata:
|
|
name: vault-certificates
|
|
stringData:
|
|
cert: |
|
|
MIIC2DCCAcCgAwIBAgIBATANBgkqh...
|
|
cert: |
|
|
BATANBgkqcCgAwIBAgIBATANBAwI...
|
|
key: |
|
|
KNSC2DVVXcCgkqcCgAwIBAgIwewrvx...
|
|
```
|