Supposons que vous utilisiez des secrets dans une application ou des données chiffrées dans le fichier values.yaml, comme des données de certificat ou des données de base de données. Ces données doivent être lues par des outils GitOps comme ArgoCD lors du déploiement de l’application.
Comment allez-vous masquer la valeur dans ce fichier ? Avant d’en savoir plus sur ArgoCD avec le plugin ArgoCD-Vault, vous pouvez réviser vos concepts sur argocd en utilisant ce lien :
Argo CD — Un outil GitOps pour la livraison continue
Avant d’en savoir plus sur Argo CD, découvrons d’abord comment ArgoCD est couramment implémenté.
L’équipe d’ArgoCD a présenté argocd-vault-plugin. Ce plugin vise à résoudre le problème de la gestion des secrets avec GitOps et Argo CD. Grâce à ce plugin, on peut facilement utiliser Vault sans avoir à s’appuyer sur un opérateur ou une définition de ressource personnalisée.
Ce plugin peut être utilisé non seulement pour les secrets, mais aussi pour les déploiements, les configMaps ou toute autre ressource Kubernetes. Dans ArgoCD, il existe un moyen d’intégrer des plugins personnalisés si vous avez besoin de quelque chose en dehors des outils pris en charge qui sont intégrés. Le plugin fonctionne en récupérant d’abord les valeurs de Vault en fonction d’un chemin qui peut être spécifié comme une variable d’environnement ou une annotation à l’intérieur du fichier YAML.
Pré-requis: Helm doit être installé
Supposons que vous n’ayez pas configuré le coffre-fort dans le cluster K8. Dans un premier temps, nous pouvons installer un coffre-fort à l’intérieur du cluster. Pour notre objectif, nous pouvons utiliser le graphique Vault Helm pour installer le serveur Vault sur le cluster Kubernetes. À des fins d’exercice, nous pouvons utiliser le mode dev dans le serveur Vault. Dans ce mode, Vault s’exécute en mémoire et démarre comme non scellé. Comme son nom l’indique, il s’agit d’un environnement de développement, n’exécutez pas le mode dev dans l’environnement de production. En mode production, configurez HA.
Installer openBAO:
helm repo add hashicorp https://helm.releases.hashicorp.com
To install the vault helm chart in dev mode run the command:
helm install vault hashicorp/vault --set “server.dev.enabled=true”
vault deployment through helm
After some time the pods will be up and running:
kubectl get pods
Once the pods are up and running we can use port-forwarding to see the UI and you will be able to access the UI from http://localhost:8200
kubectl port-forward vault-0 8200
And after this you can login from UI and CLI also for our understanding we will use CLI to login to the vault.
To login from CLI just export the URL and run the login command to log in to the vault server.
export VAULT_ADDR=http://127.0.0.1:8200
vault login root
If you want to login to UI then run vault token create command which will give us the token to login into the vault UI. After running this command you will get a token of the form hvs**. Keep that token as we need to create a secret in argocd using the same token.
vault token create
Before this let’s have a quick look on what are the ways in which we can integrate/authenticate with the vault.
Vault Authentication Methods
There are many ways we can authenticate with a vault. Mainly we use these three methods:
But we will make use of token authentication. Also ArgoCD supports AppRole, Github, Kubernetes and Userpass Auth Method for getting secrets from Vault.
VAULT TOKEN AUTHENTICATION
For Vault Token Authentication, these are the required parameters:
VAULT_ADDR: Your HashiCorp Vault Address
VAULT_TOKEN: Your Vault token
AVP_TYPE: vault
AVP_AUTH_TYPE: token
This option may be the easiest to test with locally, depending on your Vault setup.
So let’s create a Kubernetes secret named vault-configuration using the following:
apiVersion: v1
kind: Secret
metadata:
name: vault-configuration
namespace: argocd
data:
VAULT_ADDR: aHR0cDovL3ZhdWx0LmRlZmF1bHQ6ODIwMA==
VAULT_TOKEN: aHZzLkRQekVyQnFQaVQ2clRhSnA4WVUyZUFkVg==
AVP_TYPE: dmF1bHQ=
AVP_AUTH_TYPE: dG9rZW4=
type: Opaque
Configmap:
There are two ways to integrate plugins using Installation via argocd-cm ConfigMap and other is Installation via a sidecar container. As the Installation via argocd-cm ConfigMap is deprecated in argocd version 2.6, We will choose the option based on sidecar and initContainer.
apiVersion: v1
kind: ConfigMap
metadata:
name: cmp-plugin
namespace: argocd
data:
avp-helm.yaml: |
---
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: argocd-vault-plugin-helm
spec:
allowConcurrency: true
discover:
find:
command:
- sh
- "-c"
- "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
generate:
command:
- bash
- "-c"
- |
helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
argocd-vault-plugin generate -s vault-configuration -
lockRepo: false
avp.yaml: |
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: argocd-vault-plugin
spec:
allowConcurrency: true
discover:
find:
command:
- sh
- "-c"
- "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
generate:
command:
- argocd-vault-plugin
- generate
- "."
- "-s"
- "vault-configuration"
lockRepo: false
---
This config map has two plugin configuration defined. One configuration defines the plugin is used if we are using helm templates and want to make use of argocd-vault-plugin for env variables. Another plugin configuration is for the plane k8 manifest files which can we stored in github itself. So these secret and configmap should be created inside the argocd namespace so that the argocd reposerver reaches it.
kubectl create ns argocd
kubectl apply -f secret.yaml -n argocd
kubectl apply -f cmp-plugin.yaml -n argocd
After this, To patch the argocd-repo-server to add an initContainer to download argocd-vault-plugin and define the sidecar, we install argoCD with with custom values.yaml.
repoServer:
rbac:
- verbs:
- get
- list
- watch
apiGroups:
- ''
resources:
- secrets
- configmaps
initContainers:
- name: download-tools
image: registry.access.redhat.com/ubi8
env:
- name: AVP_VERSION
value: 1.11.0
command: [sh, -c]
args:
- >-
curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
chmod +x argocd-vault-plugin &&
mv argocd-vault-plugin /custom-tools/
volumeMounts:
- mountPath: /custom-tools
name: custom-tools
extraContainers:
# argocd-vault-plugin with plain YAML
- name: avp
command: [/var/run/argocd/argocd-cmp-server]
image: quay.io/argoproj/argocd:v2.4.0
securityContext:
runAsNonRoot: true
runAsUser: 999
volumeMounts:
- mountPath: /var/run/argocd
name: var-files
- mountPath: /home/argocd/cmp-server/plugins
name: plugins
- mountPath: /tmp
name: tmp
# Register plugins into sidecar
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: avp.yaml
name: cmp-plugin
# Important: Mount tools into $PATH
- name: custom-tools
subPath: argocd-vault-plugin
mountPath: /usr/local/bin/argocd-vault-plugin
- name: avp-helm
command: [/var/run/argocd/argocd-cmp-server]
image: quay.io/argoproj/argocd:v2.5.3
securityContext:
runAsNonRoot: true
runAsUser: 999
volumeMounts:
- mountPath: /var/run/argocd
name: var-files
- mountPath: /home/argocd/cmp-server/plugins
name: plugins
- mountPath: /tmp
name: tmp
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: avp-helm.yaml
name: cmp-plugin
- name: custom-tools
subPath: argocd-vault-plugin
mountPath: /usr/local/bin/argocd-vault-plugin
volumes:
- configMap:
name: cmp-plugin
name: cmp-plugin
- name: custom-tools
emptyDir: {}
Now we can install argocd with these values.yaml using:
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd -f values.yaml
You can check if all pods are up and running using:
kubectl get pods -n argocd
# note that the reposerver should show 2/2 pods as we added initContainers.
# we can access UI and get password for the ArgoCD using the following command
kubectl port-forward svc/argocd-server -n argocd 8080:443
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
Creating Secret in Vault and Deploying Application
Now as the whole infrastructure is set up, we can deploy the application by creating the secret. We can create a key-value kind of secret from UI and CLI also, as we are logged into the vault from CLI, run the following commands to add a secret to the vault.
# enable key-value engine
vault secrets enable kv-v2
# add the password to path kv-v2/argocd
vault kv put kv-v2/argocd password="argocd"
# add a policy to read the previously created secret
vault policy write argocd - <<EOF
path "kv-v2/data/argocd" {
capabilities = ["read"]
}
EOF
After this, we can create an application CRD to deploy the application. As we have installed the plugin using sidecar we will not use the plugin name. And we can use the path to a secret like this.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd-vault-plugin
namespace: argocd
spec:
destination:
name: ''
namespace: default
server: https://kubernetes.default.svc
project: default
source:
path: argocd-vault-plugin
repoURL: https://github.com/sharadhirao/ArgoCD-Example.git
targetRevision: HEAD
# need to add sync policy and check this file
Similarly, let’s say we want to create a secret in GitOps way, then we can use the path to secret as annotation in secret and get a username and password like this:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
annotations:
avp.kubernetes.io/path: "kv-v2/data/argocd"
type: Opaque
stringData:
TOKEN: <password>
If we apply this using the command:
kubectl apply -f app.yaml -n argocd
The application will be created with the secret.
As ArgoCD is becoming popular, there comes this question that how we manage secrets in a GitOps manner. We have a solution for Bitnami Sealed Secret also (https://medium.com/@raosharadhi11/bitnami-sealed-secret-new-way-to-encrypt-kubernetes-secret-d341914460cb). But the vault is the most preferred one in the companies. So I hope you got an idea of how we can integrate the ArgoCD with Vault as a plugin.
If you liked this article, please consider donating whatever amount you can using this link. It would mean a lot.
How it Works
Summary
The argocd-vault-plugin works by taking a directory of YAML or JSON files that have been templated out using the pattern of where you would want a value from Vault to go. The inside of the <> would be the actual key in Vault.
An annotation can be used to specify exactly where the plugin should look for the vault values. The annotation needs to be in the format avp.kubernetes.io/path: « path/to/secret ».
For example, if you have a secret with the key password-vault-key that you would want to pull from vault, you might have a yaml that looks something like the below code. In this yaml, the plugin will pull the value of the latest version of the secret at path/to/secret/password-vault-key and inject it into the Secret.
As YAML:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/path: « path/to/secret »
type: Opaque
data:
password:
As JSON:
{
« kind »: « Secret »,
« apiVersion »: « v1 »,
« metadata »: {
« name »: « example-secret »,
« annotations »: {
« avp.kubernetes.io/path »: « path/to/secret »
}
},
« type »: « Opaque »,
« data »: {
« password »: « »
}
}
And then once the plugin is done doing the substitutions, it outputs the manifest as YAML to standard out to then be applied by Argo CD. The resulting YAML would look like:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/path: « path/to/secret »
type: Opaque
data:
password: cGFzc3dvcmQK # The Value from the key password-vault-key in vault
Replacement behavior
By default the plugin does not perform any transformation of the secrets in transit. So if you have plain text secrets in Vault, you will need to use the stringData field and if you have a base64 encoded secret in Vault, you will need to use the data field according to the Kubernetes documentation.
There are 2 exceptions to this:
Placeholders that are in base64 format - see Base64 placeholders for details
Modifiers - see Modifiers for details
Types of placeholders
Generic placeholders
The example in the Summary uses a generic placeholder, which is just the name of the key of the secret in the secrets manager you want to inject. All placeholders have to be keys in the same secret in the secrets manager.
Valid examples:
Specifying the path of a secret
The only way to specify the path of a secret for generic placeholders is to use the avp.kubernetes.io/path annotation like this:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/path: « path/to/secret »
Specifying the version of a secret
The only way to specify the version of a secret for generic placeholders is to use the avp.kubernetes.io/secret-version annotation like this:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/secret-version: « 2 » # Requires at least 2 revisions to exist to work
Note: This ignored for secret managers that don’t allow versioning, meaning the latest version is returned
Inline-path placeholders
An inline-path placeholder allows you to specify the path, key, and optionally, the version to use for a specific placeholder. This means you can inject values from multiple distinct secrets in your secrets manager into the same YAML.
Valid examples:
<path:some/path#secret-key>
<path:some/path#secret-key#version>
If the version is omitted (first example), the latest version of the secret is retrieved.
Specifying the path of a secret
The only way to specify the path is in the placeholder itself: the string path: followed by the path in your secret manager to the secret. The avp.kubernetes.io/path annotation has no effect on these placeholders.
Specifying the version of a secret
The only way to specify the version is in the placeholder itself: the string following the last # in the placeholder should be the ID of the version of the secret in your secret manager. The avp.kubernetes.io/secret-version annotation has no effect on these placeholders.
Note: This ignored for secret managers that don’t allow versioning, meaning the latest version is returned
Special behavior
Base64 placeholders
Some tools like Kustomize secret generator will create Secrets with data fields containing base64 encoded strings from the source files. If you try to use s in the source files, they will be output in a base64 format.
The plugin can handle this case by finding any base64 encoded placeholders (either generic or inline-path), replace them, and re-base64 encode the result.
For example, given this input:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/path: « path/to/secret »
type: Opaque
data:
POSTGRES_URL: cG9zdGdyZXM6Ly88dXNlcm5hbWU+OjxwYXNzd29yZD5APGhvc3Q+Ojxwb3J0Pi88ZGF0YWJhc2U+P3NzbG1vZGU9cmVxdWlyZQ==
and these values for the secrets:
username: user
password: pass
host: host
port: 9443
database: my-db
the output is:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/path: « path/to/secret »
type: Opaque
data:
POSTGRES_URL: cG9zdGdyZXM6Ly91c2VyOnBhc3NAaG9zdDo5NDQzL215LWRiP3NzbG1vZGU9cmVxdWlyZQ==
Automatically ignoring strings
The plugin tries to be helpful and will ignore strings in the format if the avp.kubernetes.io/path annotation is missing, and only try to replace inline-path placeholders
This can be very useful when using AVP with YAML/JSON that uses 's for other purposes, for example in CRD’s with usage information:
kind: CustomResourceDefinition
apiVersion: v1
metadata:
name: some-crd
avp.kubernetes.io/path annotation hereannotations: {}
type: Opaque
fieldRef:
description: ‹ Selects a field of
the pod: supports metadata.name,
metadata.namespace, metadata.labels[''<KEY>''],
metadata.annotations[''<KEY>''],
spec.nodeName, spec.serviceAccountName,
status.hostIP, status.podIP, status.podIPs. ›
some-credential: path:somewhere/in/my/vault#credential
Ignoring entire YAML/JSON files
The plugin will ignore any given YAML/JSON file outright with the avp.kubernetes.io/ignore annotation set to « true »:
kind: CustomResourceDefinition
apiVersion: v1
metadata:
name: some-crd
avp.kubernetes.io/ignore annotation is setannotations:
avp.kubernetes.io/ignore: « true »
type: Opaque
fieldRef:
description: ‹ Selects a field of
the pod: supports metadata.name,
metadata.namespace, metadata.labels[''<KEY>''],
metadata.annotations[''<KEY>''],
spec.nodeName, spec.serviceAccountName,
status.hostIP, status.podIP, status.podIPs. ›
some-credential: path:somewhere/in/my/vault#credential
some-credential: path:somewhere/in/my/vault#credential#version3
Removing keys with missing values
By default, AVP will return an error if there is a that has no matching key in the secrets manager.
You can override this by using the annotation avp.kubernetes.io/remove-missing. This will remove keys whose values are missing from Vault from the entire YAML.
For example, given this input:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/remove-missing: « true »
stringData:
username:
password:
And this data in the secrets manager:
username: user
output is:
kind: Secret
apiVersion: v1
metadata:
name: example-secret
annotations:
avp.kubernetes.io/remove-missing: « true »
stringData:
username: user
This only works with generic placeholders.
Modifiers
base64encode
The base64encode modifier allows you to base64 encode a plain-text value retrieved from a secrets manager before injecting it into a Kubernetes secret.
Valid examples:
<username | base64encode>
<path:secrets/data/my-db#username | base64encode>
<path:secrets/data/my-db#username#version3 | base64encode>
This can be used for both generic and inline-path placeholders.
base64decode
The base64decode modifier decodes base64 encoded values into plain-text.
Valid examples:
<b64_username | base64decode>
<path:secrets/data/my-db#b64_username | base64decode>
<path:secrets/data/my-db#b64_username#version3 | base64decode>
jsonPath
The jsonPath modifier allows you use jsonpath to post-process objects or json, retrieved from a secrets manager, before injecting into a Kubernetes manifest. The output is a string. If your desired datatype is not a string, pass the output through jsonParse.
See the Kubernetes jsonPath documentation for more detail: https://kubernetes.io/docs/reference/kubectl/jsonpath/
Valid examples:
<credentials | jsonPath {.username}>
<path:secrets/data/my-db#credentials | jsonPath {.username}{':'}{.password}>
<path:secrets/data/my-db#credentials#version3 | jsonPath {.username} | base64encode>
<path:secrets/data/my-db#config | jsonPath {.replicas} | jsonParse>
jsonParse
The jsonParse modifier parses json strings into objects.
Valid examples:
<credentialsJson | jsonParse>
<path:secrets/data/my-db#credentialsJson | jsonParse>
<path:secrets/data/my-db#credentialsJson#version3 | jsonParse>
yamlParse
The yamlParse modifier converts YAML data into JSON.
Valid examples:
<credentials_yaml | yamlParse | jsonPath {.username}>
<path:secrets/data/db_yaml#yaml | yamlParse | jsonPath {.username}{':'}{.password}>
<path:secrets/data/db_yaml#yaml#version2 | yamlParse | jsonPath {.username} | base64encode>
indent
The indent modifier indents the secret data by the specified number of space characters (0x20), largely useful when injecting secrets into YAML strings embedded in YAML.
Valid examples:
<path:secrets/data/db#certs | jsonPath {.certificate} | indent 3>
sha256sum
The sha256sum modifier computes the SHA256 checksum of the string. Can be used to detect changes in a secret.
Valid examples:
kind: Deployment
spec:
template:
metadata:
annotations:
checksum/secret: <path:secrets/data/db#certs | sha256sum>
Error Handling
Detecting errors in chained commands
By default argocd-vault-plugin will read valid kubernetes YAMLs and replace variables with values from Vault. If a previous command failed and outputs nothing to stdout and AVP reads the input from stdin with the - argument, AVP will forward an empty YAML output downstream. To catch and prevent accientental errors in chained commands, please use the -o pipefail bash option like so:
$ sh -c ‹ ((>&2 echo « some error » && exit 1) | argocd-vault-plugin generate - | kubectl diff -f -); echo $?; ›
some error
0
$ set -o pipefail
$ sh -c ‹ ((>&2 echo « some error » && exit 1) | argocd-vault-plugin generate - | kubectl diff -f -); echo $?; ›
some error
1