Secrets

K8s
How to handle secrets in K8s

Obfuscation, not encryption

Unlike [[ConfigMap]], K8s doesn’t like to show you plain text version of your secret. You must decode it with base64 -d, this is not encrypted, just obfuscated.

The container still sees the original plain text data. Let’s manually create a secret like this:

kl create secret generic sleep-secret-literal --from-literal=secret=shh...

Then, we reference this seret in a deployment like this:

% cat sleep/sleep-with-secret.yaml                                                                                                    
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
        - name: sleep
          image: kiamol/ch03-sleep
          env:
          - name: KIAMOL_SECRET
            valueFrom:
              secretKeyRef:
                name: sleep-secret-literal
                key: secret

If we apply this kl apply -f sleep/sleep-with-secret.yaml , we can see the secret lke this:

% kl exec -it deploy/sleep -- printenv | grep KIAMOL_SECRET                                                                           
KIAMOL_SECRET=shh...

You shouldn’t expose secrets as environment variables as that is not very secure. You should store secrets in files that have restricted premissions.

You can also store your secrets in plain text in a YAML file like so:

%cat todo-list/secrets/todo-db-secret-test.yaml                                                                                      
apiVersion: v1
kind: Secret
metadata:
  name: todo-db-secret-test
type: Opaque
stringData:                          # use this field when using plain text stuff
  POSTGRES_PASSWORD: "kiamol-2*2*"   # this is the plain text password

Interestingly, you will be able to see the plain text password if you do this! See the metadata.annotations field:

%kl get secret/sleep-secret-literal -o yaml
apiVersion: v1
data:
  POSTGRES_PASSWORD: a2lhbW9sLTIqMio=
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"todo-db-secret-test","namespace":"default"},"stringData":{"POSTGRES_PASSWORD":"kiamol-2*2*"},"type":"Opaque"}
  creationTimestamp: "2022-12-01T18:22:56Z"
  name: todo-db-secret-test
  namespace: default
  resourceVersion: "629050"
  uid: 35b42a79-a8dd-447d-a191-a295ca1e4d66
type: Opaque

Mounting Secrets as Files

This is recommended over env variables

% cat todo-list/todo-db-test.yaml                                                                                                     
apiVersion: apps/v1
kind: Deployment
metadata:
  name: todo-db
spec:
  selector:
    matchLabels:
      app: todo-db
  template:
    metadata:
      labels:
        app: todo-db
    spec:
      containers:
        - name: db
          image: postgres:11.6-alpine
          env:
          - name: POSTGRES_PASSWORD_FILE
            value: /secrets/postgres_password
          volumeMounts:
            - name: secret                           # Mounts a secret volume
              mountPath: "/secrets"                  # The location
      volumes:
        - name: secret
          secret:                                     # Volumen loaded
            secretName: todo-db-secret-test           #Name of secret
            defaultMode: 0400                         #Permissions set for files
            items:
            - key: POSTGRES_PASSWORD
              path: postgres_password

You can see that this secret is now mounted as a file

%kl exec deploy/todo-db -- ls /secrets                                                                                               
postgres_password

% kl exec deploy/todo-db -- cat /secrets/postgres_password                                                                            
kiamol-2*2*

Bringing together config, secrets, deployments and services

Here is an example file that uses both ConfigMaps and Secrets in a deployment

% cat todo-list/todo-web-test.yaml | pbcopy
apiVersion: v1
kind: Service
metadata:
  name: todo-web-test
spec:
  ports:
    - port: 8081
      targetPort: 80
  selector:
    app: todo-web
    environment: test
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: todo-web-test
spec:
  selector:
    matchLabels:
      app: todo-web
      environment: test
  template:
    metadata:
      labels:
        app: todo-web
        environment: test
    spec:
      containers:
        - name: web
          image: kiamol/ch04-todo-list
          env:
          - name: ASPNETCORE_ENVIRONMENT
            value: Test
          volumeMounts:
            - name: config
              mountPath: "/app/config"
              readOnly: true
            - name: secret
              mountPath: "/app/secrets"
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: todo-web-config-test
            items:
            - key: config.json
              path: config.json
        - name: secret
          secret:
            secretName: todo-web-secret-test
            defaultMode: 0400
            items:
            - key: secrets.json
              path: secrets.json

That json secret is stored like this:

% cat todo-list/secrets/todo-web-secret-test.yaml                                                                                     
apiVersion: v1
kind: Secret
metadata:
  name: todo-web-secret-test
type: Opaque
stringData:
  secrets.json: |-
    {
      "ConnectionStrings": {
        "ToDoDb": "Server=todo-db;Database=todo;User Id=postgres;Password=kiamol-2*2*;"
      }
    }

You can see that these files exist now

% kl exec deploy/todo-web-test -- ls /app/                                                                                            
config
secrets
...

Updating Configurations

Your app may only read config when it starts, so if you change configuration settings via [[ConfigMap]] or [[Secrets]] then you would have to restart your app. Two ways to do this:

  1. kl rollout restart deploy/.... (preferred method)
  2. Delete all pods related to that deployment using a label selector or something similar and let the deployment restart them.

Context From Discord chat with Michal https://discord.com/channels/1043031122721914940/1045846418331537459/1047961668426158192