AZiT

Flux and SOPS

The cluster kept in step with Git, with secrets encrypted in the repository.

Why GitOps with Flux, on the tour

On this page 4 sections

From here on, nothing is installed in the cluster by hand. Flux watches the repository and applies it; secrets are committed encrypted with SOPS and an age key, and Flux decrypts them inside the cluster.

1. Bootstrap Flux#

laptop
flux check --pre
cd ~/homelab
export GITHUB_TOKEN=$(gh auth token)
flux bootstrap github \
  --owner=@GITHUB_USER@ \
  --repository=homelab \
  --branch=main \
  --path=clusters/homelab \
  --personal \
  --private=true
unset GITHUB_TOKEN
git pull

Flux commits its own files to clusters/homelab/flux-system/ and adds a read-only deploy key to the GitHub repository. Bootstrapping a rebuilt cluster against the same repository replaces the old deploy key, which has the same name: GitHub then lists only the new one.

Check

laptop
flux check
flux get kustomizations

All checks pass, and flux-system is Ready.

2. An age key for SOPS#

One key encrypts every secret in the repository. Without it, nothing encrypted can ever be read again, so back up keys.txt to your password manager now. Until the cluster has this key, the layers below report secrets "sops-age" not found.

Rebuilding with a new key? Secrets encrypted with the old one cannot be read by the new cluster. Keep the old key aside, take those files out of the repository (and out of the kustomization.yaml that lists them) until you recreate them, and set suspend: true on a layer that cannot run without them.

laptop
mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt
chmod 600 ~/.config/sops/age/keys.txt
echo 'export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt"' >> ~/.zshrc && . ~/.zshrc
grep 'public key' ~/.config/sops/age/keys.txt
kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=$HOME/.config/sops/age/keys.txt
.sops.yaml repositoryDownload
creation_rules:
  - path_regex: .*\.sops\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: @AGE_PUBLIC_KEY@
  • Save it as .sops.yaml, with the dot, in the root of the repository.
  • Put your public key from keys.txt in place of @AGE_PUBLIC_KEY@.
  • Only files ending in .sops.yaml are encrypted, and in them only data and stringData: names and structure stay readable in Git.

The browser saves it without the dot, as sops.yaml. Move it into the root of the repository under its real name, and put your public key in:

laptop
mv ~/Downloads/sops.yaml ~/homelab/.sops.yaml
PUB=$(awk '/public key/ {print $4}' ~/.config/sops/age/keys.txt)
sed -i '' "s/@AGE_PUBLIC_KEY@/$PUB/" ~/homelab/.sops.yaml
cat ~/homelab/.sops.yaml

3. Layers, applied in order#

Controllers first, then their configuration, then applications, each waiting for the one before.

clusters/homelab/infrastructure.yaml repositoryDownload
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-controllers
  namespace: flux-system
spec:
  interval: 10m
  path: ./infrastructure/controllers
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-configs
  namespace: flux-system
spec:
  dependsOn:
    - name: infra-controllers
  interval: 10m
  path: ./infrastructure/configs
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  sourceRef:
    kind: GitRepository
    name: flux-system

4. The first controller: MetalLB#

MetalLB gives services an address on the servers network, from a pool the firewall's DHCP never hands out.

infrastructure/controllers/metallb.yaml repositoryDownload
apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: metallb
  namespace: metallb-system
spec:
  interval: 24h
  url: https://metallb.github.io/metallb
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: metallb
  namespace: metallb-system
spec:
  interval: 30m
  chart:
    spec:
      chart: metallb
      version: "0.16.1"
      sourceRef:
        kind: HelmRepository
        name: metallb
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
  values:
    # L2 mode only: no BGP, so no FRR routing daemons
    frrk8s:
      enabled: false
infrastructure/controllers/kustomization.yaml repositoryDownload
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - metallb.yaml
  • Only MetalLB for now. The next chapter adds the rest.
infrastructure/configs/metallb-pool.yaml repositoryDownload
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: homelab-services
  namespace: metallb-system
spec:
  addresses:
    - 192.168.5.200-192.168.5.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: homelab-services
  namespace: metallb-system
spec:
  ipAddressPools:
    - homelab-services
infrastructure/configs/kustomization.yaml repositoryDownload
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - metallb-pool.yaml
laptop
cd ~/homelab && git add .sops.yaml clusters infrastructure && git commit -m "Flux: SOPS and MetalLB" && git push
flux reconcile kustomization flux-system --with-source
flux get kustomizations --watch

The controllers layer takes a few minutes: it waits until every controller runs. Then the configs layer follows. If one hangs, flux get helmreleases -A shows which component it is waiting for.

Check

laptop
kubectl create deployment lb-test --image=nginx
kubectl rollout status deployment lb-test
kubectl expose deployment lb-test --type=LoadBalancer --port=80
kubectl get svc lb-test
sleep 5
ssh k8s-cp1 'curl -sS -m 5 -I http://192.168.5.200 | head -1'
kubectl delete svc,deployment lb-test

The service gets 192.168.5.200, and nginx answers HTTP/1.1 200 OK. The pause matters: right after expose, kube-proxy and MetalLB need a few seconds before the address answers, and -S makes curl say so instead of printing nothing.