TL;DR

I had Authentik handling human auth and kubeseal handling cluster secrets, which left a gap: anything that needed a real secret at runtime — API tokens, database passwords, Bedrock keys — was one kubectl get secret away from being readable in plaintext. I deployed HashiCorp Vault as a 3-node HA cluster on k3s, auto-unsealed via AWS KMS, with Authentik OIDC for human SSO and the Kubernetes auth method for workloads. Apps get their secrets injected by a sidecar; no app code touches a k8s Secret object anymore. The migration took a weekend and removed an entire category of “what if this got read” worry I’d been ignoring.

What was wrong with the old setup

I had three secret stores and none of them were great:

  1. Bitwarden for things only I touch — root passwords, recovery codes, the stuff that needs to survive without the cluster.
  2. Sealed-secrets for things I could commit to Git — TLS keys, bootstrap tokens.
  3. Raw Kubernetes Secret objects for everything else, which meant anything in the cluster that needed an API key got a Secret that anyone with kubectl get could decode in plaintext.

The third category had been growing. Every app I added with an external API key — Bedrock, Slack, GitHub, ProtonVPN, an embarrassing number of “let me try this service” experiments — ended up with a k8s Secret. My threat model had been “someone gets read access to my cluster”, which I was treating as “well, then they have all my secrets” and shrugging. Authentik fixed the human side of who could log into things. It did exactly nothing for the secret-at-rest problem.

There’s also the rotation problem. Sealed-secrets get committed to Git; rotating them means rewriting Git history or accepting that the old ciphertext is still in the repo forever, decryptable by whoever holds the controller’s private key. That’s fine for things you can re-issue and terrible for vendor tokens with annual rotation.

Architecture

                Humans
                  │
                  │ OIDC (SSO via Authentik)
                  ▼
            ┌──────────────┐
            │   Authentik  │
            └──────┬───────┘
                   │
                   ▼
       ┌──────────────────────┐         ┌────────────┐
       │   Vault HA (3 nodes) │◄────────│  AWS KMS   │  (auto-unseal)
       │   Raft storage       │         └────────────┘
       └──────┬───────────────┘
              │
   K8s auth method (ServiceAccount JWT)
              │
              ▼
       ┌──────────────────────┐
       │   Workload pods      │  ← vault-agent injector sidecar
       └──────────────────────┘

The pieces:

  • 3 Vault replicas, integrated Raft storage. No external storage backend; Raft is HA enough for a homelab and removes a Consul dependency I didn’t want.
  • Auto-unseal via AWS KMS so a reboot doesn’t require manually unsealing three replicas with key shares.
  • Authentik as the OIDC provider for human access to the Vault UI and CLI.
  • Kubernetes auth method for in-cluster workloads, which authenticate using their service account JWT — no static credentials anywhere.
  • Vault Agent Injector as a sidecar that fetches secrets and writes them to a shared volume, so the app code doesn’t have to learn Vault.

Installing Vault HA

I used the official Helm chart with values like this:

# vault-values.yaml
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable     = 1
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
        }
        seal "awskms" {
          region     = "us-east-1"
          kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/EXAMPLE-KEY-ID"
        }
        service_registration "kubernetes" {}

  dataStorage:
    storageClass: "longhorn"
    size: 10Gi

ui:
  enabled: true
  serviceType: "ClusterIP"

injector:
  enabled: true
  replicas: 2

Notes on the choices:

  • tls_disable = 1 on the listener because Traefik terminates TLS in front of Vault. This is fine if the listener is on a ClusterIP that only Traefik routes to. It is not fine on a NodePort or LoadBalancer.
  • AWS KMS is the easiest auto-unseal option if you already use AWS. GCP KMS, Azure Key Vault, or HSM are equivalent. Do not skip auto-unseal in a homelab. Manual unseal after every node reboot is the fastest way to abandon Vault three weeks in.
  • Raft on Longhorn is fine. Raft cares about durability and quorum; Longhorn gives you both.

Install:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --values vault-values.yaml

Initialize a single replica — auto-unseal handles the rest:

kubectl exec -n vault vault-0 -- vault operator init \
  -recovery-shares=5 \
  -recovery-threshold=3

Save the recovery keys in Bitwarden. They’re the only thing standing between you and a permanently sealed Vault if AWS KMS becomes unavailable for any reason — region outage, IAM mistake, key disabled. Do not skip this step.

After init, the other two replicas auto-join via Raft within ~30 seconds. kubectl exec -n vault vault-0 -- vault operator raft list-peers confirms quorum.

OIDC auth via Authentik

In Authentik, create an OAuth2/OpenID Provider with redirect URIs:

  • https://vault.k3s.internal.zolty.systems/ui/vault/auth/oidc/oidc/callback
  • https://vault.k3s.internal.zolty.systems/oidc/callback

Capture the client_id and client_secret. Note the chicken-and-egg problem: this secret needs to land somewhere before Vault knows it. I put a one-shot copy into a sealed-secret, got OIDC working, then rewrote the config from inside Vault itself (reading the secret out of Vault’s own KV) and deleted the sealed-secret. Brief discomfort during setup, clean state afterward.

Then in Vault:

vault auth enable oidc

vault write auth/oidc/config \
  oidc_discovery_url="https://authentik.k3s.internal.zolty.systems/application/o/vault/" \
  oidc_client_id="<client-id-from-authentik>" \
  oidc_client_secret="<client-secret-from-authentik>" \
  default_role="reader"

vault write auth/oidc/role/admin \
  bound_audiences="<client-id-from-authentik>" \
  allowed_redirect_uris="https://vault.k3s.internal.zolty.systems/ui/vault/auth/oidc/oidc/callback" \
  allowed_redirect_uris="https://vault.k3s.internal.zolty.systems/oidc/callback" \
  user_claim="preferred_username" \
  bound_claims='{"groups":"vault-admins"}' \
  policies="admin"

Bind a vault-admins group in Authentik to whichever humans (probably just you) should hold admin policy. UI login at vault.k3s.internal.zolty.systems then redirects through Authentik, which redirects back with a token bound to the admin Vault policy.

Kubernetes auth for workloads

Enable the auth method:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

Vault auto-discovers its own service account token and the cluster CA via its pod’s mounts. Do not pass them as flags — let the in-cluster config find them.

Bind a Kubernetes ServiceAccount to a Vault role with a Vault policy:

vault write auth/kubernetes/role/blog-content-gen \
  bound_service_account_names=content-gen \
  bound_service_account_namespaces=blog \
  policies=blog-content-gen \
  ttl=24h

And the policy:

# blog-content-gen.hcl
path "secret/data/blog/content-gen/*" {
  capabilities = ["read"]
}

vault policy write blog-content-gen blog-content-gen.hcl

A workload running as the content-gen ServiceAccount in the blog namespace can now authenticate to Vault and read anything under secret/blog/content-gen/. Nothing else.

Vault Agent Injector — secrets without code changes

The injector is the feature that makes Vault adoptable. With annotations on a pod, the injecting mutating webhook adds a vault-agent sidecar that authenticates with the pod’s service account JWT, fetches secrets, and writes them to a shared volume the app reads from. The app code doesn’t change.

metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "blog-content-gen"
    vault.hashicorp.com/agent-inject-secret-bedrock.env: "secret/data/blog/content-gen/bedrock"
    vault.hashicorp.com/agent-inject-template-bedrock.env: |
      {{- with secret "secret/data/blog/content-gen/bedrock" }}
      AWS_ACCESS_KEY_ID={{ .Data.data.access_key }}
      AWS_SECRET_ACCESS_KEY={{ .Data.data.secret_key }}
      AWS_REGION={{ .Data.data.region }}
      {{- end }}
spec:
  serviceAccountName: content-gen
  containers:
    - name: generator
      image: harbor.k3s.internal.zolty.systems/blog/content-gen:latest
      command: ["sh", "-c", "source /vault/secrets/bedrock.env && python -m generator"]

The file at /vault/secrets/bedrock.env is rendered by vault-agent and re-rendered if the underlying secret changes. No k8s Secret involved. Token renewal happens transparently as long as the pod stays running.

Verification

$ kubectl run vault-test --rm -it --restart=Never \
    --serviceaccount=content-gen --namespace=blog \
    --image=alpine -- sh

# After vault-agent injects:
$ cat /vault/secrets/bedrock.env
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1

And confirm the absence of the old k8s Secret:

$ kubectl get secret -n blog | grep bedrock
(no output — and that's the point)

Gotchas

  • Don’t tls_disable = 1 without Traefik in front. Tempting in dev; never in production. Even on the cluster network.
  • Auto-unseal is not a substitute for backups. Raft snapshots go to S3 nightly via a CronJob. Recovery keys still live in Bitwarden. KMS unavailable + no recovery keys = permanently sealed Vault. Don’t let that be the failure mode.
  • The injector needs serviceAccountName on the pod. I missed this once on a Deployment and spent 20 minutes wondering why the sidecar couldn’t authenticate. The default ServiceAccount has no Vault role; the binding is per-SA.
  • Vault policies are not RBAC. The mental model from k8s RBAC will mislead you. Vault policies are path-and-capability based — path "..." { capabilities = [...] }. Read the policy docs; don’t wing it from k8s intuition.
  • Authentik client secrets rotate. When they do, Vault’s OIDC config has to update. I run a small cron that pulls the latest secret from Authentik’s API and writes it back via vault write auth/oidc/config. The cron itself authenticates to Vault via the Kubernetes auth method, naturally.

Don’t have a homelab? The exact same pattern works on managed Kubernetes — the Vault Helm chart deploys identically, and a managed object store can hold the Raft snapshots. DigitalOcean’s Kubernetes service plus Spaces is a clean place to try this without an on-prem cluster.

Lessons

  • Sealed-secrets is fine for cluster bootstrap, terrible for the long tail. Anything that gets rotated or audited belongs in a real secret store.
  • Auto-unseal is non-negotiable for a homelab. Manual unseal turns Vault into a thing you reboot less, not a thing you actually use.
  • Don’t recurse Vault through Vault without an out-of-band recovery path. Authentik secret stored in Vault, Vault auth through Authentik — this works if you bootstrap it carefully, but the day Authentik is down you want a documented path back in. I keep an emergency token in Bitwarden with a 1h TTL policy that I rotate monthly.
  • The agent injector is what makes Vault adoptable. Without it, every app has to learn the Vault API or use a wrapper script. With it, “add an annotation” is the entire app-side change.

What’s next

The next phase is migrating GitHub Actions Secrets (the blog publishing pipeline, the content-gen scheduler, every long-lived CI token) into Vault, with the self-hosted runners authenticating via the Kubernetes auth method instead of holding static tokens. Same model as the in-cluster workloads, extended to CI. After that: Vault’s PKI engine as the cert authority for the cluster’s internal services — a natural pairing with the Cosign + SBOM work I’m planning for Harbor.