Gigawave

GitOps repo for the phonk Talos Linux homelab cluster (3 nodes), managed by Flux v2.

Cluster Overview

• OS: Talos Linux
• Hardware
  • HP ProDesk 400 G5 (4c, 48GB, 500GB SATA SSD, 2TB NVME)
  • HP ProDesk 400 G5 (4c, 48GB, 500GB SATA SSD, 2TB NVME)
  • HP ProDesk 400 G5 (4c, 64GB, 500GB SATA SSD, 2TB NVME)
• Network: Local 2.5GbE
• CNI
  • Cilium
    • kube-proxy replacement
    • Hubble (tbh i dont see much value here... thinking of turning it off)
• Storage
  • Local Path — workloads with PVCs are pinned to a node for rescheduling
  • TrueNAS synthwave — read-only NFS media for Jellyfin
  • TrueNAS retrowave — local replication, toka Time Machine, phonk backup targets and disposable log archive (see .docs/backups.md)
• Secrets
  • OpenBao
    • Integrated Raft storage: three voting replicas, one per node (tolerates one node failure)
    • Internal HTTPS with cert-manager certificates and trust-manager CA distribution
    • Unseal via static seal secret applied before flux bootstrap
  • External Secrets Operator
• Gateway
  • Envoy
    • external
      • 1bleim.de
      • anuth.dev
      • fullbyte.de
    • internal
      • ops.phonk.jukebox.fullbyte.de
      • home.anuth.dev

GitOps

• Bootstrap from forge.fullbyte.de/fullbyte/gigawave
• Self-hosted RenovateBot is acting as @helferlein
  • no Flux image automation

Docs

• Start here: Agent instructions and operations index

• Infrastructure map, access and checks

• Current handoff — unfinished work and user decisions

• Bootstrap — Flux install, manual secrets, SSH deploy key, reconcile reference

• OpenBao — init, zone KV + auth setup, OIDC, initial secrets

• Talos / BGP — node operations, MikroTik BGP config

• Each app has its own README detailing why it exists and what should be deployed here

Repository Structure

gigawave/
├── clusters/phonk/          # Flux Kustomization CRs
│   ├── flux-system/         # flux-operator FluxInstance
│   ├── infrastructure/      # one .yaml per infra layer
│   └── apps/                # one .yaml per zone
├── infrastructure/
│   ├── namespaces/          # zone + operator namespaces
│   ├── sources/             # HelmRepository + GitRepository CRs
│   ├── cni/cilium/          # Cilium HelmRelease (kube-system)
│   ├── csi/local-path/      # local-path-provisioner HelmRelease
│   ├── gateway/envoy/       # Envoy Gateway and internal/external gateways
│   ├── certificates/        # Bao-dependent Hetzner webhook + ACME issuer
│   ├── secrets/stores/      # namespace-scoped OpenBao SecretStores
│   └── operators/
│       ├── cert/            # cert-manager base operator only
│       ├── postgresql/cnpg/
│       ├── redis/redis-operator/
│       ├── secrets/external-secrets/
│       └── storage/garage-operator/
└── apps/
    ├── ops/                 # namespace: ops — cluster services
    │   ├── bao/             # OpenBao Raft (three voting replicas)
    │   ├── gitops/flux-operator/
    │   ├── monitoring/      # kube-prometheus-stack + Thanos
    │   └── network/dns/zone/# CoreDNS for *.ops.phonk.jukebox.fullbyte.de
    ├── fullbyte/            # namespace: fullbyte — fullbyte.de services
    │   ├── ci/              # Woodpecker CI
    │   ├── forge/           # Forgejo + forgejo-db
    │   ├── httpbin/
    │   ├── id/              # PocketID + pocketid-db
    │   └── s3/              # GarageCluster
    ├── home/                # namespace: home — home.anuth.dev services
    │   └── network/dns/
    │       ├── zone/        # CoreDNS for *.home.anuth.dev
    │       └── resolver/    # Blocky (upstreams: ops zone + home zone)
    ├── 1bleim/              # stub (future: 1bleim.de)
    ├── anuth/               # stub (future: anuth.dev)
    └── vanknel/             # stub (future: vanknel.de)

Dependency Graph

graph TD
    ns[infrastructure-namespaces]
    src[infrastructure-sources]
    gwcrd[infrastructure-crds-gateway-api]
    promcrd[infrastructure-crds-prometheus]
    cni[infrastructure-cni]
    csi[infrastructure-csi]
    operators[infrastructure-operators]
    gateway[infrastructure-gateway]
    tls[infrastructure-bao-tls]
    bao[apps-bao]
    stores[infrastructure-secret-stores]
    webhook[infrastructure-cert-webhook]
    issuers[infrastructure-cert-issuers]
    apps["apps-ops / apps-fullbyte / apps-home / portfolios"]

    src --> gwcrd
    src --> promcrd
    gwcrd --> cni
    ns --> csi
    src --> csi
    ns --> operators
    cni --> operators
    promcrd --> operators
    ns --> gateway
    cni --> gateway
    promcrd --> gateway
    operators --> tls
    tls --> bao
    operators --> bao
    csi --> bao
    bao --> stores
    stores --> webhook
    webhook --> issuers
    issuers --> apps
    gateway --> apps

Applications reconcile independently of the ops service bundle. Bao core starts before SecretStores and certificate issuance; its HTTPS exposure stays in apps-ops. Sources live in the already-bootstrapped flux-system namespace and reconcile independently of tenant namespaces. See dependency ownership and recovery for details.

Key Conventions

• New namespace → add to infrastructure/namespaces/namespaces.yaml
• New Helm repo → add to infrastructure/sources/helm-repos.yaml
• Operator HelmRelease → goes in infrastructure/operators/<category>/<name>/helmrelease.yaml
• Operator instances: one cluster per tenant (e.g. grafana-db, <app>-db). Always use instances: 3 with enablePodAntiAffinity: required and topologyKey: kubernetes.io/hostname — one replica per node, survives two node failures. No nodeSelector needed.
• New HTTPRoute → add the corresponding link to apps/ops/glance/configmap.yaml under the matching namespace/tenant group in the Services page bookmarks widget
• New HTTPRoute → add an endpoint entry to apps/ops/gatus/helmrelease.yaml under values.config.endpoints for uptime monitoring (Gatus does not support Gateway API HTTPRoute autodiscovery, static config only)
• Secrets pattern: ExternalSecret (pulls from OpenBao) → Kubernetes Secret → consumed by HelmRelease valuesFrom
• SecretStore: one namespace-scoped SecretStore named openbao per zone, managed centrally in infrastructure/secrets/stores/<namespace>.yaml, pointing to https://openbao.ops.svc.cluster.local:8200 and trusting the namespace-local openbao-ca ConfigMap. The certificate webhook and Flux UI also use namespace-scoped stores. Add each new store to that directory’s Kustomization.
• Stages: most workloads prod-only. If dev is required, create namespace <name>-dev and deploy a separate instance.
• Variable substitution: cluster-wide vars in cluster-config ConfigMap; Flux postBuild.substituteFrom injects them. Reference as ${VAR_NAME} in manifests.
• Deployment mode: single instance where possible. Stateful workloads: pin to node, deploy ≥2 if HA is possible, otherwise add backup TODO.
• Storage: prefer emptyDir for ephemeral state; PVCs only for data that must survive pod restarts.
• Storage / subPath: never mount a local-path PVC with subPath. The Talos kubelet runs in its own mount namespace where /var/lib/local-path-provisioner is not propagated, so the kubelet's subPath bind mount points at its own overlay rootfs and the data never reaches the PV (bit us with Prometheus: disableMountSubPath: true in the kube-prometheus-stack HelmRelease). Mount the whole volume instead. If a future workload truly needs a subPath on a PVC, add to the Talos machine config: machine.kubelet.extraMounts: [{destination: /var/lib/local-path-provisioner, type: bind, source: /var/lib/local-path-provisioner, options: [bind, rshared, rw]}].
• Email: outbound via Brevo (SMTP relay)
  • Verified senders
    • id@fullbyte.de (PocketID)
    • git@fullbyte.de (Forgejo)
    • noreply@fullbyte.de (all others)

Learnings

• Prometheus persistence confirmed by the user on 2026-10-29: metrics survived node restarts during the recent Talos upgrade. The earlier restart data-loss issue is resolved.

• routing

  • less is more
  • bridge is like a partition for network ports on a switch/router
  • add DNS and NTP server to your network early
  • wait for NTP to fully sync before relying on time-sensitive services