Steven's Knowledge

Patterns

App-of-apps, ApplicationSets, promotion workflows, secrets management, multi-cluster, drift detection, progressive delivery

Patterns

The patterns that scale GitOps from "one app, one cluster" to "fifty teams, ten clusters, hundreds of services."

App of Apps

The bootstrap pattern: one root Application that points at a directory of Applications.

argocd/
├── root.yaml                 # the root Application
└── applications/
    ├── monitoring.yaml       # → Prometheus chart
    ├── ingress.yaml          # → nginx-ingress
    ├── cert-manager.yaml
    └── team-payments.yaml    # → payments team's apps

kubectl apply -f argocd/root.yaml once; everything else follows. Cluster bootstrap is a single command, reproducible from Git.

ApplicationSets

When you need generated Applications — one per cluster, one per team, one per branch — use ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: per-cluster-apps
spec:
  generators:
    - list:
        elements:
          - cluster: staging
            url: https://staging.api.example.com
          - cluster: prod-east
            url: https://prod-east.api.example.com
          - cluster: prod-west
            url: https://prod-west.api.example.com
  template:
    metadata:
      name: '{{cluster}}-platform'
    spec:
      source:
        repoURL: https://github.com/youruser/config
        path: 'clusters/{{cluster}}'
      destination:
        server: '{{url}}'
        namespace: platform
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Other generators: git (one app per directory or per branch), cluster (one per ArgoCD-registered cluster), matrix (cross-product), pull request (preview environments).

Flux's equivalent: Kustomization with substitutions + Tenant CRDs.

Promotion Workflows

How does a Git change in staging/ become a Git change in prod/?

Pattern: directory-based overlays

apps/checkout/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml
    │   └── image-tag.yaml     # pins image: checkout:v1.2.3-rc1
    └── prod/
        ├── kustomization.yaml
        └── image-tag.yaml     # pins image: checkout:v1.2.2

CI auto-bumps staging/image-tag.yaml on every build. A separate workflow (or human PR) promotes the same tag to prod/image-tag.yaml.

Pattern: branch-based

main → prod, develop → staging. ArgoCD points different applications at different branches. Promotion = merging developmain. Simple but mixes concerns.

Pattern: image-tag PRs

Flux Image Update Controller / Argo Image Updater watch the registry; new image matching a regex opens a PR to bump the manifest. Merging the PR deploys it.

# Flux: ImagePolicy + ImageUpdateAutomation
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata: { name: checkout }
spec:
  imageRepositoryRef: { name: checkout-registry }
  policy:
    semver:
      range: '>=1.0.0'
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata: { name: bump }
spec:
  interval: 1m
  sourceRef: { kind: GitRepository, name: flux-system }
  git:
    commit:
      author: { email: ci@example.com, name: ci }
      messageTemplate: 'chore: bump checkout to {{range .Updated.Images}}{{.}}{{end}}'
    push: { branch: main }
  update: { strategy: Setters }

Combined with # {"$imagepolicy": "flux-system:checkout"} annotations in your manifests, this fully closes the loop.

Secret Management

Plain secrets can't go in Git. Three GitOps-compatible approaches:

Sealed Secrets (Bitnami)

Encrypt a Secret to a cluster-specific public key; the controller decrypts. The sealed form is safe in Git.

kubectl create secret generic mysecret \
  --from-literal=password=hunter2 \
  --dry-run=client -o yaml \
  | kubeseal --controller-namespace kube-system -o yaml > sealed.yaml
git add sealed.yaml

Pro: works offline; simple. Con: key lives only in the cluster — DR plan needs to back it up.

SOPS + age

Encrypt the values inline using sops + age (or KMS). Flux has native SOPS support; ArgoCD via plugin.

sops --encrypt --age $AGE_PUBLIC_KEY --in-place secret.yaml

Pro: works for any YAML, multi-key. Con: a bit more setup.

External Secrets Operator

Reference a secret name; the operator pulls the actual value from Vault / AWS Secrets Manager / GCP Secret Manager at runtime.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: db-creds }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target: { name: db-creds }
  data:
    - secretKey: password
      remoteRef: { key: prod/db, property: password }

Pro: centralizes secrets in Vault; rotation works. Con: another moving piece.

For most teams: External Secrets Operator + Vault is the right scaling choice; Sealed Secrets for the bootstrap secrets the others need.

Multi-Cluster

ArgoCD can manage many clusters from one ArgoCD instance:

argocd cluster add prod-east-context
argocd cluster add prod-west-context

Or run "ArgoCD per cluster" (hub-and-spoke or fully decentralized).

ModelProsCons
Hub: 1 ArgoCD → N clustersSingle pane; consistent UI; central auditHub failure affects all; network connectivity from hub needed
Spoke: ArgoCD per clusterIsolation; resilience; each cluster bootstraps independentlyMore to operate; no single UI

Flux defaults to ArgoCD per cluster by design. ArgoCD is comfortable with either.

Progressive Delivery

GitOps + canaries / blue-green:

  • Argo Rollouts integrates with ArgoCD: Rollout CRD instead of Deployment gives canary, blue-green, analysis hooks.
  • Flagger (Flux ecosystem) does the same for Flux.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: checkout }
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100
      analysis:
        templates: [{ templateName: success-rate }]
        startingStep: 2

Promotion now happens through canary analysis, not just merge. Code in Git → 20% traffic → metrics OK → 100% traffic. Failure auto-rolls-back.

Drift Detection and Self-Healing

syncPolicy.automated.selfHeal: true corrects drift continuously. Sometimes you don't want that:

  • Audit mode: alert on drift but don't auto-correct. Good for shared clusters where ops still make manual changes during transition.
  • Selective ignore: ignoreDifferences in the Application spec to ignore fields known to be controller-managed (e.g., HPA changes replicas, autoscaler-set annotations).
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

If HPA owns replicas, GitOps should leave it alone.

PR Preview Environments

ArgoCD ApplicationSet with the pullRequest generator creates a Namespace + Application per open PR:

spec:
  generators:
    - pullRequest:
        github:
          owner: youruser
          repo: app
          tokenRef: { secretName: github-token, key: token }
        requeueAfterSeconds: 60
  template:
    metadata: { name: 'preview-pr-{{number}}' }
    spec:
      source:
        repoURL: ...
        path: manifests
        helm:
          parameters:
            - name: image.tag
              value: 'pr-{{number}}'
      destination:
        namespace: 'preview-{{number}}'

PR closes → namespace + app destroyed. Per-PR full environment, automatically.

Anti-Patterns

kubectl apply in prod. Anyone with cluster access bypassing Git is breaking the model. Lock down direct prod access.

Auto-sync in prod with no review. Image automation auto-promoting to prod = a bad image in CI deploys instantly. Use PR-based promotion for prod or wrap auto-sync with canary analysis.

Manifests in app repo + multiple clusters. The app team owns the manifest, but multiple environments share it; promotion becomes a coordination nightmare. Config repo separate from app repo.

Secrets in Git unencrypted. Yes, it happens. Sealed Secrets / SOPS / ESO from day one.

One huge ApplicationSet generator. Generates 200 applications all syncing simultaneously, hammers the ArgoCD server. Use multiple ApplicationSets, project boundaries, sync windows.

What's Next

On this page