Patterns
App-of-apps, ApplicationSets, promotion workflows, secrets management, multi-cluster, drift detection, progressive delivery
Patterns
The patterns that scale GitOps from "one app, one cluster" to "fifty teams, ten clusters, hundreds of services."
App of Apps
The bootstrap pattern: one root Application that points at a directory of Applications.
argocd/
├── root.yaml # the root Application
└── applications/
├── monitoring.yaml # → Prometheus chart
├── ingress.yaml # → nginx-ingress
├── cert-manager.yaml
└── team-payments.yaml # → payments team's appskubectl apply -f argocd/root.yaml once; everything else follows. Cluster bootstrap is a single command, reproducible from Git.
ApplicationSets
When you need generated Applications — one per cluster, one per team, one per branch — use ApplicationSet:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: per-cluster-apps
spec:
generators:
- list:
elements:
- cluster: staging
url: https://staging.api.example.com
- cluster: prod-east
url: https://prod-east.api.example.com
- cluster: prod-west
url: https://prod-west.api.example.com
template:
metadata:
name: '{{cluster}}-platform'
spec:
source:
repoURL: https://github.com/youruser/config
path: 'clusters/{{cluster}}'
destination:
server: '{{url}}'
namespace: platform
syncPolicy:
automated:
prune: true
selfHeal: trueOther generators: git (one app per directory or per branch), cluster (one per ArgoCD-registered cluster), matrix (cross-product), pull request (preview environments).
Flux's equivalent: Kustomization with substitutions + Tenant CRDs.
Promotion Workflows
How does a Git change in staging/ become a Git change in prod/?
Pattern: directory-based overlays
apps/checkout/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
└── overlays/
├── staging/
│ ├── kustomization.yaml
│ └── image-tag.yaml # pins image: checkout:v1.2.3-rc1
└── prod/
├── kustomization.yaml
└── image-tag.yaml # pins image: checkout:v1.2.2CI auto-bumps staging/image-tag.yaml on every build. A separate workflow (or human PR) promotes the same tag to prod/image-tag.yaml.
Pattern: branch-based
main → prod, develop → staging. ArgoCD points different applications at different branches. Promotion = merging develop → main. Simple but mixes concerns.
Pattern: image-tag PRs
Flux Image Update Controller / Argo Image Updater watch the registry; new image matching a regex opens a PR to bump the manifest. Merging the PR deploys it.
# Flux: ImagePolicy + ImageUpdateAutomation
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata: { name: checkout }
spec:
imageRepositoryRef: { name: checkout-registry }
policy:
semver:
range: '>=1.0.0'
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata: { name: bump }
spec:
interval: 1m
sourceRef: { kind: GitRepository, name: flux-system }
git:
commit:
author: { email: ci@example.com, name: ci }
messageTemplate: 'chore: bump checkout to {{range .Updated.Images}}{{.}}{{end}}'
push: { branch: main }
update: { strategy: Setters }Combined with # {"$imagepolicy": "flux-system:checkout"} annotations in your manifests, this fully closes the loop.
Secret Management
Plain secrets can't go in Git. Three GitOps-compatible approaches:
Sealed Secrets (Bitnami)
Encrypt a Secret to a cluster-specific public key; the controller decrypts. The sealed form is safe in Git.
kubectl create secret generic mysecret \
--from-literal=password=hunter2 \
--dry-run=client -o yaml \
| kubeseal --controller-namespace kube-system -o yaml > sealed.yaml
git add sealed.yamlPro: works offline; simple. Con: key lives only in the cluster — DR plan needs to back it up.
SOPS + age
Encrypt the values inline using sops + age (or KMS). Flux has native SOPS support; ArgoCD via plugin.
sops --encrypt --age $AGE_PUBLIC_KEY --in-place secret.yamlPro: works for any YAML, multi-key. Con: a bit more setup.
External Secrets Operator
Reference a secret name; the operator pulls the actual value from Vault / AWS Secrets Manager / GCP Secret Manager at runtime.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: db-creds }
spec:
refreshInterval: 1h
secretStoreRef: { name: vault, kind: ClusterSecretStore }
target: { name: db-creds }
data:
- secretKey: password
remoteRef: { key: prod/db, property: password }Pro: centralizes secrets in Vault; rotation works. Con: another moving piece.
For most teams: External Secrets Operator + Vault is the right scaling choice; Sealed Secrets for the bootstrap secrets the others need.
Multi-Cluster
ArgoCD can manage many clusters from one ArgoCD instance:
argocd cluster add prod-east-context
argocd cluster add prod-west-contextOr run "ArgoCD per cluster" (hub-and-spoke or fully decentralized).
| Model | Pros | Cons |
|---|---|---|
| Hub: 1 ArgoCD → N clusters | Single pane; consistent UI; central audit | Hub failure affects all; network connectivity from hub needed |
| Spoke: ArgoCD per cluster | Isolation; resilience; each cluster bootstraps independently | More to operate; no single UI |
Flux defaults to ArgoCD per cluster by design. ArgoCD is comfortable with either.
Progressive Delivery
GitOps + canaries / blue-green:
- Argo Rollouts integrates with ArgoCD:
RolloutCRD instead ofDeploymentgives canary, blue-green, analysis hooks. - Flagger (Flux ecosystem) does the same for Flux.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: checkout }
spec:
replicas: 5
strategy:
canary:
steps:
- setWeight: 20
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 100
analysis:
templates: [{ templateName: success-rate }]
startingStep: 2Promotion now happens through canary analysis, not just merge. Code in Git → 20% traffic → metrics OK → 100% traffic. Failure auto-rolls-back.
Drift Detection and Self-Healing
syncPolicy.automated.selfHeal: true corrects drift continuously. Sometimes you don't want that:
- Audit mode: alert on drift but don't auto-correct. Good for shared clusters where ops still make manual changes during transition.
- Selective ignore:
ignoreDifferencesin the Application spec to ignore fields known to be controller-managed (e.g., HPA changes replicas, autoscaler-set annotations).
spec:
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicasIf HPA owns replicas, GitOps should leave it alone.
PR Preview Environments
ArgoCD ApplicationSet with the pullRequest generator creates a Namespace + Application per open PR:
spec:
generators:
- pullRequest:
github:
owner: youruser
repo: app
tokenRef: { secretName: github-token, key: token }
requeueAfterSeconds: 60
template:
metadata: { name: 'preview-pr-{{number}}' }
spec:
source:
repoURL: ...
path: manifests
helm:
parameters:
- name: image.tag
value: 'pr-{{number}}'
destination:
namespace: 'preview-{{number}}'PR closes → namespace + app destroyed. Per-PR full environment, automatically.
Anti-Patterns
kubectl apply in prod. Anyone with cluster access bypassing Git is breaking the model. Lock down direct prod access.
Auto-sync in prod with no review. Image automation auto-promoting to prod = a bad image in CI deploys instantly. Use PR-based promotion for prod or wrap auto-sync with canary analysis.
Manifests in app repo + multiple clusters. The app team owns the manifest, but multiple environments share it; promotion becomes a coordination nightmare. Config repo separate from app repo.
Secrets in Git unencrypted. Yes, it happens. Sealed Secrets / SOPS / ESO from day one.
One huge ApplicationSet generator. Generates 200 applications all syncing simultaneously, hammers the ArgoCD server. Use multiple ApplicationSets, project boundaries, sync windows.
What's Next
- Best Practices — repo structure, RBAC, DR, scaling