Sharding

Overview

In Argo CD, as the number of managed applications increases, the load on the Application Controller becomes significant. While Argo CD supports sharding, it can only shard controllers per Kubernetes cluster. (ref. https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/ )

Cattage provides the capability to shard controllers on a per-tenant basis using applications in any namespace. By specifying a controller name in the Tenant resource, you can designate which controller will process Applications created in that tenant's Namespaces.

How to use

Setup stakater/Reloader

stakater/Reloader is a Kubernetes controller that watches for changes in ConfigMaps and Secrets, executing rolling updates on Deployments and StatefulSets as needed. Cattage uses stakater/Reloader to roll out updates to the Argo CD Application Controller whenever a ConfigMap is modified.

Follow these steps to set it up:

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install --create-namespace --namespace reloader reloader -f manifests/reloader-values.yaml stakater/reloader

Setup ArgoCD

Set up Argo CD with the following commands:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install --create-namespace --namespace argocd argocd argo/argo-cd

Copy and rename the StatefulSet for the Application Controller, then deploy it. Repeat for as many controllers as required for sharding:

helm template argo/argo-cd | yq ea '. as $$i ireduce ([]; . + $$i) | .[] | select(.kind=="StatefulSet") | .metadata.name="second-application-controller"' | kubectl apply -f -

Apply patches to the Application Controllers, ArgoCD Server, and Notification Controller to use the ConfigMaps generated by Cattage, and add an annotation to enable stakater/Reloader.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: default-application-controller-cm
                  optional: true
          name: application-controller
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: second-application-controller
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: second-application-controller-cm
                  optional: true
          name: application-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-notifications-controller
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: all-tenant-namespaces-cm
                  optional: true
          name: notifications-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: all-tenant-namespaces-cm
                  optional: true
          name: server

Cattage generates the following configmaps:

all-tenant-namespaces-cm: Lists namespaces belonging to all tenants
default-application-controller-cm: Lists namespaces for tenants without a specified controller
<controller name>-application-controller-cm: Lists namespaces for tenants with a specified controller

Setup Cattage

Follow the setup instructions to install Cattage.

Ensure controller.config.argocd.preventAppCreationInArgoCDNamespace in values.yaml is enabled to avoid multiple controllers processing the same Application in argocd namespace.

Creating Tenant Resources

When creating a Tenant resource, specify the controllerName.

apiVersion: cattage.cybozu.io/v1beta1
kind: Tenant
metadata:
  name: a-team
spec:
  rootNamespaces:
    - name: app-a
  controllerName: second

Applications created in the Namespace of that tenant will then be processed by the specified application controller. If no controller name is specified, it will be processed by the default application controller.

Cattage Documentation