Cattage documentation

Cattage is a Kubernetes controller that enhances the multi-tenancy of Argo CD with Accurate. It is currently developed and maintained by Cybozu.

The repository is at https://github.com/cybozu-go/cattage .

Overview

Cattage is a Kubernetes controller that enhances the multi-tenancy of Argo CD with Accurate.

Features

• Management of root-namespaces for tenants

  When an administrator creates a tenant resource, root-namespaces for the tenant will be created. A RoleBinding resource will be created in the namespace so that the namespace can only be accessed by the tenant's users. Tenant users can create sub-namespaces in those root-namespaces.

• Automatic update of Argo CD AppProject resources

  An AppProject resource can control namespaces where users can deploy manifests. When a tenant user creates a sub-namespace, the AppProject will be automatically updated accordingly. Tenant users will be able to deploy applications with Argo CD to the namespaces.

• The ownership of sub-namespaces can be changed between tenants

  Sometimes users may want to move the ownership of an application to another tenant. When the parent of a sub-namespace is changed, Cattage will automatically update the permissions.

• Sharding application-controller instances

  Cattage can shard application-controller instances by the tenant. This feature is useful when you have a large number of tenants and want to avoid a single application-controller instance from being overloaded.

Setup

Kubernetes cluster

Cattage is a controller that runs in a soft multi-tenancy Kubernetes cluster. Namespaces must be isolated for each tenant.

There are many ways to achieve Namespace isolation. In EKS and GKE, you can integrate RBAC with IAM. For on-premises, Teleport and Loft may help you.

Argo CD

Install Argo CD as shown in the following page:

https://argo-cd.readthedocs.io/en/stable/getting_started/

Cattage isolates AppProject resource for each tenant.

So, please refer to the following page to enable user management. Argo CD supports a lot of authentication methods.

https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/

Cattage expects tenant users to be able to create Application resources. Apply the following manifest:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: application-admin
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups:
  - argoproj.io
  resources:
  - applications
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  - deletecollection

Cattage requires Argo CD's Applications in any namespace is enabled. In order to enable the feature, add --application-namespace="*" parameter to argocd-server and argocd-application-controller.

cert-manager

Cattage and Accurate depend on [cert-manager][] to issue TLS certificate for admission webhooks. If cert-manager is not installed on your cluster, install it as follows:

$ curl -fsLO https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
$ kubectl apply -f cert-manager.yaml

Accurate

Cattage depends on Accurate. It expects cattage.cybozu.io/tenant labels and RoleBinding resources to be propagated.

Include the following settings in your values.yaml:

controller:
  config:
    labelKeys:
      - cattage.cybozu.io/tenant
    watches:
      - group: rbac.authorization.k8s.io
        version: v1
        kind: RoleBinding

Install Accurate with the values.yaml as follows:

$ helm install --create-namespace --namespace accurate accurate -f values.yaml accurate/accurate

For more information, see the following page:

https://cybozu-go.github.io/accurate/helm.html

Cattage

Prepare values.yaml as follows:

controller:
  config:
    namespace:
      roleBindingTemplate: |
        apiVersion: rbac.authorization.k8s.io/v1
        kind: RoleBinding
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: admin
        subjects:
          - apiGroup: rbac.authorization.k8s.io
            kind: Group
            name: {{ .Name }}
          {{- range .Roles.admin }}
          - apiGroup: rbac.authorization.k8s.io
            kind: Group
            name: {{ .Name }}
          {{- end }}
    argocd:
      namespace: argocd
      appProjectTemplate: |
        apiVersion: argoproj.io/v1alpha1
        kind: AppProject
        spec:
          destinations:
          {{- range .Namespaces }}
            - namespace: {{ . }}
              server: '*'
          {{- end }}
          roles:
            - groups:
                - {{ .Name }}
                {{- range .Roles.admin }}
                - {{ .Name }}
                {{- end }}
              name: admin
              policies:
                - p, proj:{{ .Name }}:admin, applications, *, {{ .Name }}/*, allow
          sourceNamespaces:
            {{- range .Namespaces }}
            - {{ . }}
            {{- end }}
          sourceRepos:
            {{- range .Repositories }}
            - '{{ . }}'
            {{- else }}
            - '*'
            {{- end }}

appProjectTemplate and roleBindingTemplate should be configured appropriately for your multi-tenancy environment. Read Configurations for details.

Setup Helm repository:

$ helm repo add cattage https://cybozu-go.github.io/cattage
$ helm repo update

Install the Helm chart with your values.yaml:

$ helm install --create-namespace --namespace cattage cattage cattage/cattage -f values.yaml

Usage

Create a Tenant resource

Administrators can create the following tenant resource for a tenant team.

apiVersion: cattage.cybozu.io/v1beta1
kind: Tenant
metadata:
  name: your-team
spec:
  rootNamespaces:
    - name: your-root

The name of the tenant resource must match the name of the group in Kubernetes and Argo CD. The namespaces specified in spec.rootNamespaces will be created automatically.

$ kubectl get ns your-root
NAME        STATUS   AGE
your-root   Active   1m

RoleBinding and AppProject resource are also automatically created.

$ kubeclt get rolebinding -n your-root
NAME              ROLE                AGE
your-team-admin   ClusterRole/admin   2m

$ kubectl get appproject -n argocd your-team
NAME        AGE
your-team   2m

Create an Application resource

Tenant users can create a SubNamespace on their namespaces.

$ kubectl accurate sub create your-sub your-root

Tenant users can create an Application resource in the sub-namespace.

Prepare an Application resource as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: testhttpd
  namespace: your-sub
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: your-team
  source:
    repoURL: https://github.com/cybozu-go/cattage.git
    targetRevision: main
    path: samples/testhttpd
  destination:
    server: https://kubernetes.default.svc
    namespace: your-sub
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply the resource:

$ kubectl apply -f application.yaml

Make sure that the Application resource is synchronized.

$ kubectl get application -n your-sub
NAME        SYNC STATUS   HEALTH STATUS
testhttpd   Synced        Healthy

How to manage resources that already exist

Cattage can manage resources that have existed before with Tenant and Application.

You can make an existing namespace belong to Tenant. However, the namespace must be root or not managed by accurate.

A RoleBinding resource named <tenant-name>-admin will be created on a namespace belonging to a tenant. If a resource with the same name already exists, it will be overwritten.

An AppProject resource with the same name as a tenant will be created in argocd namespace. If a resource with the same name already exists, it will be overwritten.

How to change ownership

The ownership of sub-namespace can be transferred to other tenant.

Prepare a new tenant:

apiVersion: cattage.cybozu.io/v1beta1
kind: Tenant
metadata:
  name: new-team
spec:
  rootNamespaces:
    - name: new-root

Use kubectl accurate sub move command to change the parent of your-sub namespace to new-root.

$ kubectl accurate sub move your-sub new-root

As a result, application/testhttpd in your-sub will be out of sync. Please change the project of application/testhttpd correctly.

$ kubectl patch app testhttpd -n your-sub --type='json' -p '[{ "op": "replace", "path": "/spec/project", "value": "new-team"}]'

The application will be synced again.

Remove resources

When an administrator deleted a tenant resource:

• Root-namespaces and sub-namespaces for the tenant will remain
• RoleBinding on the namespaces will be deleted
• Applications on the namespaces will be deleted
• AppProject for the tenant will be deleted

Sharding

Overview

In Argo CD, as the number of managed applications increases, the load on the Application Controller becomes significant. While Argo CD supports sharding, it can only shard controllers per Kubernetes cluster. (ref. https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/ )

Cattage provides the capability to shard controllers on a per-tenant basis using applications in any namespace. By specifying a controller name in the Tenant resource, you can designate which controller will process Applications created in that tenant's Namespaces.

How to use

Setup stakater/Reloader

stakater/Reloader is a Kubernetes controller that watches for changes in ConfigMaps and Secrets, executing rolling updates on Deployments and StatefulSets as needed. Cattage uses stakater/Reloader to roll out updates to the Argo CD Application Controller whenever a ConfigMap is modified.

Follow these steps to set it up:

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install --create-namespace --namespace reloader reloader -f manifests/reloader-values.yaml stakater/reloader

Setup ArgoCD

Set up Argo CD with the following commands:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install --create-namespace --namespace argocd argocd argo/argo-cd

Copy and rename the StatefulSet for the Application Controller, then deploy it. Repeat for as many controllers as required for sharding:

helm template argo/argo-cd | yq ea '. as $$i ireduce ([]; . + $$i) | .[] | select(.kind=="StatefulSet") | .metadata.name="second-application-controller"' | kubectl apply -f -

Apply patches to the Application Controllers, ArgoCD Server, and Notification Controller to use the ConfigMaps generated by Cattage, and add an annotation to enable stakater/Reloader.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: default-application-controller-cm
                  optional: true
          name: application-controller
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: second-application-controller
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: second-application-controller-cm
                  optional: true
          name: application-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-notifications-controller
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: all-tenant-namespaces-cm
                  optional: true
          name: notifications-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - env:
            - name: ARGOCD_APPLICATION_NAMESPACES
              valueFrom:
                configMapKeyRef:
                  key: application.namespaces
                  name: all-tenant-namespaces-cm
                  optional: true
          name: server

Cattage generates the following configmaps:

• all-tenant-namespaces-cm: Lists namespaces belonging to all tenants
• default-application-controller-cm: Lists namespaces for tenants without a specified controller
• <controller name>-application-controller-cm: Lists namespaces for tenants with a specified controller

Setup Cattage

Follow the setup instructions to install Cattage.

Ensure controller.config.argocd.preventAppCreationInArgoCDNamespace in values.yaml is enabled to avoid multiple controllers processing the same Application in argocd namespace.

Creating Tenant Resources

When creating a Tenant resource, specify the controllerName.

apiVersion: cattage.cybozu.io/v1beta1
kind: Tenant
metadata:
  name: a-team
spec:
  rootNamespaces:
    - name: app-a
  controllerName: second

Applications created in the Namespace of that tenant will then be processed by the specified application controller. If no controller name is specified, it will be processed by the default application controller.

Custom Resources

• Tenant

Sub Resources

• ArgoCDSpec
• DelegateSpec
• RootNamespaceSpec
• TenantList
• TenantSpec
• TenantStatus

ArgoCDSpec

ArgoCDSpec defines the desired state of the settings for Argo CD.

Back to Custom Resources

DelegateSpec

DelegateSpec defines a tenant that is delegated access to a tenant.

Back to Custom Resources

RootNamespaceSpec

RootNamespaceSpec defines the desired state of Namespace.

Back to Custom Resources

Tenant

Tenant is the Schema for the tenants API.

Back to Custom Resources

TenantList

TenantList contains a list of Tenant.

Back to Custom Resources

TenantSpec

TenantSpec defines the desired state of Tenant.

Back to Custom Resources

TenantStatus

TenantStatus defines the observed state of Tenant.

Back to Custom Resources

Configurations

Configuration file

cattage-controller reads a configuration file on startup. The default location is /etc/cattage/config.yaml. The location can be changed with --config-file flag.

The configuration file should be a JSON or YAML file having the following keys:

The repository includes an example as follows:

namespace:
  commonLabels:
    accurate.cybozu.com/template: init-template
  commonAnnotations:
    foo: bar
  roleBindingTemplate: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: admin
    subjects:
      - apiGroup: rbac.authorization.k8s.io
        kind: Group
        name: {{ .Name }}
      {{- range .Roles.admin }}
      - apiGroup: rbac.authorization.k8s.io
        kind: Group
        name: {{ .Name }}
      {{- end }}
argocd:
  namespace: argocd
  appProjectTemplate: |
    apiVersion: argoproj.io/v1alpha1
    kind: AppProject
    spec:
      destinations:
      {{- range .Namespaces }}
        - namespace: {{ . }}
          server: '*'
      {{- end }}
      roles:
        - groups:
            # If `GitHubName` is specified as `ExtraParams`, use it, otherwise use `Name`.
            - {{with .ExtraParams.GitHubTeam}}{{ . }}{{else}}{{ .Name }}{{end}}
            {{- range .Roles.admin }}
            - {{with .ExtraParams.GitHubTeam}}{{ . }}{{else}}{{ .Name }}{{end}}
            {{- end }}
          name: admin
          policies:
            - p, proj:{{ .Name }}:admin, applications, *, {{ .Name }}/*, allow
      sourceNamespaces:
        {{- range .Namespaces }}
        - {{ . }}
        {{- end }}
      sourceRepos:
        {{- range .Repositories }}
        - '{{ . }}'
        {{- else }}
        - '*'
        {{- end }}

roleBindingTemplate and appProjectTemplate can be written in go-template format.

roleBindingTemplate can use the following variables:

appProjectTemplate can use the following variables:

Environment variables

Command-line flags

Flags:
      --add_dir_header                   If true, adds the file directory to the header
      --alsologtostderr                  log to standard error as well as files
      --cert-dir string                  webhook certificate directory
      --config-file string               Configuration file path (default "/etc/cattage/config.yaml")
      --health-probe-addr string         Listen address for health probes (default ":8081")
  -h, --help                             help for cattage-controller
      --leader-election-id string        ID for leader election by controller-runtime (default "cattage")
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
      --metrics-addr string              The address the metric endpoint binds to (default ":8080")
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          number for the log level verbosity
      --version                          version for cattage-controller
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
      --webhook-addr string              Listen address for the webhook endpoint (default ":9443")
      --zap-devel                        Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
      --zap-encoder encoder              Zap log encoding (one of 'json' or 'console')
      --zap-log-level level              Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level       Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').

Design notes

Overview

Cattage is a Kubernetes controller that enhances the multi-tenancy of Argo CD with Accurate.

Motivation

There is a known limitation for Argo CD to implement app-of-apps pattern in a multi-tenancy environment.

https://github.com/argoproj/argo-cd/issues/2785

We have developed the following mechanism to resolve the problem.

https://blog.kintone.io/entry/production-grade-delivery-workflow-using-argocd#Multi-tenancy

However, the mechanism still has the following problems:

• When a SubNamespace is created in HNC or Accurate, an administrator needs to add it to the destinations of the Application resource. (Argo CD supports specifying wildcards in destinations, but that is not enough for us.)

We need to build a better solution.

Goals

• Develop a Kubernetes custom controller to automate the configuration for multi-tenancy of Argo CD with Accurate.
• Automates the creation of root-namespaces and AppProject for each tenant.
• Allow tenant users to create Application resources in any namespace.
• Perform strict validation of Application resources for security.

User stories

Adding a tenant

An administrator only need to create one custom resource to add a tenant to a Kubernetes cluster. No more manual operations to add Applications, AppProjects and Namespaces.

Tenant users can create sub-namespaces within their tenant.

Adding an app-of-apps Application

Tenant users can create Application resources within their sub-namespaces. Application resources are strictly validated. No more deploying to another tenant's namespace by mistake.

Changing ownership

There are cases where you want to move ownership of an application between tenants. Accurate supports kubectl accurate sub move command to change the parent of a sub-namespace.

https://cybozu-go.github.io/accurate/subnamespaces.html#changing-the-parent-of-a-sub-namespace

An administrators can use this command to move the sub-namespace to another tenant. The permission of AppProjects, Applications and Namespaces will be updated automatically.

Alternatives

ApplicationSet

ApplicationSet is one of the features of Argo CD which generates Application resources based on user input.

https://argo-cd.readthedocs.io/en/stable/user-guide/application-set/

However, this feature does not give tenant users enough flexibility in their settings.

AppSource Controller

AppSource controller is similar to our proposal.

https://github.com/argoproj-labs/appsource

But AppSource is still not production-ready. Also, it does not solve our some problems.

Multiple Argo CD instance (argocd-operator)

We considered having an Argo CD instance for each tenant team, but it turned out to be a permissions problem.

Other Continuous Delivery tools

Other Continuous Delivery tools support multi-tenancy.

• https://github.com/fluxcd/flux2
• https://github.com/pipe-cd/pipe

However, we love Argo CD (the many features and the useful UI). We already have a lot of manifests managed by Argo CD. It's hard to switch to another tool now.

Development

1. Prepare a Linux box running Docker.

2. Checkout this repository.

   $ git clone https://github.com/cybozu-go/cattage
   

Setup CLI tools

1. Install aqua.

   https://aquaproj.github.io/docs/tutorial-basics/quick-start

2. Install CLI tools.

   $ cd cybozu-go/cattage
   $ aqua i -l
   

Development & Debug

1. Launch local Kubernetes cluster.

   $ cd cybozu-go/cattage
   $ make dev
   

2. Start Tilt.

   $ tilt up
   

3. Access: http://localhost:10350/

4. Stop the Kubernetes cluster.

   $ make stop-dev
   

Release procedure

This document describes how to release a new version.

Labeling

Release notes are automatically generated based on PRs included in the release. Those PRs are categorized based on the label assigned to them. Please refer to .github/release.yml for the kind of labels.

Versioning

Follow semantic versioning 2.0.0 to choose the new version number.

Bump version

1. Determine a new version number. Then set VERSION variable.

   # Set VERSION and confirm it. It should not have "v" prefix.
   $ VERSION=x.y.z
   $ echo $VERSION
   

2. Add a git tag to the main HEAD, then push it.

   $ git switch main
   $ git tag -a -m "Release v$VERSION" "v$VERSION"
   $ git tag -ln | grep $VERSION
   $ git push origin v$VERSION
   

Maintenance

How to update supported Kubernetes

Cattage supports the three latest Kubernetes versions. If a new Kubernetes is released, please update the following files.

• Update Kubernetes version in e2e/Makefile, .github/workflows/ci.yaml and cluster.yaml.
• Update kubectl version in aqua.yaml.
• Update k8s.io/* and sigs.k8s.io/controller-runtime packages version in go.mod.

If Kubernetes or controller-runtime API has changed, please fix the relevant source code.

How to update supported Argo CD

Cattage supports one Argo CD version. If a new Argo CD is released, please update the following files.

• Update Argo CD Version in aqua.yaml, e2e/Makefile and Makefile.
• Run make crds.

If Argo CD API has changed, please fix the relevant source code.

How to update dependencies

Renovate will create PRs that update dependencies once a week. However, Argo CD is not subject to Renovate. Also, Kubernetes is only updated with patched versions.