Accurate documentation
Accurate is a Kubernetes controller for soft multi-tenancy environments. It is currently developed and maintained by Cybozu.
The repository is at https://github.com/cybozu-go/accurate .
Overview
Accurate is a Kubernetes controller to help operations in large soft multi-tenancy environments.
Soft multi-tenancy in Kubernetes
Kubernetes does not provide multi-tenancy functions on its own. It merely provides Namespaces along with Role-Based Access Control (RBAC) to isolate resources such as Pods.
Soft multi-tenancy is a kind of technique to implement Namespace-based multi-tenancy on Kubernetes.
On the other hand, hard multi-tenancy provides a virtual kube-apiserver
for each tenant to isolate privileges completely.
In a soft multi-tenancy environment, a cluster admin grants privileges in a Namespace to a group of tenant users by creating RoleBinding object like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: tenant
name: admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- kind: Group
name: group-for-tenant
apiGroup: rbac.authorization.k8s.io
admin
ClusterRole is a built-in role to give admin privileges on any kind of namespace-scope resources.
With this RoleBinding, users in group-for-tenant
can freely create/edit/delete namespace-scope resources in tenant
Namespace.
In many cases, a tenant needs to have multiple Namespaces to run multiple independent applications. However, tenant users are not allowed to create or delete Namespaces because Namespace is a cluster-scope resource. Otherwise, they would be able to delete other tenants' Namespaces!
What is Accurate?
Accurate introduces a namespace-scope custom resource called SubNamespace. With SubNamespace, tenant users can create a Namespace by creating a SubNamespace, and delete the created Namespace by deleting the SubNamespace.
The created Namespace is considered a child of the Namespace where the SubNamespace is created. The child Namespace may inherit labels and annotations from its parent Namespace.
Accurate also propagates resources such as Role, RoleBinding, or Secret from a parent Namespace to its children Namespaces. Without propagating Role/RoleBinding, the tenant user would be able to do nothing in newly created Namespaces.
Features
-
Resource propagation between namespaces
Accurate can propagate any namespace-scope resource including custom resources between Namespaces.
-
Inheriting labels and annotations creation/update from parent namespaces
Namespace labels often play important roles. For example, Pod Security Admission, a new feature planned for Kubernetes 1.22, uses Namespace labels to control security policies.
-
SubNamespace custom resource for tenant users
This is the feature to allow tenant users to create and delete Namespaces by themselves. SubNamespaces can be created in either a root Namespace or a Namespace created by SubNamespace. A root Namespace is a Namespace labeled with
accurate.cybozu.com/type=root
. -
Template namespaces
A template Namespace is a Namespace labeled with
accurate.cybozu.com/type=template
. Labels, annotations, and resources can be propagated from a template Namespace to other Namespaces referencing the template withaccurate.cybozu.com/template=<name>
label.This feature is implemented to allow resource propagation between normal Namespaces.
-
kubectl
pluginkubectl-accurate
is akubectl
plugin to make the operations for Accurate easy.
Concepts
Namespace types
Accurate defines the following types of Namespaces:
- Template: Namespace labeled with
accurate.cybozu.com/type=template
- Root: Namespace labeled with
accurate.cybozu.com/type=root
- Sub-namespace: Namespace labeled with
accurate.cybozu.com/parent=<name>
Any Namespace other than sub-namespaces can reference a template Namespace with accurate.cybozu.com/template=<name>
label.
Sub-namespace can reference a root or another sub-namespace as its parent.
When configured to do so, Accurate propagates the Namespace labels, annotations, and namespace-scope resources from a referenced Namespace to referencing Namespaces.
Circular references are prohibited by an admission webhook.
Resource propagation
Accurate propagates any namespace-scope resource that are annotated with accurate.cybozu.com/propagate=<mode>
.
Mode is one of the following:
create
: the resource will be created in referencing Namespaces if missing.update
: the resource will be created in referencing Namespaces if missing, or will be updated if not identical, or will be deleted when the resource in the referenced Namespace is deleted.
Propagating generated resources (DEPRECATED)
[!WARNING]
Propagating generated resources is a deprecated feature and is subject for removal soon.
If a resource annotated with accurate.cybozu.com/propagate-generated=<mode>
creates a resource and set an owner reference in the created resource, Accurate automatically adds accurate.cybozu.com/propagate=<mode>
to the created resource.
This can be used, for example, to propagate Secret created from SealedSecret.
Getting started
Follow these steps to install Accurate on your cluster and start using it.
- Adjust configurations for your environment
- Install cert-manager and apply Accurate manifests
- Install
kubectl-accurate
to local machines
Configurations
Helm Chart values
Read Helm Chart for details.
Configuration file
accurate-controller
reads its configurations from a configuration file.
The repository includes an example as follows:
# Labels to be propagated to sub-namespaces.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
# https://pkg.go.dev/path#Match
labelKeys:
- team
# Annotations to be propagated to sub-namespaces.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
# https://pkg.go.dev/path#Match
annotationKeys:
# An example to propagate an annotation for MetalLB
# https://metallb.universe.tf/usage/#requesting-specific-ips
- metallb.universe.tf/address-pool
# Labels to be propagated to sub-namespaces from SubNamespace resource.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
# https://pkg.go.dev/path#Match
subNamespaceLabelKeys:
- app
# Annotations to be propagated to sub-namespaces from SubNamespace resource.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
# https://pkg.go.dev/path#Match
subNamespaceAnnotationKeys:
- foo.bar/baz
# List of GVK for namespace-scoped resources that can be propagated.
# Any namespace-scoped resource is allowed.
watches:
- group: rbac.authorization.k8s.io
version: v1
kind: Role
- group: rbac.authorization.k8s.io
version: v1
kind: RoleBinding
- version: v1
kind: Secret
- version: v1
kind: ResourceQuota
# List of nameing policy for SubNamespaces.
# root and match are both regular expressions.
# When a SubNamespace is created in a tree starting from a root namespace and the root namespace's name matches the "root" regular expression, the SubNamespace name is validated with the "match" regular expression.
#
# "match" namingPolicies can use variables of regexp capture group naming of "root" namingPolicies.
# example:
# root: ^app-(?P<team>.*)
# match: ^app-${team}-.*
# root namespace: app-team1
# compiled match naming policy: ^app-team1-.*
# This feature is provided using https://pkg.go.dev/regexp#Regexp.Expand
namingPolicies: []
Only labels and annotations specified in the configuration file will be inherited.
Be careful that some labels or annotations affect security configurations or the system.
For example, pod-security.kubernetes.io/*
labels control the security capabilities of Pods in a Namespace.
Likewise, Accurate watches only namespace-scope resources specified in the configuration file.
You can edit the Helm Chart values as needed.
<snip>
controller:
config:
# controller.config.labelKeys -- Labels to be propagated to sub-namespaces.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
## https://pkg.go.dev/path#Match
labelKeys: []
# - team
# controller.config.annotationKeys -- Annotations to be propagated to sub-namespaces.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
## https://pkg.go.dev/path#Match
annotationKeys: []
# An example to propagate an annotation for MetalLB
# https://metallb.universe.tf/usage/#requesting-specific-ips
# - metallb.universe.tf/address-pool
# Labels to be propagated to sub-namespaces from SubNamespace resource.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
# https://pkg.go.dev/path#Match
subNamespaceLabelKeys:
- app
# Annotations to be propagated to sub-namespaces from SubNamespace resource.
# It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func.
# https://pkg.go.dev/path#Match
subNamespaceAnnotationKeys:
- foo.bar/baz
# controller.config.watches -- List of GVK for namespace-scoped resources that can be propagated.
# Any namespace-scoped resource is allowed.
watches:
- group: rbac.authorization.k8s.io
version: v1
kind: Role
- group: rbac.authorization.k8s.io
version: v1
kind: RoleBinding
- version: v1
kind: Secret
# controller.config.namingPolicies -- List of nameing policy for SubNamespaces.
# root and match are both regular expressions.
# When a SubNamespace is created in a tree starting from a root namespace and the root namespace's name matches the "root" regular expression, the SubNamespace name is validated with the "match" regular expression.
#
# "match" namingPolicies can use variables of regexp capture group naming of "root" namingPolicies.
# example:
# root: ^app-(?P<team>.*)
# match: ^app-${team}-.*
# root namespace: app-team1
# compiled match naming policy: ^app-team1-.*
# This feature is provided using https://pkg.go.dev/regexp#Regexp.Expand
namingPolicies:
- root: foo
match: foo_.*
- root: bar
match: bar_.*
- root: ^app-(?P<team>.*)
match: ^app-${team}-.*
<snip>
ClusterRoleBindings
A built-in ClusterRole admin
is bound by default to allow accurate-controller
to watch and propagate namespace-scope resources. However, admin
does not contain verbs for ResourceQuota and may not contain custom resources.
If you need to watch and propagate resources not included in admin
ClusterRole, add additional ClusterRole/ClusterRoleBinding to accurate-controller-manager
ServiceAccount.
Set the controller.additionalRBAC.rules
in the Helm Chart values.
The following example Helm chart values is to watch and propagate ResourceQuotas.
<snip>
controller:
additionalRBAC:
# controller.additionalRBAC.rules -- Specify the RBAC rules to be added to the controller.
# ClusterRole and ClusterRoleBinding are created with the names `{{ release name }}-additional-resources`.
# The rules defined here will be used for the ClusterRole rules.
rules:
- apiGroups:
- ""
resources:
- resourcequotas
verbs:
- get
- list
- watch
- create
- patch
- delete
<snip>
Feature Gates
Feature gates are a set of key=value pairs that describe operator features.
You can turn these features on or off using the --feature-gates
command line flag.
Use -h
flag to see a full set of feature gates.
To set feature gates, use the --feature-gates
flag assigned to a list of feature pairs:
--feature-gates=...,DisablePropagateGenerated=false
The following table is a summary of the feature gates that you can set.
- The "Since" column contains the Accurate release when a feature is introduced or its release stage is changed.
- The "Until" column, if not empty, contains the last Accurate release in which you can still use a feature gate.
{{< table caption="Feature gates for features in Alpha or Beta states" sortable="true" >}}
Feature | Default | Stage | Since | Until |
---|---|---|---|---|
DisablePropagateGenerated | false | Alpha | 1.2.0 | 1.3.0 |
DisablePropagateGenerated | true | Beta | 1.3.0 |
Each feature gate is designed for enabling/disabling a specific feature:
DisablePropagateGenerated
: Disable propagating generated resources, which is a feature subject for removal soon.
Deploying Accurate
-
(Optional) Prepare cert-manager
Accurate depends on cert-manager to issue TLS certificate for admission webhooks. If cert-manager is not installed on your cluster, install it as follows:
$ curl -fsLO https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml $ kubectl apply -f cert-manager.yaml
-
Setup Accurate Helm repository
$ helm repo add accurate https://cybozu-go.github.io/accurate/ $ helm repo update
-
Configuration Helm chart values
Read Configurations for details.
-
Install the Accurate Helm chart
$ helm install --create-namespace --namespace accurate accurate accurate/accurate -f values.yaml
Accurate Helm Chart
How to use Accurate Helm repository
You need to add this repository to your Helm repositories:
helm repo add accurate https://cybozu-go.github.io/accurate/
helm repo update
Quick start
Installing cert-manager
$ curl -fsL https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml | kubectl apply -f -
Installing CustomResourceDefinitions (optional)
Accurate does not use the official helm method of installing CRD resources. This is because it makes upgrading CRDs impossible with helm CLI alone. The helm team explain the limitations of their approach here.
The Accurate Helm chart default is to install and manage CRDs with Helm and add annotations preventing Helm from uninstalling the CRD when the Helm release is uninstalled.
The recommended approach is to let helm manage CRDs, but if you want to manage CRDs yourself, now is the time.
$ kubectl apply -k https://github.com/cybozu-go/accurate//config/crd-only/
NOTE:
Since the CRDs contain configuration of conversion webhooks, you may have to tweak the webhook settings if installing the chart using non-standard values.
If you decided to manage CRDs outside of Helm, make sure you set the crds.enabled
Helm value to false
.
Installing the Chart
NOTE:
This installation method requires cert-manager to be installed beforehand.
To install the chart with the release name accurate
using a dedicated namespace(recommended):
$ helm install --create-namespace --namespace accurate accurate accurate/accurate
Specify parameters using --set key=value[,key=value]
argument to helm install
.
Alternatively a YAML file that specifies the values for the parameters can be provided like this:
$ helm install --create-namespace --namespace accurate accurate -f values.yaml accurate/accurate
Values
Key | Type | Default | Description |
---|---|---|---|
controller.additionalRBAC.rules | list | [] | Specify the RBAC rules to be added to the controller. ClusterRole and ClusterRoleBinding are created with the names {{ release name }}-additional-resources . The rules defined here will be used for the ClusterRole rules. |
controller.additionalRBAC.clusterRoles | list | [] | Specify additional ClusterRoles to be granted to the accurate controller. "admin" is recommended to allow the controller to manage common namespace-scoped resources. |
controller.config.annotationKeys | list | [] | Annotations to be propagated to sub-namespaces. It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func. |
controller.config.labelKeys | list | [] | Labels to be propagated to sub-namespaces. It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func. |
controller.config.watches | list | [{"group":"rbac.authorization.k8s.io","kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","kind":"RoleBinding","version":"v1"},{"kind":"Secret","version":"v1"}] | List of GVK for namespace-scoped resources that can be propagated. Any namespace-scoped resource is allowed. |
controller.config.propagateAnnotationKeyExcludes | list | ["*kubernetes.io/*"] | Annotations to exclude when propagating resources. It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func. |
controller.config.propagateLabelKeyExcludes | list | ["*kubernetes.io/*"] | Labels to exclude when propagating resources. It is also possible to specify a glob pattern that can be interpreted by Go's "path.Match" func. |
controller.extraArgs | list | [] | Optional additional arguments. |
controller.replicas | int | 2 | Specify the number of replicas of the controller Pod. |
controller.resources | object | {"requests":{"cpu":"100m","memory":"20Mi"}} | Specify resources. |
controller.terminationGracePeriodSeconds | int | 10 | Specify terminationGracePeriodSeconds. |
webhook.allowCascadingDeletion | bool | false | Enable to allow cascading deletion of namespaces. Accurate webhooks will only allow deletion of a namespace with children if this option is enabled. |
image.pullPolicy | string | nil | Accurate image pullPolicy. |
image.repository | string | "ghcr.io/cybozu-go/accurate" | Accurate image repository to use. |
image.tag | string | {{ .Chart.AppVersion }} | Accurate image tag to use. |
crds.enabled | bool | true | Decides if the CRDs should be installed as part of the Helm installation. |
crds.keep | bool | true | Setting this to true will prevent Helm from uninstalling the CRD when the Helm release is uninstalled. |
installCRDs | bool | true | Controls if CRDs are automatically installed and managed as part of your Helm release. Deprecated: Use crds.enabled and crds.keep instead. |
Generate Manifests
You can use the helm template
command to render manifests.
$ helm template --namespace accurate accurate accurate/accurate
Installing kubectl plugin
kubectl-accurate
is a plugin for kubectl
to make operations of Accurate easy.
It is strongly recommended to install kubectl-accurate
though Accurate can be used without the plugin.
Installing using Krew
Krew is the plugin manager for kubectl command-line tool.
See the documentation for how to install Krew.
$ kubectl krew update
$ kubectl krew install accurate
Installing manually
-
Set
OS
to the operating system nameOS is one of
linux
,windows
, ordarwin
(MacOS).If Go is available,
OS
can be set automatically as follows:$ OS=$(go env GOOS)
-
Set
ARCH
to the operating system nameARCH is one of
amd64
orarm64
.If Go is available,
ARCH
can be set automatically as follows:$ ARCH=$(go env GOARCH)
-
Set
VERSION
to the accurate versionSee the Accurate release page: https://github.com/cybozu-go/accurate/releases
$ VERSION=< The version you want to install >
-
Download the binary and put it in a directory of your
PATH
.The following is an example to install the plugin in
/usr/local/bin
.$ curl -L -sS https://github.com/cybozu-go/accurate/releases/download/$(VERSION)/kubectl-accurate_$(VERSION)_$(OS)_$(ARCH).tar.gz \ | tar xz -C /usr/local/bin kubectl-accurate
-
Check the installation
Run
kubectl accurate -h
and see the output looks like:$ kubectl accurate -h accurate is a subcommand of kubectl to manage Accurate features. Usage: accurate [command] Available Commands: completion generate the autocompletion script for the specified shell help Help about any command list List namespace trees hierarchically namespace namespace subcommand sub sub-namespace command template template subcommand ...
Usage
Accurate can be used imperatively using kubectl
or declaratively with YAML manifests.
The following sections will show you the usage in both ways.
Showing information
Show the hierarchical list of sub-namespaces
Use kubectl accurate list
:
$ kubectl accurate list
root1
root2
⮡sub1
root3
subroot1
⮡sn1
subroot2
Show all template Namespaces
Use kubectl accurate template list
:
$ kubectl accurate template list
template1
template2
⮡reference1
⮡reference2
template3
Show the properties of a Namespace
Use kubectl accurate ns describe
:
$ kubectl accurate ns describe root2
Name: root2
Type: root
# of children: 1
Resources:
Kind Name From Mode
-------- -------- -------- --------
Role role1 tmpl3 create
Secret mysecret create
Setting up templates
Template is a feature of Accurate to propagate labels, annotations, and resources between normal Namespaces.
Any Namespace except for sub-namespaces can reference a template Namespace. So, a template Namespace can reference another template Namespace.
In the following examples, <name>
represents a Namespace name to be changed.
Likewise, <template>
represents a template Namespace name.
Setting a Namespace as a template
Using kubectl accurate
:
$ kubectl accurate ns set-type <name> template
Applying YAML manifests:
apiVersion: v1
kind: Namespace
metadata:
name: <name>
labels:
accurate.cybozu.com/type: template
Reverting a template Namespace to a normal one
Using kubectl accurate
:
$ kubectl accurate ns set-type <name> none
Applying YAML manifests:
Remove accurate.cybozu.com/type
label.
Setting a reference to a template Namespace
Using kubectl accurate
:
$ kubectl accurate template set <name> <template>
Applying YAML manifests:
apiVersion: v1
kind: Namespace
metadata:
name: <name>
labels:
accurate.cybozu.com/template: <template>
Unsetting a reference to a template Namespace
Using kubectl accurate
:
$ kubectl accurate template unset <name>
Applying YAML manifests:
Remove accurate.cybozu.com/template
label.
Propagating resources
Accurate propagates only resources annotated with accurate.cybozu.com/propagate=<mode>
.
The Group/Version/Kind of the resource must be listed in the configuration file.
In the following examples, <mode>
represents either create
or update
.
Read Concepts about the propagation modes.
Annotating a resource for propagation
The following is an example to propagate Secrets.
Using kubectl
:
$ kubectl annotate secrets <name> accurate.cybozu.com/propagate=<mode>
Applying YAML manifests:
apiVersion: v1
kind: Secret
metadata:
namespace: default
name: <name>
annotations:
accurate.cybozu.com/propagate: <mode>
Annotating a resource to propagate resources created from it (DEPRECATED)
[!WARNING]
Propagating generated resources is a deprecated feature and is subject for removal soon. Commonly used tools like cert-manager and sealed-secrets now provide features for adding annotations/labels to resources created from user-facing custom resources. These features can be used for migration to ensure the standardaccurate.cybozu.com/propagate
annotation is added to generated resources.
For example, a Secret created from cert-manager's Certificate can automatically be propagated.
To do this, Certificate should be annotated with accurate.cybozu.com/propagate-generated=<mode>
at the time of creation.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
namespace: default
name: example-cert
annotations:
accurate.cybozu.com/propagate-generated: <mode>
spec:
...
accurate-controller
needs to be able to get Certificate objects.
Sub-namespace operations
Sub-namespaces is a feature of Accurate to allow tenant users to create Namespaces and delete the created Namespaces.
Sub-namespaces can be created under either a root Namespace or a sub-namespace.
In the following examples, <name>
represents a Namespace name to be changed.
Likewise, <parent>
represents a root or another sub-namespace.
Setting a Namespace as a root Namespace
Suppose that Accurate is configured to propagate team
label.
Using kubectl accurate
:
$ kubectl accurate ns set-type <name> root
$ kubectl label ns <name> team=foo
Applying YAML manifests:
apiVersion: v1
kind: Namespace
metadata:
name: <name>
labels:
accurate.cybozu.com/type: root
team: foo
Accurate only propagates labels/annotations that have been configured in that respect via the labelKeys
and annotationKeys
parameters in config.yaml
. This prevents the propagation of labels/annotations that were not meant to do so.
Preparing resources for tenant users
In almost all cases, a root Namespace should have RoleBinding for a group of tenant users.
The RoleBinding should be annotated with accurate.cybozu.com/propagate=update
.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: <name>
name: admin
annotations:
accurate.cybozu.com/propagate: update
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- kind: Group
name: foo
apiGroup: rbac.authorization.k8s.io
You may want to prepare more objects such as ResourceQuotas.
Reverting a root Namespace to a normal one
Using kubectl accurate
:
$ kubectl accurate ns set-type <name> none
Applying YAML manifests:
Remove accurate.cybozu.com/type
label.
Creating a sub-namespace
Using kubectl accurate
:
$ kubectl accurate sub create <name> <parent>
Applying YAML manifests:
apiVersion: accurate.cybozu.com/v2
kind: SubNamespace
metadata:
namespace: <parent>
name: <name>
Creating a sub-namespace with additional labels/annotations
Using kubectl accurate
:
$ kubectl accurate sub create <name> <parent> --labels=foo=bar --annotations=baz=zot
Applying YAML manifests:
apiVersion: accurate.cybozu.com/v2
kind: SubNamespace
metadata:
namespace: <parent>
name: <name>
spec:
labels:
foo: bar
annotations:
baz: zot
You can edit these spec.labels/spec.annotations
with kubectl edit
:
$ kubectl edit SubNamespace -n=<parent> <name>
The spec.labels/spec.annotations
that can be propagated to sub-namespaces can be set with the subNamespaceLabelKeys/subNamespaceAnnotationKeys
parameters in config.yaml.
Deleting a created sub-namespace
Using kubectl accurate
:
$ kubectl accurate sub delete <name>
Applying YAML manifests:
Delete the created SubNamespace object.
Changing the parent of a sub-namespace
Only cluster admins can do this.
Using kubectl accurate
:
$ kubectl accurate sub move <name> <new-parent>
Applying YAML manifests:
apiVersion: v1
kind: Namespace
metadata:
name: <name>
labels:
accurate.cybozu.com/parent: <new-parent>
Converting a normal Namespace to a sub-namespace
Only cluster admins can do this.
Using kubectl accurate
:
$ kubectl accurate sub graft <name> <parent>
Applying YAML manifests:
apiVersion: v1
kind: Namespace
metadata:
name: <name>
labels:
accurate.cybozu.com/parent: <parent>
Converting a sub-namespace to a root Namespace
Only cluster admins can do this.
Using kubectl accurate
:
$ kubectl accurate sub cut <name>
Applying YAML manifests:
apiVersion: v1
kind: Namespace
metadata:
name: <name>
labels:
accurate.cybozu.com/type: root
# and remove accurate.cybozu.com/parent label
Custom Resources
Sub Resources
SubNamespace
SubNamespace is the Schema for the subnamespaces API Deprecated: This type will be removed in one of the next releases.
Field | Description | Scheme | Required |
---|---|---|---|
metadata | metav1.ObjectMeta | false | |
spec | Spec is the spec of SubNamespace. | SubNamespaceSpec | false |
status | Status is the status of SubNamespace. | SubNamespaceStatus | false |
SubNamespaceList
SubNamespaceList contains a list of SubNamespace Deprecated: This type will be removed in one of the next releases.
Field | Description | Scheme | Required |
---|---|---|---|
metadata | metav1.ListMeta | false | |
items | []SubNamespace | true |
SubNamespaceSpec
SubNamespaceSpec defines the desired state of SubNamespace
Field | Description | Scheme | Required |
---|---|---|---|
labels | Labels are the labels to be propagated to the sub-namespace | map[string]string | false |
annotations | Annotations are the annotations to be propagated to the sub-namespace. | map[string]string | false |
Commands
kubectl-accurate
kubectl-accurate
is a kubectl
plugin for Accurate.
Features
- Hierarchical view of namespace trees
- Show the information about a namespace.
- List of propagating/propagated resources in the namespace.
- Root or not.
- Template or not.
- The parent namespace, if it is a sub-namespace.
- The template namespace, if set.
- Operations for root namespaces
- Make an independent namespace to a root namespace.
- Make a root namespace back to an independent namespace, if it has no child sub-namespaces.
- Operations for setting a template namespace
- Operations for sub-namespaces
- Create a sub-namespace under a root namespace or another sub-namespace.
- Deleting a sub-namespace.
- Move a sub-namespace under a different root or sub-namespace.
- Convert an independent namespace to a sub-namespace.
- Convert a sub-namespace to a root namespace.
Generic options
kubectl-accurate
takes the same generic options as kubectl
including:
Flags:
--as string Username to impersonate for the operation
--as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--cache-dir string Default cache directory (default "$HOME/.kube/cache")
--certificate-authority string Path to a cert file for the certificate authority
--client-certificate string Path to a client certificate file for TLS
--client-key string Path to a client key file for TLS
--cluster string The name of the kubeconfig cluster to use
--context string The name of the kubeconfig context to use
--insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
--kubeconfig string Path to the kubeconfig file to use for CLI requests.
-n, --namespace string If present, the namespace scope for this CLI request
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
-s, --server string The address and port of the Kubernetes API server
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
--token string Bearer token for authentication to the API server
--user string The name of the kubeconfig user to use
Note that kubectl-accurate
does not use the namespace given by -n
/ --namespace
flag.
It always take namespace names as positional arguments.
Commands
There is an alias for namespace
sub-command that is ns
.
list [ROOT]
List namespace trees hierarchically.
If ROOT
is given, only the tree starting from ROOT
namespace is shown.
namespace describe NS
Describe the information about a namespace NS
related to Accurate.
namespace set-type NS TYPE
Set the type of a namespace NS
to TYPE
.
Valid types are root
or template
.
To unset the type, specify none
as the type.
template list [TEMPLATE]
List template namespace trees hierarchically. If TEMPLATE is not given, all template namespaces are shown hierarchically. If TEMPLATE is given, only the tree under the TEMPLATE namespace will be shown.
template set NS TEMPLATE
Set TEMPLATE
namespace as the template of NS
namespace.
template unset NS
Unset the template of NS
namespace.
sub create NAME NS
Create a SubNamespace named NAME
in NS
namespace.
After that, Accurate will create a namespace NAME
as a sub-namespace of NS
.
sub delete NAME
Delete a SubNamespace named NAME
in the parent namespace of NAME
namespace.
After that, Accurate will delete NAME
namespace.
sub move NS PARENT
Move a sub-namespace NS
to a different root or sub-namespace.
After that, Accurate will create SubNamespace object in the new parent namespace.
sub graft NS PARENT
Like sub move
, but this converts a non-sub-namespace NS
to a sub-namespace of PARENT
.
NS
must not be a root namespace or have a template.
sub cut NS
Make a sub-namespace NS
a new root namespace.
The child sub-namespaces under NS
will be moved along with it.
Propagated resources with mode update
in NS
will be deleted.
sub list [ROOT]
Alias for kubectl-accurate list
command.
accurate-controller
accurate-controller
is a Kubernetes controller to manage sub-namespaces and
to propagate resources from parents to their children namespaces.
Configuration file
accurate-controller
reads a configuration file on startup.
The default location is /etc/accurate/config.yaml
.
The location can be changed with --config-file
flag.
The configuration file should be a JSON or YAML file having the following keys:
Key | Type | Description |
---|---|---|
labelKeys | []string | Keys of namespace labels to be propagated. |
annotationKeys | []string | Keys of namespace annotations to be propagated. |
subNamespaceLabelKeys | []string | Keys of SubNamespace labels to be propagated. |
subNamespaceAnnotationKeys | []string | Keys of SubNamespace annotations to be propagated. |
watches | []object | GroupVersionKind of namespace-scoped objects to be propagated. |
Example:
labelKeys:
- team
annotationKeys:
- foo.bar/baz
subNamespaceLabelKeys:
- app
subNamespaceAnnotationKeys:
- foo.bar/baz
watches:
- group: rbac.authorization.k8s.io
version: v1
kind: Role
- group: rbac.authorization.k8s.io
version: v1
kind: RoleBinding
- version: v1
kind: Secret
Environment variables
Name | Required | Description |
---|---|---|
POD_NAMESPACE | Yes | The namespace name where accurate-controller is running. |
Command-line flags
Flags:
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--apiserver-qps-throttle int The maximum QPS to the API server. (default 50)
--cert-dir string webhook certificate directory
--config-file string Configuration file path (default "/etc/accurate/config.yaml")
--health-probe-addr string Listen address for health probes (default ":8081")
-h, --help help for accurate-controller
--leader-election-id string ID for leader election by controller-runtime (default "accurate")
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--metrics-addr string The address the metric endpoint binds to (default ":8080")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity
--version version for accurate-controller
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
--webhook-addr string Listen address for the webhook endpoint (default ":9443")
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
Labels used by Accurate
The table below is a list of labels used by Accurate.
Key | Value | Resource | Description |
---|---|---|---|
accurate.cybozu.com/type | template or root | Namespace | The type of namespace. |
accurate.cybozu.com/template | Namespace name | Namespace | The template namespace name. |
accurate.cybozu.com/parent | Namespace name | Namespace | The parent namespace name. |
app.kubernetes.io/created-by | accurate | Copied or propagated resources, sub-namespaces | Informational |
Annotations used by Accurate
The table below is a list of annotations used by Accurate.
Key | Value | Resource | Description |
---|---|---|---|
accurate.cybozu.com/from | Namespace name | Copied or propagated resources | The namespace name from which the source resource was copied. |
accurate.cybozu.com/propagate | "create" or "update" | Namespace-scoped resources | Specify propagation mode. |
accurate.cybozu.com/propagate-generated ⚠️ | "create" or "update" | Namespace-scoped resources | DEPRECATED Specify propagation mode of generated resources. |
accurate.cybozu.com/generated ⚠️ | false | Namespace-scoped resources | DEPRECATED The result of checking if this is generated from another resource. |
Design notes
- Overview
- Why do we need another namespace controller in the first place?
- Goals
- Things to be avoided
- SubNamespaces are not related to parent-child relationships
Overview
Accurate aims to implement functionalities found in Hierarchical Namespace Controller (HNC), but in a different way.
Accurate provides 1) resource-propagation between namespaces, and 2) sub-namespace concept for multi-tenancy. Since we consider resource-propagation alone is highly useful, the feature is available between any namespaces.
Why do we need another namespace controller in the first place?
Some of the HNC designs and specifications contradict our use cases.
- HNC opt-outs resources when propagating them.
- For safety and accuracy, we need opt-in propagation.
- HNC opt-outs root namespaces.
- For easier-maintenance, we need opt-in root namespaces.
- HNC does not propagate namespace labels and annotations.
- We need to propagate some namespace labels/annotations.
Since these are fundamentally different requirements, we decided to develop our own solution.
Goals
- Any namespace-scoped resource can be copied or propagated
- The kinds of resources are given by the configuration file of Accurate.
- Only resources annotated with
accurate.cybozu.com/propagate: <mode>
will be propagated. - Of course, Accurate controller needs to be privileged to manage them.
- Support the following propagation modes:
create
: if the resource does not exist, copy the resource from the parent namespace.update
: if the resource is missing or different from the parent namespace, create or update it. If the parent resource is deleted, the copy will also be deleted.
- ⚠️ Propagate generated resources (DEPRECATED)
- Resources created and controlled by another resource can be automatically propagated.
- The generator resource should be annotated with
accurate.cybozu.com/propagate-generated: <mode>
.
- Propagate labels and annotations of parent or template namespaces
- The label/annotation keys are given through the configuration file of Accurate.
- Only labels/annotations specified in the configuration file of Accurate will be propagated.
- Label/annotation deletions from parent or template namespaces will not be propagated.
- Opt-in root namespaces
- Only namespaces labeled with
accurate.cybozu.com/type: root
can be the root of a namespace tree.
- Only namespaces labeled with
- Tenant users can create and delete sub-namespaces by creating and deleting a custom resource in a root or a sub-namespace.
- If a namespace has one or more sub-namespaces, Accurate prevents the deletion of the namespace - unless allow cascading deletion of namespaces is enabled.
- Template namespace
- Namespaces that are not a sub-namespace can specify a template from which labels, annotations, and resources can be propagated.
- Admins can change the parent namespace of a sub-namespace.
Things to be avoided
Accurate prevents the following problems by a validating admission webhook for Namespace.
- Circular references among namespaces.
- Allowing a sub-namespace to set a template (sub-namespaces should inherit things only from the parent).
- Marking a sub-namespace as a root namespace.
- Deleting
accurate.cybozu.com/type=root
label from root namespaces having one or more sub-namespaces. - Deleting
accurate.cybozu.com/type=template
label from template namespaces having one or more instance namespaces. - Dangling sub-namespaces (sub-namespaces whose parent namespace is missing).
- Dangling instance namespaces (namespaces whose template namespace is missing).
- Changing a sub-namespace to a non-root namespace when it has child sub-namespaces.
Accurate prevents the following problem by a validating admission webhook for SubNamespace.
- Creating a SubNamespace object in a non-root and non-sub- namespace.
No webhooks for propagated resources
Accurate does not use admission webhooks for resources propagated from a parent namespace to its sub-namespaces. The decision was made from the following points:
-
Since Accurate can propagate any namespace-scoped resource, adding an admission webhook for them might cause troubles.
For instance, admission webhooks for Pods may cause chicken-and-egg problem upon bootstrapping.
-
Avoid interrupting users who do not expect limitations from Accurate.
SubNamespaces are not related to parent-child relationships
Accurate does not rely on SubNamespace resources to look up sub-namespaces of a namespace or to find the parent of a sub-namespace.
By doing so, Accurate can easily restructure existing namespaces.
How Accurate reconciles resources
Accurate primarily watches Namespaces and SubNamespaces. In addition, it needs to watch any kind of resources specified in its config file.
SubNamespace (custom resource)
- For a new SubNamespace, Accurate creates a sub-namespace.
- For a deleting SubNamespace, Accurate deletes the sub-namespace if the sub-namespace exists and its
accurate.cybozu.com/parent
is the same asmetadata.namespace
of SubNamespace.
Namespaces
Namespaces that are labeled with accurate.cybozu.com/template
These namespaces reference a template namespace and propagate the labels, annotations, and watched resources from the template namespace.
- Accurate should propagate labels and/or annotations from the template namespace.
- Accurate should create copies of resources in the template namespace whose
accurate.cybozu.com/propagate
annotation iscreate
if they are missing. - Accurate should create or update copies of resources in the template namespace whose
accurate.cybozu.com/propagate
annotation isupdate
if they are missing or different. - Accurate should delete resources in the reconciling namespace that are annotated with
accurate.cybozu.com/propagate=update
provided that:- the value of
accurate.cybozu.com/from
annotation is not the template namespace name, or - there is not a resource of the same kind and the same name in the template namespace.
- the value of
Namespaces w/o accurate.cybozu.com/type
and accurate.cybozu.com/template
labels
If these labels are removed from the Namespace, Accurate should delete propagated resources with mode == update
.
Template namespace
Template namespaces are namespaces labeled with accurate.cybozu.com/type=template
.
- Accurate should propagate labels and/or annotations to namespaces that references the template namespace with
accurate.cybozu.com/template
label.
Root namespace
Root namespaces are namespaces labeled with accurate.cybozu.com/type=root
.
- Accurate should propagate labels and/or annotations to its sub-namespaces.
Sub-namespace
Sub-namespaces are namespaces created by Accurate.
Sub-namespaces have accurate.cybozu.com/parent
label.
- Accurate should propagate labels and/or annotations from the parent namespace.
- Accurate should propagate labels and/or annotations of the reconciling namespace to its sub-namespaces, if any.
- Accurate should create copies of resources in the parent namespace whose
accurate.cybozu.com/propagate
annotation iscreate
if they are missing. - Accurate should create or update copies of resources in the parent namespace whose
accurate.cybozu.com/propagate
annotation isupdate
if they are missing or different. - Accurate should delete resources in the reconciling namespace that are annotated with
accurate.cybozu.com/propagate=update
provided that:- the value of
accurate.cybozu.com/from
annotation is not the parent namespace name, or - there is not a resource of the same kind and the same name in the parent namespace.
- the value of
Watched namespace-scoped resources
Any namespace-scoped resource can be propagated from a template or from a parent namespace.
Resources annotated with accurate.cybozu.com/from
These resources are propagated from a parent or a template namespace. The annotation value is the parent namespace name.
- If the parent resource exists and is annotated with
accurate.cybozu.com/propagate=update
, Accurate compares the resource with the parent resource, and if they differ, updates the resource. - If the resource is annotated with
accurate.cybozu.com/propagate=update
and there isn't a resource of the same kind and the same name in the parent namespace, Accurate deletes the resource.
The last rule is for cases where the parent resource is deleted while the controller is stopped. With this rule, Accurate can delete such orphaned resources when the controller starts.
Resources annotated with accurate.cybozu.com/propagate
These resources can be propagated to other namespaces.
- If the resource exists and the annotation value is
create
, Accurate creates a copy in all sub-namespaces if missing. - If the resource exists and the annotation value is
update
, Accurate creates or updates a copy in all sub-namespaces if missing or different. - When a resource is deleted, Accurate checks sub-namespaces and delete the resource of the same kind and the same name if the resource is annotated with
accurate.cybozu.com/propagate=update
.
Resources owned by another resource that is annotated with accurate.cybozu.com/propagate-generated
(DEPRECATED)
Accurate annotates the resource with accurate.cybozu.com/propagate
.
The annotation value is the same as accurate.cybozu.com/propagate-generated
annotation.
Release procedure
This document describes how to release a new version.
Labeling
Release notes are automatically generated based on PRs included in the release.
Those PRs are categorized based on the label assigned to them.
Please refer to .github/release.yml
for the kind of labels.
Versioning
Follow semantic versioning 2.0.0 to choose the new version number.
Bump version
-
Determine a new version number. Then set
VERSION
variable.# Set VERSION and confirm it. It should not have "v" prefix. $ VERSION=x.y.z $ echo $VERSION
-
Add a git tag to the main HEAD, then push it.
$ git checkout main $ git tag -a -m "Release v$VERSION" "v$VERSION" $ git tag -ln | grep $VERSION $ git push origin v$VERSION
Maintenance
How to update supported Kubernetes
Accurate supports the three latest Kubernetes versions. If a new Kubernetes is released, please update the following files.
- Update Kubernetes version in
e2e/Makefile
and.github/workflows/ci.yaml
. - Update kubectl version in
aqua.yaml
. - Update
k8s.io/*
andsigs.k8s.io/controller-runtime
packages version ingo.mod
.
If Kubernetes or controller-runtime API has changed, please fix the relevant source code.
How to update dependencies
Renovate will create PRs that update dependencies once a week. However, Kubernetes is only updated with patched versions.