Metrics

moco-controller
- MySQL clusters
- Backup
MySQL instance
Scrape rules

moco-controller

moco-controller provides the following kind of metrics in Prometheus format. Aside from the standard Go runtime and process metrics, it exposes metrics related to controller-runtime, MySQL clusters, and backups.

MySQL clusters

All these metrics are prefixed with moco_cluster_ and have name and namespace labels.

Name	Description	Type
`checks_total`	The number of times MOCO checked the cluster	Counter
`errors_total`	The number of times MOCO encountered errors when managing the cluster	Counter
`available`	1 if the cluster is available, 0 otherwise	Gauge
`healthy`	1 if the cluster is running without any problems, 0 otherwise	Gauge
`switchover_total`	The number of times MOCO changed the live primary instance	Counter
`failover_total`	The number of times MOCO changed the failed primary instance	Counter
`replicas`	The number of mysqld instances in the cluster	Gauge
`ready_replicas`	The number of ready mysqld Pods in the cluster	Gauge
`current_replicas`	The number of current replicas	Gauge
`updated_replicas`	The number of updated replicas	Gauge
`last_partition_updated`	The timestamp of the last successful partition update	Gauge
`clustering_stopped`	1 if the cluster is clustering stopped, 0 otherwise	Gauge
`reconciliation_stopped`	1 if the cluster is reconciliation stopped, 0 otherwise	Gauge
`errant_replicas`	The number of mysqld instances that have errant transactions	Gauge
`processing_time_seconds`	The length of time in seconds processing the cluster	Histogram
`volume_resized_total`	The number of successful volume resizes	Counter
`volume_resized_errors_total`	The number of failed volume resizes	Counter
`statefulset_recreate_total`	The number of successful StatefulSet recreates	Counter
`statefulset_recreate_errors_total`	The number of failed StatefulSet recreates	Counter

Backup

All these metrics are prefixed with moco_backup_ and have name and namespace labels.

Name	Description	Type
`timestamp`	The number of seconds since January 1, 1970 UTC of the last successful backup	Gauge
`elapsed_seconds`	The number of seconds taken for the last backup	Gauge
`dump_bytes`	The size of compressed full backup data	Gauge
`binlog_bytes`	The size of compressed binlog files	Gauge
`workdir_usage_bytes`	The maximum usage of the working directory	Gauge
`warnings`	The number of warnings in the last successful backup	Gauge

MySQL instance

For each mysqld instance, moco-agent exposes a set of metrics. Read github.com/cybozu-go/moco-agent/blob/main/docs/metrics.md for details.

Also, if you give a set of collector flag names to spec.collectors of MySQLCluster, a sidecar container running mysqld_exporter exposes the collected metrics for each mysqld instance.

Scrape rules

This is an example kubernetes_sd_config for Prometheus to collect all MOCO & MySQL metrics.

scrape_configs:
- job_name: 'moco-controller'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_app_kubernetes_io_component,__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: moco-system;moco-controller;metrics

- job_name: 'moco-agent'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name,__meta_kubernetes_pod_container_port_name,__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name]
    action: keep
    regex: mysql;agent-metrics;moco-.*
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace

- job_name: 'moco-mysql'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name,__meta_kubernetes_pod_container_port_name,__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name]
    action: keep
    regex: mysql;mysqld-metrics;moco-.*
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
    action: replace
    target_label: name
  - source_labels: [__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name]
    action: replace
    target_label: index
    regex: .*-([0-9])
  - source_labels: [__meta_kubernetes_pod_label_moco_cybozu_com_role]
    action: replace
    target_label: role

The collected metrics should have these labels:

namespace: MySQLCluster's metadata.namespace
name: MySQLCluster's metadata.name
index: The ordinal of MySQL instance Pod