Metrics

moco-controller

moco-controller provides the following kind of metrics in Prometheus format. Aside from the standard Go runtime and process metrics, it exposes metrics related to controller-runtime, MySQL clusters, and backups.

MySQL clusters

All these metrics are prefixed with moco_cluster_ and have name and namespace labels.

NameDescriptionType
checks_totalThe number of times MOCO checked the clusterCounter
errors_totalThe number of times MOCO encountered errors when managing the clusterCounter
available1 if the cluster is available, 0 otherwiseGauge
healthy1 if the cluster is running without any problems, 0 otherwiseGauge
switchover_totalThe number of times MOCO changed the live primary instanceCounter
failover_totalThe number of times MOCO changed the failed primary instanceCounter
replicasThe number of mysqld instances in the clusterGauge
ready_replicasThe number of ready mysqld Pods in the clusterGauge
errant_replicasThe number of mysqld instances that have errant transactionsGauge

Backup

All these metrics are prefixed with moco_backup_ and have name and namespace labels.

NameDescriptionType
timestampThe number of seconds since January 1, 1970 UTC of the last successful backupGauge
elapsed_secondsThe number of seconds taken for the last backupGauge
dump_bytesThe size of compressed full backup dataGauge
binlog_bytesThe size of compressed binlog filesGauge
workdir_usage_bytesThe maximum usage of the working directoryGauge
warningsThe number of warnings in the last successful backupGauge

MySQL instance

For each mysqld instance, moco-agent exposes a set of metrics. Read github.com/cybozu-go/moco-agent/blob/main/docs/metrics.md for details.

Also, if you give a set of collector flag names to spec.collectors of MySQLCluster, a sidecar container running mysqld_exporter exposes the collected metrics for each mysqld instance.

Scrape rules

This is an example kubernetes_sd_config for Prometheus to collect all MOCO & MySQL metrics.

scrape_configs:
- job_name: 'moco-controller'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_app_kubernetes_io_component,__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: moco-system;moco-controller;metrics

- job_name: 'moco-agent'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name,__meta_kubernetes_pod_container_port_name,__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name]
    action: keep
    regex: mysql;agent-metrics;moco-.*
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace

- job_name: 'moco-mysql'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name,__meta_kubernetes_pod_container_port_name,__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name]
    action: keep
    regex: mysql;mysqld-metrics;moco-.*
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
    action: replace
    target_label: name
  - source_labels: [__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name]
    action: replace
    target_label: index
    regex: .*-([0-9])
  - source_labels: [__meta_kubernetes_pod_label_moco_cybozu_com_role]
    action: replace
    target_label: role

The collected metrics should have these labels:

  • namespace: MySQLCluster's metadata.namespace
  • name: MySQLCluster's metadata.name
  • index: The ordinal of MySQL instance Pod