Monitoring

Basically we use Prometheus for infrastructure and service monitoring. Prometheus rans as k8s operator and discover metrics automaticaly. Sometimes we need exporters.

Auto-discover metrics

Prometheus use ServiceMonitor (kind) for discover services in k8s cluster. If you use our standard chart, service monitor was generated automaticaly.

The Prometheus Operator introduces additional resources in Kubernetes to declare the desired state of a Prometheus and Alertmanager cluster as well as the Prometheus configuration. The resources it introduces are:

• Prometheus
• Alertmanager
• ServiceMonitor

The Prometheus resource declaratively describes the desired state of a Prometheus deployment, while a ServiceMonitor describes the set of targets to be monitored by Prometheus.

The Prometheus resource includes a field called serviceMonitorSelector, which defines a selection of ServiceMonitors to be used. By default and before the version v0.19.0, ServiceMonitors must be installed in the same namespace as the Prometheus instance. With the Prometheus Operator v0.19.0 and above, ServiceMonitors can be selected outside the Prometheus namespace via the serviceMonitorNamespaceSelector field of the Prometheus resource.

• namespace must be set to monitoring, it's namespace where prometheus located
• lables need to had release: kube-prometheus-stack (serviceMonitorSelector configuration of prometheus)

Example

service-monitor.yml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ include "service.fullname" . }}-monitor
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      {{- include "service.selectorLabels" . | nindent 6 }}
  endpoints:
  - port: metrics
    scheme: http
    interval: 15s
  namespaceSelector:
    matchNames:
    - {{ .Release.Namespace }}

Exporters

@todo