A monitoring stack that watches your own fleet

What this builds

One small instance that scrapes every other machine you own, over a private tunnel, and sends you mail when something is about to break. Node metrics, blackbox probes between sites, alert rules with sensible thresholds, and an Alertmanager configuration that groups notifications instead of sending you four hundred of them at three in the morning.

There is no dashboard in this build. Dashboards are pleasant and they are not monitoring; a graph nobody is looking at has never woken anybody up. Add one later if you want it, on top of the same data.

Before you start

• An R-4 for the monitor. A hundred nodes at fifteen-second intervals is a few hundred thousand samples a minute, which this handles without noticing.
• Put it somewhere your fleet is not. Any monitor sharing a building with everything it watches stays admirably quiet on precisely the day you need it to shout. If your estate is European, Toronto or Singapore is the sensible place for it.
• A tunnel to each monitored node, from the WireGuard guide. Exporters expose a great deal about a machine and belong nowhere near a public interface.

1. On every monitored node

apt update && apt install -y prometheus-node-exporter
mkdir -p /etc/systemd/system/prometheus-node-exporter.service.d
cat > /etc/systemd/system/prometheus-node-exporter.service.d/bind.conf <<EOF
[Service]
Environment=ARGS=--web.listen-address=10.7.0.11:9100 --collector.systemd --collector.processes
EOF
systemctl daemon-reload && systemctl restart prometheus-node-exporter
ss -ltn | grep 9100

Substitute each node’s own tunnel address. Binding to the tunnel rather than to everything means the firewall is a second line of defence rather than the only one.

2. On the monitor

apt install -y prometheus prometheus-alertmanager prometheus-blackbox-exporter

Write /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    fleet: main

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['127.0.0.1:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: nodes
    file_sd_configs:
      - files: ['/etc/prometheus/targets/*.yml']

  - job_name: probe_https
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://app.example.com/health
          - https://cloud.example.com/status.php
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

File-based discovery rather than a static list is worth the extra directory. Adding a node becomes one file, and the file can be written by whatever provisions your machines. /etc/prometheus/targets/ams.yml:

- targets: ['10.7.0.11:9100']
  labels: { site: AMS-01, role: web }
- targets: ['10.7.0.12:9100']
  labels: { site: AMS-01, role: db }

Those labels are what make alerts readable later. An alert saying db in AMS-01 is actionable; one saying 10.7.0.12 sends you to a spreadsheet.

3. Rules worth having

/etc/prometheus/rules/fleet.yml:

groups:
  - name: fleet
    rules:
      - alert: NodeDown
        expr: up{job="nodes"} == 0
        for: 3m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.instance }} in {{ $labels.site }} stopped answering"

      - alert: DiskFillingUp
        expr: predict_linear(node_filesystem_avail_bytes{fstype=~"ext4|xfs|zfs|btrfs"}[6h], 4*3600) < 0
        for: 30m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.instance }} fills {{ $labels.mountpoint }} within four hours"

      - alert: MemoryPressure
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 15m
        labels: { severity: warn }

      - alert: IOWaitHigh
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m])) > 0.25
        for: 20m
        labels: { severity: warn }

      - alert: CertificateExpiring
        expr: probe_ssl_earliest_cert_expiry - time() < 10*86400
        labels: { severity: warn }

      - alert: ProbeFailing
        expr: probe_success == 0
        for: 5m
        labels: { severity: page }

predict_linear is the rule that earns this build its keep. A threshold on free space tells you a disk is full; a six-hour trend extrapolated forward tells you it will be full at about four this afternoon, which is a problem you can still solve calmly. Disk alerts based on a fixed percentage are either noisy on small volumes or useless on large ones.

4. Alertmanager

route:
  receiver: mail
  group_by: [alertname, site]
  group_wait: 45s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: [severity="warn"]
      repeat_interval: 72h

inhibit_rules:
  - source_matchers: [alertname="NodeDown"]
    target_matchers: [severity="warn"]
    equal: [instance]

receivers:
  - name: mail
    email_configs:
      - to: [email protected]
        from: [email protected]
        smarthost: mail.example.com:587
        auth_username: [email protected]
        auth_password: "<the mailbox password>"

The inhibit rule is the difference between one message and forty. When a node goes down, every warning about that node is suppressed, because you already know: the node is down.

Grouping by alertname and site means an entire site losing power produces one mail listing every machine, rather than one mail per machine. That mail server is the one from the mail guide, and having your alerting path depend on infrastructure you also monitor is a known trade; a second receiver pointing at a webhook somewhere else is cheap insurance.

systemctl enable --now prometheus prometheus-alertmanager prometheus-blackbox-exporter

Retention is worth setting deliberately. Ninety days of a hundred nodes is a few tens of gigabytes:

echo 'ARGS="--storage.tsdb.retention.time=90d --storage.tsdb.retention.size=40GB"' > /etc/default/prometheus
systemctl restart prometheus

Verify it

Syntax first, because a rules file with a typo fails silently at load and you find out during the incident:

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/fleet.yml

Then confirm every target is being scraped:

curl -s http://127.0.0.1:9090/api/v1/targets | grep -o "\"health\":\"[a-z]*\"" | sort | uniq -c

Every target should report up. One reporting down is either a firewall rule or an exporter bound to the wrong address.

Now the only test that proves the whole chain works, which is to break something on purpose. On any monitored node:

systemctl stop prometheus-node-exporter

Three minutes later the alert should be pending, then firing. Watch it move:

curl -s http://127.0.0.1:9090/api/v1/alerts | head -c 400
amtool --alertmanager.url=http://127.0.0.1:9093 alert query

Within about a minute of firing, mail arrives. If the alert fires but no mail appears, the problem is the smarthost credentials, and journalctl -u prometheus-alertmanager will say so plainly. Start the exporter again and confirm you receive the resolved notification as well, since an alerting system that never tells you things are better trains you to ignore it.

systemctl start prometheus-node-exporter

Afterwards

Add blackbox probes between sites, from each site to each other site, and you get a latency matrix of your own estate that is far more useful during an incident than any status page. Ours is at status, and the looking glass answers the other half of the question when a route looks wrong.