Skip to content
Esc

Monitoring

Metrics history and charts, Needs attention, running operations, Prometheus metrics and health endpoints.

Dockmesh keeps a history of host and container metrics for its own charts, lists what needs attention, shows running work in the top bar, and exposes Prometheus metrics and health endpoints for external monitoring.

Every 30 seconds Dockmesh samples CPU, memory, network and block I/O of each container, and CPU, memory and disk of the host. The server records its own host; agents record theirs on the remote host and send the samples to the server every minute. An agent keeps unsent samples for up to 30 minutes, so a short disconnect or a server restart leaves no gap. Agents older than the server keep working with live values only, and their charts say so.

The samples are stored in the server’s SQLite database:

ResolutionKept for
raw (30 s)24 hours
1 minute30 days
1 hour365 days

These periods are fixed. The history lives in the main database file, which grows with the number of containers. SQLite doesn’t give freed space back on its own, so the file doesn’t shrink after old samples are deleted.

WhereWhat
Container page, Overview tabCPU, memory, network and block I/O of the container, with markers for the last start, image updates and rollbacks
Host page (Hosts → a host)CPU, memory and disk of the host, plus Top consumers: the containers using the most CPU or memory

Pick a range from live (streamed values) to 1 h, 6 h, 24 h, 7 d or 30 d. Drag across a chart to zoom, double-click to reset.

  • CPU is a share of the whole host: 100 % means every core is busy. The API also returns the per-core value (cpu_percent_core, 100 per busy core, as docker stats shows it).
  • Container memory is relative to the container’s limit, or to the host’s RAM when it has no limit, and excludes page cache, as docker stats shows it on cgroup v2.
  • The overview and the host page list disk usage for every real filesystem of a host, with the mounts holding Docker, containerd, stacks, Dockmesh data and local backups labelled. The host’s disk figure is the fullest of the Docker-related mounts.

Reading container metrics needs containers.view, host metrics need hosts.view, both on the host. The API is GET /api/v1/containers/{id}/metrics, GET /api/v1/hosts/{id}/metrics and GET /api/v1/hosts/{id}/stats/containers.

Needs attention collects what someone should look at. The overview lists the five most severe items, and View all opens the full list under Activity → Needs attention.

ItemSeverity
Offline hostcritical
Container that keeps restartingcritical
Unhealthy container, container that exited with an errorwarning
Backup job whose last run failedcritical
Firing alertthe rule’s severity
Proxy certificate that expires within 30 dayswarning, critical within 7 days
Open warning or critical noticethe notice’s severity
Agents on a different build than the server (one item)info
Available image updatesinfo

Items are sorted most severe first. Each source only appears for users who hold its view permission, and items are filtered to the hosts and stacks in the user’s scope. The API is GET /api/v1/attention.

Image updates, backup runs, restores and new proxy sites run as operations with steps and a live log. The operations indicator in the top bar lists running and recently finished operations; click one to open its progress from any page. The progress dialog can be closed while the work continues. A toast tells you when your own operation finishes, and a failed one shows the step it failed at with its last log lines.

Running work can be cancelled from its progress dialog and from the backup and update lists:

OperationCan be cancelled
Backup runIn any step. Post-backup hooks still run, and the unfinished archive is deleted from the target.
Image updateWhile it pulls
RestoreWhile it downloads or verifies the archive
New proxy siteDuring the certificate wait; the site stays

A cancelled operation names the step it stopped in. Cancelling needs the permission that started the work.

Operations are kept for 7 days. Credentials are masked before anything is stored or shown, and you only see operations on hosts and stacks in your scope. Work still running when the server stops is marked failed on the next start. The API is GET /api/v1/operations, GET /api/v1/operations/{id} and POST /api/v1/operations/{id}/cancel; updates arrive live on /ws/events.

The server serves Prometheus metrics at /metrics on its HTTP port (not under /api/v1).

By default the endpoint needs a bearer token with the metrics.view permission on everything. Create a custom role with only Settings & audit: Metrics, create an API token restricted to that role on everything, and give it to Prometheus. The Metrics level grants nothing but the scrape; it doesn’t read the audit log:

yaml
scrape_configs:
  - job_name: dockmesh
    scheme: https
    authorization:
      type: Bearer
      credentials: dmt_…
    static_configs:
      - targets: ["dockmesh.example.com"]

Set DOCKMESH_METRICS_AUTH=false in the server environment to serve /metrics without authentication. Only do that when the port isn’t reachable from untrusted networks.

MetricTypeLabels
dockmesh_api_requests_totalcountermethod, path (route pattern), status
dockmesh_audit_entries_totalcounteraction
dockmesh_alerts_fired_totalcounterseverity, rule
dockmesh_hosts_totalgaugestatus (local for the server, then the agents by status)
dockmesh_agent_last_seen_secondsgaugeagent (name), host (agent ID)
dockmesh_backup_last_run_timestamp_secondsgaugejob
dockmesh_backup_last_duration_secondsgaugejob
Go runtime (go_*) and process (process_*) metrics

The counters start at zero when the server starts; the gauges are refreshed every 30 seconds. The backup gauges describe each job’s newest successful run (with or without warnings). dockmesh_stacks_total, dockmesh_containers_total, dockmesh_deploy_duration_seconds and dockmesh_agent_rtt_seconds are registered but have no values, so they don’t show up in a scrape.

Two useful alerts:

yaml
- alert: DockmeshAgentStale
  expr: dockmesh_agent_last_seen_seconds > 120
  for: 5m
- alert: DockmeshBackupStale
  expr: time() - dockmesh_backup_last_run_timestamp_seconds > 2 * 86400
EndpointAuthResponse
GET /healthz/livenone200 ok while the process runs
GET /healthz/readynone200 ready when the database answers. 503 if it doesn’t, or while the server is shutting down
GET /api/v1/healthnoneJSON with status, version and docker (whether the Docker daemon is connected)
GET /api/v1/system/healthsystem.viewChecks shown in the UI: system backup, reverse proxy, agents, disk, Docker daemon, CVE scanner

Use /healthz/ready for load balancers and container health checks, and /healthz/live for liveness probes. During shutdown, /healthz/ready returns 503 with Retry-After: 10, so a load balancer can stop sending traffic before the process exits.

See also Alerts, Notifications, Ports and Troubleshooting.