Health and Metrics

Asset Core daemons expose health endpoints for monitoring and Prometheus metrics for observability. These signals are designed to show readiness, freshness, and lag so operators can catch issues before they impact users.

Overview

Both write and read daemons provide:

Health endpoint: JSON status for load balancers and monitoring
Metrics endpoint: Prometheus-format metrics for dashboards and alerting (disabled by default, served on a dedicated listener)

Structure

Health Endpoints

Daemon	Endpoint	Description
Write	`GET /v1/write/health`	Status, version, and uptime
Read	`GET /v1/read/health`	Readiness plus freshness for a namespace (uses `x-assetcore-namespace` when not configured)

Metrics Endpoints

Daemon	Endpoint	Format
Write	`GET /v1/write/metrics`	Prometheus text (on metrics listener)
Read	`GET /v1/read/metrics`	Prometheus text (on metrics listener)

Metrics are disabled by default. Enable them in daemon-write ([logging] metrics_enabled) and daemon-read ([observability] metrics_enabled), set distinct metrics_address values, and optionally allowlist principals for /v1/write/metrics and /v1/read/metrics. Plan to scrape these endpoints early in production so you can see freshness lag trends.

Fields

Health Response

{
  "status": "ready",
  "freshness": {
    "namespace": 1,
    "world_seq": 128,
    "commit_log_world_seq": 130,
    "lag": 2,
    "lag_ms": 250
  }
}

Field	Description
`status`	Read: `starting`, `ready`, `degraded`; Write: `healthy`
`version`	Write-only daemon version string
`api_version`	Write-only API version string
`build_git_sha`	Write-only Git commit SHA when available
`uptime_secs`	Write-only seconds since process start
`freshness`	Read-only freshness metadata for the namespace

Key Metrics

Write Daemon

Metric	Type	Description
`ingress_requests_total`	Counter	Total ingress requests by route/outcome
`ingress_request_duration_seconds`	Histogram	Ingress request latency
`commit_duration_seconds`	Histogram	End-to-end commit latency
`ingress_queue_depth`	Gauge	Items waiting in queue
`ingress_inflight`	Gauge	Requests being processed
`commit_log_end_seq`	Gauge	Latest committed sequence
`commit_log_driver_lag_events`	Gauge	Commit log driver lag in events

Read Daemon

Metric	Type	Description
`http_requests_total`	Counter	Total HTTP requests by route/outcome
`http_request_duration_seconds`	Histogram	HTTP request latency
`read_checkpoint_seq`	Gauge	Applied sequence
`read_commit_log_end_seq`	Gauge	Observed commit log sequence
`freshness_lag_events`	Gauge	Events behind commit log
`freshness_lag_ms`	Gauge	Milliseconds behind commit log
`snapshot_publish_duration_seconds`	Histogram	Snapshot publish time

Examples

Check Write Health

curl -H "Authorization: Bearer $ASSETCORE_ADMIN_TOKEN" \
  http://localhost:8080/v1/write/health | jq .

Check Read Freshness

curl -H "Authorization: Bearer $ASSETCORE_READ_TOKEN" \
  http://localhost:8081/v1/read/namespaces/5001/freshness | jq .

Scrape Metrics

Add to your Prometheus configuration:

scrape_configs:
  - job_name: 'assetcore-write'
    static_configs:
      - targets: ['localhost:9100']
    metrics_path: '/v1/write/metrics'

  - job_name: 'assetcore-read'
    static_configs:
      - targets: ['localhost:9101']
    metrics_path: '/v1/read/metrics'

Example Prometheus Queries

Request rate:

rate(ingress_requests_total[5m])

P99 latency:

histogram_quantile(0.99, rate(ingress_request_duration_seconds_bucket[5m]))

Read daemon lag:

freshness_lag_events

Error rate:

rate(http_requests_total{outcome!="success"}[5m])

Example Alerts

groups:
  - name: assetcore
    rules:
      - alert: HighReadLag
        expr: freshness_lag_events > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Read daemon is behind commit log"

      - alert: WriteQueueFull
        expr: ingress_queue_depth > 900
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Write queue approaching capacity"

Deployment Basics - Running daemons
Freshness and Replay - Understanding lag
HTTP API - Endpoint reference