Arxi Decision Gate Asset Core
Docs
Arxi Docs

Proof recording and tamper-evident evidence documentation.

Other product docs

Arxi Sidecar Architecture

Audience: Engineers implementing or operating arxi-sidecar as the network-accessible evidence recording service boundary.

Table of Contents

  1. Executive Overview
  2. Runtime Composition
  3. HTTP Surface and Routing
  4. Security and Resource Controls
  5. Idempotency and Retry Safety
  6. Transport and Lifecycle
  7. Container Packaging and Docker Operations
  8. Validation and Test Coverage
  9. File-by-File Cross Reference

Executive Overview

arxi-sidecar is the Arxi HTTP/JSON service runtime that exposes recorder, query, and bundle workflows over Unix socket and/or TCP listeners.

The sidecar composes RecorderEngine, BundleBuilder, Verifier, and a single SqliteStore instance under explicit middleware gates for correlation identity, authentication, enterprise identity/policy, bounds enforcement, idempotency, and timeout control.

F:crates/arxi-sidecar/src/server.rs L42-L133 F:crates/arxi-sidecar/src/state.rs L29-L56 F:crates/arxi-sidecar-config/src/config.rs

Runtime Composition

Startup flow:

  1. Load and validate TOML config (config_version, api.major_version, recorder, transport, security, signer, and limits).
  2. Initialize tracing and open SqliteStore.
  3. Build recorder runtime components.
  4. Initialize persistent idempotency manager on store_meta.
  5. Build axum router with middleware stack.
  6. Bind configured transport listeners and serve until cancellation.

F:crates/arxi-sidecar-config/src/config.rs F:crates/arxi-sidecar-config/src/validation.rs F:crates/arxi-sidecar/src/server.rs L43-L247 F:crates/arxi-sidecar/src/idempotency.rs L61-L138

HTTP Surface and Routing

The router currently exposes:

  • Probes: GET /health, GET /startup, GET /ready
  • Segment lifecycle: POST /v1/segments, POST /v1/segments/seal, GET /v1/segments, GET /v1/segments/active
  • Envelope ingest: POST /v1/envelopes, POST /v1/envelopes/with-attachments
  • Query: POST /v1/query/envelopes
  • Bundle workflows: POST /v1/bundles/build, POST /v1/bundles/verify, POST /v1/bundles/inspect
  • Runtime inspection: GET /v1/config

F:crates/arxi-sidecar/src/server.rs L135-L160

Health behavior:

  • GET /health is a lightweight liveness probe returning process metadata.
  • GET /startup is a startup-completion probe returning startup metadata, including configured startup_verification_depth.
  • GET /ready returns readiness state with dependency-aware fail-closed semantics:
    • 503 with reason=storage_unavailable when store access fails.
    • Optional 503 with reason=request_capacity_exhausted when probes.ready_fail_on_admission_saturation=true and admission/execution semaphores are exhausted.
    • Optional 503 with reason=enterprise_control_unavailable when probes.readiness_mode=storage_and_enterprise, enterprise mode is enabled, and control-plane readiness checks fail.
    • 200 with ready=true in steady state.

F:crates/arxi-sidecar/src/handlers/health.rs L42-L94

Security and Resource Controls

Config Gate

Config validation is fail-closed and enforces:

  • strict schema and API versioning,
  • recorder ID and auto-seal validity,
  • transport host/port constraints,
  • non-loopback transport requires security.mode=token and keeps require_token_for_non_loopback=true (fail-closed),
  • token file hardening (security.token_file cannot be a symlink, must be a regular file, and on Unix must not grant group/other permissions),
  • signer key parseability,
  • request/header/idempotency numeric limits and hard cap checks,
  • probe config fail-closed semantics: probes.enterprise_health_path must be absolute/safe and probes.readiness_mode=storage_and_enterprise requires enterprise mode not disabled.

F:crates/arxi-sidecar-config/src/config.rs F:crates/arxi-sidecar-config/src/validation.rs

Middleware Stack

Middleware order and controls:

  1. Request body limit layer (max_request_body_bytes).
  2. Tracing layer.
  3. Correlation middleware (server correlation IDs, sanitized client x-correlation-id, legacy x-request-id compatibility).
  4. Auth middleware (Bearer token mode with constant-time compare and RFC 6750 challenge metadata on 401 responses).
  5. Enterprise identity middleware (x-tenant-id, x-principal-id, x-authn-method, optional x-namespace-id, optional writer lease assertion) for enterprise-enabled modes.
  6. Enterprise policy middleware (control-plane ingress checks for protected routes plus sidecar boundary hook emission at admission and result phases).
  7. Bounds middleware (header bytes, content negotiation, content-length checks, bounded admission queue, bounded active concurrency, request timeout).

Probe-priority behavior:

  • /health, /startup, and /ready are exempted from auth, enterprise policy, and bounds gating so liveness/startup/readiness state remains observable under overload or policy failures.

F:crates/arxi-sidecar/src/server.rs L155-L160 F:crates/arxi-sidecar/src/middleware/request_id.rs L15-L30 F:crates/arxi-sidecar/src/middleware/auth.rs L21-L84 F:crates/arxi-sidecar/src/control_plane_bridge.rs F:crates/arxi-sidecar/src/middleware/enterprise.rs F:crates/arxi-sidecar/src/middleware/bounds.rs L21-L172

Enterprise Boundary Controls

When enterprise.mode is enabled, the sidecar adds explicit control-plane boundary enforcement while preserving recorder/evidence semantics:

  • crates/arxi-sidecar/src/control_plane_bridge.rs is a boundary adapter only. It translates sidecar ingress identities/operations into control-plane admission and hook calls; it does not implement or alter Arxi evidence math.

  • Enterprise config is fail-closed: enabled mode requires token security mode, control-plane URL/token configuration, and hardened token file checks.

  • Enterprise policy gate calls POST /v1/enterprise/ingress/check before protected sidecar operations.

  • Managed-cloud mode enforces writer-lease assertion for mutating operations when require_writer_lease_header=true.

  • Sidecar emits deterministic enterprise boundary hooks to POST /v1/enterprise/sidecar/hooks at:

    • admission (allow/deny/error before handler execution),
    • result (allow/deny/error after handler response status).

  • Hook emission is fail-closed for successful protected operations: if enterprise hook export fails, the sidecar returns an error instead of silently accepting an unaudited success path.

  • Enterprise readiness dependency checks use probes.enterprise_health_path (default /health) against the configured enterprise control-plane base URL when probes.readiness_mode=storage_and_enterprise.

F:crates/arxi-sidecar-config/src/validation.rs F:crates/arxi-sidecar/src/control_plane_bridge.rs F:crates/arxi-sidecar/src/middleware/enterprise.rs

Error Contract

Errors are returned as RFC 9457 Problem Details (application/problem+json) with stable error codes, retryability metadata, and server/client correlation fields.

Security posture details:

  • malformed JSON rejections are normalized to typed bad_request problem details instead of leaking framework/plaintext extractor failures,
  • server-side (5xx) responses suppress internal detail fields to avoid leaking storage/runtime internals while preserving diagnostics in service logs,
  • 401 unauthorized responses include RFC 6750 WWW-Authenticate challenges.

F:crates/arxi-sidecar/src/error.rs L13-L229 F:crates/arxi-sidecar/src/extract.rs

Idempotency and Retry Safety

Mutating endpoints use persistent idempotency state via SqliteStore::store_meta keys scoped by (recorder_id, method, path, idempotency_key) and canonical JCS request hashes.

Behavior:

  • same key + same hash: replay prior status/body,
  • same key + different hash: 409 idempotency_conflict,
  • storage/index failure: fail-closed idempotency_unavailable.

Retention is bounded by TTL and idempotency_max_entries with deterministic eviction.

F:crates/arxi-sidecar/src/idempotency.rs L20-L316

Transport and Lifecycle

Transport modes:

  • Unix domain socket,
  • TCP listener,
  • both simultaneously.

Runtime lifecycle:

  • stale Unix socket cleanup before bind (socket-type-validated fail-closed),
  • signal-driven cancellation token,
  • graceful shutdown per listener with enforced shutdown_drain_seconds timeout,
  • Unix socket removal on shutdown path.

F:crates/arxi-sidecar/src/server.rs L163-L262 F:crates/arxi-sidecar/src/shutdown.rs L8-L34

Operational command surface in binary:

  • normal service startup with --config / ARXI_SIDECAR_CONFIG,
  • healthcheck command supporting TCP URL and Unix-socket probe modes.

F:crates/arxi-sidecar/src/main.rs L18-L175

Container Packaging and Docker Operations

The repository now ships a first-party sidecar container profile under docker/sidecar/:

  • Dockerfile: multi-stage Rust build to distroless nonroot runtime image.
  • docker-compose.yml: local operator profile with explicit UID/GID mapping, bind-mounted config/data/run paths, healthcheck, and bounded port mapping.
  • config/sidecar.toml: container-oriented baseline config with token-mode auth required for non-loopback transport.
  • .env.example: canonical environment variable contract for image/tag/paths and UID/GID mapping.

Operator contract:

  • mount token_file as a regular non-symlink file with owner-only permissions (0600 on Unix),
  • run non-root with host UID/GID mapping so strict token checks remain readable while avoiding root execution,
  • treat docker/sidecar/config/token as secret material and keep it out of VCS.

F:docker/sidecar/Dockerfile F:docker/sidecar/docker-compose.yml F:docker/sidecar/config/sidecar.toml F:docker/sidecar/README.md

Validation and Test Coverage

Current sidecar validation stack includes:

  • config fail-closed validation assertions (port, header hard cap, auto-seal bounds),
  • content-length bounds middleware behavior,
  • correlation-id sanitization behavior,
  • canonical idempotency request-hash determinism checks.
  • generated error-registry compatibility assertions (Docs/generated/arxi/apis/sidecar.errors.json vs runtime baseline).
  • restart-boundary idempotency replay/conflict integration checks over real sidecar subprocess + SQLite persistence.
  • contract hardening checks for RFC 9457 payload shape, RFC 6750 auth challenge headers, and correlation header propagation.
  • probe hardening checks for unauthenticated probe routes and startup probe response contract.
  • readiness hardening checks for admission saturation fail-closed behavior and enterprise control-plane dependency mode transitions.
  • generated sidecar contract compatibility gates (openapi/errors/enums/examples/compat) enforced in generate/check orchestration.
  • optional release perf gate (ARXI_SIDECAR_PERF_GATE=1) for SLO promotion enforcement, including generation of target/sidecar-perf/latest.json via the sidecar perf system-test suite.
  • system-test sidecar suites for end-to-end HTTP record/seal/build/verify and restart idempotency persistence.
  • system-test Docker suite validating Dockerfile/Compose/config hardening plus Docker Compose build/up/down with containerized startup/readiness probes plus open/record/query workflow.
  • system-test sidecar perf suite for write/read latency, throughput, memory growth, and overload-429 report fields used by the perf gate.

F:crates/arxi-sidecar-config/src/config/tests.rs F:crates/arxi-sidecar/tests/unit/middleware/bounds/tests.rs F:crates/arxi-sidecar/tests/unit/idempotency/tests.rs F:crates/arxi-sidecar/tests/error_registry_contract.rs F:crates/arxi-sidecar/tests/idempotency_restart.rs F:crates/arxi-sidecar/tests/http_contract_hardening.rs F:scripts/ci/sidecar_contract_gates.py F:scripts/ci/sidecar_perf_gate.py F:scripts/ci/generate_all.sh F:system-tests/tests/suites/sidecar.rs F:system-tests/tests/suites/sidecar_docker.rs F:system-tests/tests/suites/sidecar_perf.rs

File-by-File Cross Reference

AreaFileNotes
Crate boundarycrates/arxi-sidecar/src/lib.rsPublic crate entrypoints and module composition.
Binary runtimecrates/arxi-sidecar/src/main.rsCLI, service start, and healthcheck behavior.
Config + validationcrates/arxi-sidecar-config/src/config.rs, crates/arxi-sidecar-config/src/validation.rs, crates/arxi-sidecar-config/src/schema.rs, crates/arxi-sidecar-config/src/compat.rsCanonical sidecar config model, startup fail-closed validation, and deterministic config projection artifacts.
Server bootstrapcrates/arxi-sidecar/src/server.rsStore/runtime wiring, router build, transport lifecycle.
Shared statecrates/arxi-sidecar/src/state.rsRuntime handles and concurrency gates.
Error surfacecrates/arxi-sidecar/src/error.rsStructured API error model.
JSON extractorscrates/arxi-sidecar/src/extract.rsStrict JSON boundary normalization for typed parse failures.
Runtime contract baselinecrates/arxi-sidecar/src/contract.rsExported API version and error-registry constants used by contract tests.
Idempotencycrates/arxi-sidecar/src/idempotency.rsPersistent replay/conflict semantics and canonical request hash.
Enterprise policy gatecrates/arxi-sidecar/src/control_plane_bridge.rsEnterprise boundary adapter only: control-plane ingress checks and sidecar hook emission client.
Auth middlewarecrates/arxi-sidecar/src/middleware/auth.rsBearer token checks with constant-time compare.
Enterprise middlewarecrates/arxi-sidecar/src/middleware/enterprise.rsEnterprise identity extraction and boundary enforcement/hook sequencing.
Bounds middlewarecrates/arxi-sidecar/src/middleware/bounds.rsHeader/content/queue/concurrency/timeout controls.
Docker packagingdocker/sidecar/Dockerfile, docker/sidecar/docker-compose.yml, docker/sidecar/config/sidecar.toml, docker/sidecar/README.mdFirst-party container build/runtime profile and operator usage guide.
Sidecar unit-test modulescrates/arxi-sidecar-config/src/config/tests.rs, crates/arxi-sidecar/tests/unit/idempotency/tests.rs, crates/arxi-sidecar/tests/unit/middleware/bounds/tests.rsUnit tests are externalized from production source files for cleaner file organization.
Sidecar Docker system-testssystem-tests/tests/suites/sidecar_docker.rsDockerfile/Compose hardening and containerized sidecar workflow validation.
Probe handlerscrates/arxi-sidecar/src/handlers/health.rsLiveness/startup/readiness behavior, including saturation and enterprise dependency readiness modes.
Contract conformance testscrates/arxi-sidecar/tests/error_registry_contract.rs, crates/arxi-sidecar/tests/idempotency_restart.rs, crates/arxi-sidecar/tests/http_contract_hardening.rsRegistry, restart-idempotency, and HTTP boundary hardening checks.
Contract CI gatesscripts/ci/sidecar_contract_gates.py, scripts/ci/sidecar_perf_gate.py, scripts/ci/generate_all.shAdditive-only compatibility plus optional release perf-gate orchestration.
Sidecar system testssystem-tests/tests/suites/sidecar.rs, system-tests/tests/suites/sidecar_perf.rsEnd-to-end HTTP lifecycle coverage and perf report generation for SLO gating.
Security alignmentDocs/security/threat_model.mdAuthoritative threat mapping and residual risks.