Recorder Sidecar Architecture

Audience: Engineers implementing or operating recorder-sidecar as the network-accessible evidence recording service boundary.

Executive Overview
Runtime Composition
HTTP Surface and Routing
Security and Resource Controls
Idempotency and Retry Safety
Transport and Lifecycle
Container Packaging and Docker Operations
Validation and Test Coverage
File-by-File Cross Reference

Executive Overview

recorder-sidecar is the Recorder HTTP/JSON service runtime that exposes recorder, query, and bundle workflows over Unix socket and/or TCP listeners.

The sidecar composes RecorderEngine, BundleBuilder, Verifier, and a single SqliteStore instance under explicit middleware gates for correlation identity, authentication, enterprise identity/policy, bounds enforcement, and timeout control. Mutating envelope ingest now flows through a bounded queue and single authoritative writer runtime.

F:crates/recorder-sidecar/src/server.rs L42-L133 F:crates/recorder-sidecar/src/state.rs L29-L56 F:crates/recorder-sidecar-config/src/config.rs

Runtime Composition

Startup flow:

Load and validate TOML config (config_version, api.major_version, recorder, transport, security, signer, and limits).
Initialize tracing and open SqliteStore.
Build recorder runtime components.
Initialize bounded ingest gateway + dedicated writer runtime task.
Build axum router with middleware stack.
Bind configured transport listeners and serve until cancellation.

F:crates/recorder-sidecar-config/src/config.rs F:crates/recorder-sidecar-config/src/validation.rs F:crates/recorder-sidecar/src/server.rs L43-L247 F:crates/recorder-sidecar/src/ingest.rs

HTTP Surface and Routing

The router currently exposes:

Probes: GET /health, GET /startup, GET /ready
Segment observability: GET /v1/segments, GET /v1/segments/active
Envelope ingest: POST /v1/envelopes, POST /v1/envelopes/with-attachments
Query: POST /v1/query/envelopes
Bundle workflows: POST /v1/bundles/build, POST /v1/bundles/verify, POST /v1/bundles/inspect
Compatibility ingest: POST /v1/ingest/otel (OTel JSON export v1)
Runtime inspection: GET /v1/config

F:crates/recorder-sidecar/src/server.rs L135-L160 F:crates/recorder-sidecar/src/handlers/otel.rs

Segment observability contract notes:

GET /v1/segments/active now returns the same paginated list shape as GET /v1/segments (not a singleton resource), filtered to status=open.
Segment response objects now include stream identity (stream_key.tenant_id, stream_key.recorder_id) to make per-stream roll state explicit at the API boundary.

F:crates/recorder-sidecar/src/handlers/segment.rs

Mutating request contract highlights:

Idempotency-Key is required for mutating endpoints.
Envelope routes require tenant_id and recorder_id in request bodies.
recorder_id must match the configured sidecar recorder identity.
In enterprise-enabled modes, mutating envelope routes require x-tenant-id header/body binding.

Health behavior:

GET /health is a lightweight liveness probe returning process metadata.
GET /startup is a startup-completion probe returning startup metadata, including configured startup_verification_depth.
GET /ready returns readiness state with dependency-aware fail-closed semantics:
- 503 with reason=storage_unavailable when store access fails.
- Optional 503 with reason=request_capacity_exhausted when probes.ready_fail_on_admission_saturation=true and admission/execution semaphores are exhausted.
- Optional 503 with reason=enterprise_control_unavailable when probes.readiness_mode=storage_and_enterprise, enterprise mode is enabled, and control-plane readiness checks fail.
- 200 with ready=true in steady state.

F:crates/recorder-sidecar/src/handlers/health.rs L42-L94

Security and Resource Controls

Config Gate

Config validation is fail-closed and enforces:

strict schema and API versioning,
recorder ID and auto-seal validity,
transport host/port constraints,
non-loopback transport requires security.mode=token and keeps require_token_for_non_loopback=true (fail-closed),
token file hardening (security.token_file cannot be a symlink, must be a regular file, and on Unix must not grant group/other permissions),
Secret token paths are opened with descriptor-bound hardening (O_NOFOLLOW on Unix, FILE_FLAG_OPEN_REPARSE_POINT on Windows) and validated on opened handles to close check/use races,
bounded hostile file reads (sidecar config <= 1 MiB, token/control-plane token files <= 4096 bytes, signer key file <= 65536 bytes),
signer key parseability,
request/header/idempotency numeric limits and hard cap checks,
probe config fail-closed semantics: probes.enterprise_health_path must be absolute/safe and probes.readiness_mode=storage_and_enterprise requires enterprise mode not disabled.

F:crates/recorder-sidecar-config/src/config.rs F:crates/recorder-sidecar-config/src/validation.rs

Middleware Stack

Middleware order and controls:

Request body limit layer (max_request_body_bytes).
Tracing layer.
Correlation middleware (server correlation IDs, sanitized client x-correlation-id, legacy x-request-id compatibility).
Auth middleware (Bearer token mode with constant-time compare and RFC 6750 challenge metadata on 401 responses).
Enterprise identity middleware (x-tenant-id, x-principal-id, x-authn-method, optional x-namespace-id, optional writer lease assertion) for enterprise-enabled modes.
Enterprise policy middleware (control-plane ingress checks for protected routes plus sidecar boundary hook emission at admission and result phases).
Bounds middleware (header bytes, content negotiation, content-length checks, bounded admission queue, bounded active concurrency, request timeout).

Probe-priority behavior:

/health, /startup, and /ready are exempted from auth, enterprise policy, and bounds gating so liveness/startup/readiness state remains observable under overload or policy failures.

F:crates/recorder-sidecar/src/server.rs L155-L160 F:crates/recorder-sidecar/src/middleware/request_id.rs L15-L30 F:crates/recorder-sidecar/src/middleware/auth.rs L21-L84 F:crates/recorder-sidecar/src/control_plane_bridge.rs F:crates/recorder-sidecar/src/middleware/enterprise.rs F:crates/recorder-sidecar/src/middleware/bounds.rs L21-L172 F:crates/recorder-sidecar-config/src/media_types.rs

Strict media-type policy source:

request-body routes use a centralized allowlist from recorder-sidecar-config::sidecar_request_media_type_policies,
Content-Type parsing is strict fail-closed (application/json with optional charset=utf-8 only),
Accept parsing is strict fail-closed (application/json or */* only),
generated sidecar OpenAPI/compat artifacts and CI gates consume the same media-type policy source to prevent runtime/contract drift.

Enterprise Boundary Controls

When enterprise.mode is enabled, the sidecar adds explicit control-plane boundary enforcement while preserving recorder/evidence semantics:

crates/recorder-sidecar/src/control_plane_bridge.rs is a boundary adapter only. It translates sidecar ingress identities/operations into control-plane admission and hook calls; it does not implement or alter Recorder evidence math.
Enterprise config is fail-closed: enabled mode requires token security mode, control-plane URL/token configuration, and hardened token file checks.
Enterprise policy gate calls POST /v1/enterprise/ingress/check before protected sidecar operations.
Sidecar ingress operation vocabulary exported to enterprise control-plane is restricted to active v1 routes: recorder.record_envelope, recorder.record_envelope_with_attachments, recorder.ingest_otel, recorder.query_envelopes, recorder.build_bundle, recorder.verify_bundle, and recorder.inspect_bundle. Removed mutating segment routes are not included in this vocabulary.
Managed-cloud mode enforces writer-lease assertion for mutating operations when require_writer_lease_header=true.
Sidecar emits deterministic enterprise boundary hooks to POST /v1/enterprise/sidecar/hooks at:
- admission (allow/deny/error before handler execution),
- result (allow/deny/error after handler response status).
Hook emission is fail-closed for successful protected operations: if enterprise hook export fails, the sidecar returns an error instead of silently accepting an unaudited success path.
Enterprise readiness dependency checks use probes.enterprise_health_path (default /health) against the configured enterprise control-plane base URL when probes.readiness_mode=storage_and_enterprise.

F:crates/recorder-sidecar-config/src/validation.rs F:crates/recorder-sidecar/src/control_plane_bridge.rs F:crates/recorder-sidecar/src/middleware/enterprise.rs

Error Contract

Errors are returned as RFC 9457 Problem Details (application/problem+json) with stable error codes, retryability metadata, and server/client correlation fields.

Security posture details:

malformed JSON rejections are normalized to typed bad_request problem details instead of leaking framework/plaintext extractor failures,
server-side (5xx) responses suppress internal detail fields to avoid leaking storage/runtime internals while preserving diagnostics in service logs,
401 unauthorized responses include RFC 6750 WWW-Authenticate challenges.

F:crates/recorder-sidecar/src/error.rs L13-L229 F:crates/recorder-sidecar/src/extract.rs

Idempotency and Retry Safety

POST /v1/envelopes and POST /v1/envelopes/with-attachments now submit normalized commands into a bounded in-memory queue (IngestGateway) and are processed by a dedicated single-writer runtime (IngestWriterRuntime).

Writer-owned idempotency is persisted in idempotency_entries, scoped by: (stream_tenant_id, stream_recorder_id, endpoint, idempotency_key) plus canonical request hash.

Hot-path writer behavior:

mutating handlers enqueue normalized commands only,
writer-native sealing computes deterministic attachment refs, content hash, chain hash, stream sequence, and receipts,
one SQLite transaction per micro-batch persists envelopes, projection rows, idempotency entries, stream state, and segment transitions atomically.
stream state load/create is performed once per stream per transaction so segment roll paths cannot open orphan zero-envelope segments before stream pointer persistence; this preserves the single-open-segment restart invariant.

Behavior:

same key + same hash: replay prior durable receipt (idempotency_outcome=replayed),
same key + different hash: 409 idempotency_conflict,
queue saturation: fail-fast 429 too_many_requests,
storage failure: fail-closed idempotency_unavailable / storage_unavailable.

Retention is bounded by ingest.idempotency_ttl_seconds and ingest.idempotency_max_entries_per_stream. Writer retention cleanup is applied in two phases inside the same transaction:

expiry cleanup by expires_at_unix,
overflow cleanup by ordered offset delete over created_at_unix DESC. This path relies on SQLite stream-scope ordering index idx_idempotency_stream_created(stream_tenant_id, stream_recorder_id, created_at_unix DESC) to avoid history-depth-induced trim latency spikes.

F:crates/recorder-sidecar/src/ingest.rs F:crates/recorder-sidecar/src/handlers/envelope.rs

OTel ingest compatibility route:

POST /v1/ingest/otel ingests OTel JSON export payloads through recorder-otel-adapter with deterministic mapping and strict-fail record validation.
Route behavior is additive under /v1, requires Idempotency-Key, and reuses sidecar auth/enterprise/bounds/timeout/correlation controls.
Runtime response includes mapping version and deterministic ingest counts.

F:crates/recorder-otel-adapter/src/adapter.rs

Transport and Lifecycle

Transport modes:

Unix domain socket,
TCP listener,
both simultaneously.

Runtime lifecycle:

stale Unix socket cleanup before bind (socket-type-validated fail-closed),
signal-driven cancellation token,
graceful shutdown per listener with enforced shutdown_drain_seconds timeout,
Unix socket removal on shutdown path.

F:crates/recorder-sidecar/src/server.rs L163-L262 F:crates/recorder-sidecar/src/shutdown.rs L8-L34

Operational command surface in binary:

normal service startup with --config / RECORDER_SIDECAR_CONFIG,
healthcheck command supporting TCP URL and Unix-socket probe modes.

F:crates/recorder-sidecar/src/main.rs L18-L175

Container Packaging and Docker Operations

The repository now ships a first-party sidecar container profile under docker/sidecar/:

Dockerfile: multi-stage Rust build to distroless nonroot runtime image.
docker-compose.yml: local operator profile with explicit UID/GID mapping, bind-mounted config/data/run paths, tmpfs-backed token target path, Docker secret source path, healthcheck, and bounded port mapping.
config/sidecar.toml: container-oriented baseline config with token-mode auth required for non-loopback transport.
.env.example: canonical environment variable contract for image/tag/paths and UID/GID + token-secret source mapping.

Operator contract:

supply token material through a Docker secret source file (RECORDER_SIDECAR_TOKEN_SECRET_FILE),
bootstrap writes token material to /work/secrets/token in tmpfs with owner-only permissions (0600 on Unix) before sidecar startup,
run non-root with host UID/GID mapping so strict token checks remain readable while avoiding root execution,
treat docker/sidecar/secrets/token as secret material and keep it out of VCS.

F:docker/sidecar/Dockerfile F:docker/sidecar/docker-compose.yml F:docker/sidecar/config/sidecar.toml F:docker/sidecar/README.md

Validation and Test Coverage

Current sidecar validation stack includes:

config fail-closed validation assertions (port, header hard cap, auto-seal bounds),
content-length bounds middleware behavior,
correlation-id sanitization behavior,
canonical idempotency request-hash determinism checks.
generated error-registry compatibility assertions (Docs/generated/recorder/apis/sidecar.errors.json vs runtime baseline).
restart-boundary idempotency replay/conflict integration checks over real sidecar subprocess + SQLite persistence.
parallel rolled-load restart correctness test coverage asserting exactly one persisted open segment and successful post-load restart.
contract hardening checks for RFC 9457 payload shape, RFC 6750 auth challenge headers, and correlation header propagation.
strict media-type matrix checks for content-type/accept fail-closed behavior with stable unsupported_media_type and not_acceptable error codes.
probe hardening checks for unauthenticated probe routes and startup probe response contract.
readiness hardening checks for admission saturation fail-closed behavior and enterprise control-plane dependency mode transitions.
enterprise outage mutation checks proving write paths fail closed with service-unavailable when control-plane ingress dependency is down.
generated sidecar contract compatibility gates (openapi/errors/enums/examples/compat) enforced in generate/check orchestration.
profile-matrix perf orchestration (RECORDER_PERF_MATRIX=1) that executes debug/test and release regression lanes, emits target/perf-matrix/latest.json, and applies release-gated threshold + regression policy using committed baseline contracts (system-tests/perf_baselines/*.json); capacity lane execution is optional (--include-capacity) and intended for pinned nightly/pre-release jobs.
sidecar perf architecture split:
- regression lane artifact: target/sidecar-perf/latest.json (lane=regression, fixed workload anti-regression signal),
- capacity lane artifact: target/sidecar-capacity/latest.json (lane=capacity, saturation sweep with max_sustainable_rps and knee metrics).
system-test sidecar suites for end-to-end HTTP record/query/build/verify and restart idempotency persistence.
system-test Docker suite validating Dockerfile/Compose/config hardening plus Docker Compose build/up/down with containerized startup/readiness probes plus record/query workflow.
system-test sidecar perf suite for write/read latency, throughput, memory growth, and overload-429 report fields used by the perf gate; throughput and latency lanes now use a persistent keep-alive HTTP client pool for production-representative write-path measurements. As of 2026-02-16, memory reporting distinguishes hot-path/high-water RSS growth from quiesced retained RSS growth and emits reclaim ratio context. As of the same date, durability characterization records live wal_size_bytes, post-stop wal_size_post_stop_bytes, and coupled wal_checkpoint_noop state per case, and emits an explicit comparability signal (durability_wal_ratio_vs_1000_comparable) before interpreting WAL ratios.
ignored informational retained-memory attribution lane (sidecar_perf_rss_retained_memory_attribution_characterization) emitting target/sidecar-perf/rss_attribution_latest.json with fixed-volume sequential/parallel quiesce-window checkpoints and smaps_rollup class deltas (Rss, Anonymous, Private_Dirty, Shared_Clean, Shared_Dirty).
ignored informational ingest-tuning characterization lane (sidecar_perf_rss_ingest_tuning_characterization) emitting target/sidecar-perf/rss_ingest_tuning_latest.json with ingest queue/batch matrix sweeps (queue_capacity, batch_max_commands, batch_max_bytes) across sequential and parallel workers, including data-derived decision fields for mechanism classification and mitigation recommendation.
ingest batching byte-estimation path now avoids per-command Vec<u8> allocation churn in writer_command_estimated_bytes by counting serialized JSON bytes via serde_json::to_writer against a counting sink.
system-test sidecar capacity suite for release-only concurrency sweep points (32..512), deterministic correctness sampling, and pinned-runner max_sustainable_rps/knee metric emission.

F:crates/recorder-sidecar-config/src/config/tests.rs F:crates/recorder-sidecar/tests/unit/middleware/bounds/tests.rs F:crates/recorder-sidecar/src/idempotency.rs F:crates/recorder-sidecar/tests/error_registry_contract.rs F:crates/recorder-sidecar/tests/idempotency_restart.rs F:crates/recorder-sidecar/tests/http_contract_hardening.rs F:crates/recorder-sidecar/tests/enterprise_control_plane_e2e.rs F:scripts/ci/sidecar_contract_gates.py F:scripts/ci/perf_matrix.py F:scripts/ci/perf_gate.py F:scripts/ci/perf_threshold_policy.py F:scripts/ci/generate_all.sh F:system-tests/tests/suites/sidecar.rs F:system-tests/tests/suites/sidecar_enterprise.rs F:system-tests/tests/suites/sidecar_docker.rs F:system-tests/tests/suites/sidecar_perf.rs F:system-tests/tests/suites/sidecar_capacity.rs F:system-tests/tests/helpers/perf_sidecar.rs

File-by-File Cross Reference

Area	File	Notes
Crate boundary	`crates/recorder-sidecar/src/lib.rs`	Public crate entrypoints and module composition.
Binary runtime	`crates/recorder-sidecar/src/main.rs`	CLI, service start, and healthcheck behavior.
Container bootstrap binary	`crates/recorder-sidecar/src/bin/docker_bootstrap.rs`	Fail-closed token-source ingestion and hardened in-container token-file materialization before sidecar startup.
Config + validation	`crates/recorder-sidecar-config/src/config.rs`, `crates/recorder-sidecar-config/src/validation.rs`, `crates/recorder-sidecar-config/src/schema.rs`, `crates/recorder-sidecar-config/src/compat.rs`	Canonical sidecar config model, startup fail-closed validation, and deterministic config projection artifacts.
Media-type policy	`crates/recorder-sidecar-config/src/media_types.rs`	Shared strict media-type policy for runtime parser behavior and generated sidecar contract artifacts.
Server bootstrap	`crates/recorder-sidecar/src/server.rs`	Store/runtime wiring, router build, transport lifecycle.
Shared state	`crates/recorder-sidecar/src/state.rs`	Runtime handles and concurrency gates.
Error surface	`crates/recorder-sidecar/src/error.rs`	Structured API error model.
JSON extractors	`crates/recorder-sidecar/src/extract.rs`	Strict JSON boundary normalization for typed parse failures.
Runtime contract baseline	`crates/recorder-sidecar/src/contract.rs`	Exported API version and error-registry constants used by contract tests.
Single-writer ingest	`crates/recorder-sidecar/src/ingest.rs`	Bounded queue admission, durable receipt generation, stream-scoped sequencing, and persistent idempotency state.
Enterprise policy gate	`crates/recorder-sidecar/src/control_plane_bridge.rs`	Enterprise boundary adapter only: control-plane ingress checks and sidecar hook emission client.
Auth middleware	`crates/recorder-sidecar/src/middleware/auth.rs`	Bearer token checks with constant-time compare.
Enterprise middleware	`crates/recorder-sidecar/src/middleware/enterprise.rs`	Enterprise identity extraction and boundary enforcement/hook sequencing.
Bounds middleware	`crates/recorder-sidecar/src/middleware/bounds.rs`	Header/content/queue/concurrency/timeout controls.
OTel ingest handler	`crates/recorder-sidecar/src/handlers/otel.rs`	OTel JSON ingest route, idempotency coupling, and adapter invocation.
OTel adapter	`crates/recorder-otel-adapter/src/adapter.rs`	Deterministic OTel mapping/redaction/bounds implementation used by sidecar and CLI ingestion workflows.
Docker packaging	`docker/sidecar/Dockerfile`, `docker/sidecar/docker-compose.yml`, `docker/sidecar/config/sidecar.toml`, `docker/sidecar/.env.example`, `docker/sidecar/README.md`	First-party container build/runtime profile and operator usage guide, including secret-source + tmpfs token bootstrap flow.
Sidecar unit-test modules	`crates/recorder-sidecar-config/src/config/tests.rs`, `crates/recorder-sidecar/src/idempotency.rs`, `crates/recorder-sidecar/tests/unit/middleware/bounds/tests.rs`	Unit tests are externalized from production source files for cleaner file organization.
Sidecar Docker system-tests	`system-tests/tests/suites/sidecar_docker.rs`	Dockerfile/Compose hardening and containerized sidecar workflow validation.
Probe handlers	`crates/recorder-sidecar/src/handlers/health.rs`	Liveness/startup/readiness behavior, including saturation and enterprise dependency readiness modes.
Contract conformance tests	`crates/recorder-sidecar/tests/error_registry_contract.rs`, `crates/recorder-sidecar/tests/idempotency_restart.rs`, `crates/recorder-sidecar/tests/http_contract_hardening.rs`	Registry, restart-idempotency, and HTTP boundary hardening checks.
Enterprise integration tests	`crates/recorder-sidecar/tests/enterprise_control_plane_e2e.rs`	Enterprise admission/result boundary E2E checks including control-plane outage fail-closed mutation behavior.
Contract CI gates	`scripts/ci/sidecar_contract_gates.py`, `scripts/ci/perf_matrix.py`, `scripts/ci/perf_gate.py`, `scripts/ci/perf_threshold_policy.py`, `scripts/ci/generate_all.sh`	Pre-release-break v1 contract gates plus profile-matrix perf orchestration and anti-sandbagging threshold governance.
Sidecar system tests	`system-tests/tests/suites/sidecar.rs`, `system-tests/tests/suites/sidecar_enterprise.rs`, `system-tests/tests/suites/sidecar_perf.rs`, `system-tests/tests/suites/sidecar_capacity.rs`	End-to-end HTTP lifecycle coverage, enterprise fail-closed control-plane dependency lanes, and split regression/capacity perf report generation for SLO and saturation gating.
Security alignment	Docs/security/threat_model.md	Authoritative threat mapping and residual risks.