Arquitectura del Sistema de Grabación
Audience: Engineers working across Recorder crates or integrating Recorder with sistemas de plano de control como Decision Gate.
Tabla de Contenidos
- Resumen Ejecutivo
- Arquitectura en Capas
- Flujo de Datos de Extremo a Extremo
- Límites de Confianza y Postura de Seguridad
- Modelo de Determinismo
- Alcance de Implementación Actual
- Referencia Cruzada Archivo por Archivo
Resumen Ejecutivo
Recorder es un plano de datos para la grabación de evidencia determinista. Captura eventos no sellados en los límites de los adaptadores, los sella en sobres encadenados por hash, los persiste en historiales de segmentos de solo anexado y exporta paquetes portátiles para verificación offline.
El sistema está organizado como un espacio de trabajo de Rust con límites estrictos de crate: contratos centrales, modelo de sobre, orquestación de grabadores, backends de almacenamiento, una fachada ergonómica de SDK de Rust, un SDK de sidecar de Python generado por primera parte, semánticas de configuración de sidecar canónicas, operaciones de CLI y un servicio HTTP de sidecar en tiempo de ejecución. F:Cargo.toml L9-L26
Arquitectura en Capas
- Core contracts (
recorder-core)- Identity newtypes, hash/signature abstractions, filters, schema versioning, and boundary error algebras. F:crates/recorder-core/src/lib.rs L53-L109
- Evidence model (
recorder-envelope)- Envelope/segment/bundle types, canonical encoding functions, adapter and provider contracts, and verification result algebra. F:crates/recorder-envelope/src/lib.rs L53-L111
- Integration adapters (
recorder-decision-gate-adapter,recorder-otel-adapter)- First-party deterministic adapters implementing:
- Decision Gate MCP/runpack transcript mapping, redaction/bounds policy, y realizar verificaciones de integridad de paquetes.
- OpenTelemetry JSON export mapping (
spans+logs) with deterministic IDs/event types and strict-fail parse policy. F:crates/recorder-decision-gate-adapter/src/lib.rs L1-L40 F:crates/recorder-otel-adapter/src/lib.rs
- First-party deterministic adapters implementing:
- Recording runtime (
recorder)RecorderEngine,BundleBuilder,Verifier, local adapter, and provider implementation. F:crates/recorder/src/lib.rs L34-L56
- Persistence (
recorder-store)- Store traits with SQLite and in-memory implementations. F:crates/recorder-store/src/lib.rs L49-L71
- Sidecar config authority (
recorder-sidecar-config)- Canonical sidecar config model, fail-closed runtime validation, and
deterministic config projection helpers consumed by
recorder-contract. F:crates/recorder-sidecar-config/src/lib.rs
- Canonical sidecar config model, fail-closed runtime validation, and
deterministic config projection helpers consumed by
- Rust SDK facade (
recorder-sdk)- Single-crate Rust consumer surface composing recorder, local adapter, provider, verifier, and store handles via deterministic builder defaults. F:crates/recorder-sdk/src/lib.rs F:crates/recorder-sdk/src/builder.rs
- Generated sidecar SDK (
recorder-clientfor Python)- Contract-driven sync/async HTTP client generated from Docs/generated/recorder/sdk/{types,methods}.json with strict typed request/response validation, retry-safe idempotency behavior, and fail-closed error mapping. F:sdk/python/scripts/generate_contract.py F:sdk/python/src/recorder_client/client.py F:sdk/python/src/recorder_client/sync_client.py F:sdk/python/src/recorder_client/async_client.py
- Operational surface (
recorder-cli)- CLI command groups for segment lifecycle, recording, query, bundle, config validation, diagnostics, attachment-aware ingest, Decision Gate fixture ingest, and OTel fixture ingest, with handlers split into command and support helper modules. F:crates/recorder-cli/src/main.rs L137-L272 F:crates/recorder-cli/src/commands.rs L19-L29 F:crates/recorder-cli/src/support.rs L19-L23
- Superficie del servicio de red (
recorder-sidecar)
- Sidecar HTTP/JSON runtime with auth/bounds middleware and a single-writer
ingest runtime over Unix/TCP listeners, including explicit
liveness/startup/readiness probes, enterprise-aware readiness modes,
stream-scoped deterministic query semantics, and compatibility ingest route
POST /v1/ingest/otel. F:crates/recorder-sidecar/src/lib.rs L1-L41 F:crates/recorder-sidecar/src/server.rs L42-L247
- Generación de contratos de proyección (
recorder-contract)
- Deterministic artifact generation/check (Docs/generated/recorder/**) and
manifest hash authority (
index.json). F:crates/recorder-contract/src/lib.rs
Flujo de Datos de Extremo a Extremo
Adapter / CLI input
-> Sidecar mutating route validation (tenant_id + recorder_id + idempotency key)
-> IngestGateway bounded queue admission
-> IngestWriterRuntime writer-native sealing + single-tx micro-batch commit
-> Stream/global commit index assignment + idempotency + projection update
-> BundleBuilder resolves selector + attachment closure + manifest
-> Verifier runs 7 phases offline
-> EvidenceProvider serves bundles/envelope queries to control-plane bridge
Puntos de entrada clave del flujo:
- Ingest path:
IngestGateway::submit_envelope->IngestWriterRuntime::process_batchF:crates/recorder-sidecar/src/ingest.rs - Writer-native sealing/commit path (sidecar hot path no longer calls
RecorderEngine::record_envelopefor mutating HTTP ingest) F:crates/recorder-sidecar/src/ingest.rs - Bundle export path:
BundleBuilder::buildF:crates/recorder/src/bundle_builder.rs L129-L172 - Offline verification path:
Verifier::verifyF:crates/recorder/src/verifier.rs L95-L136
Límites de Confianza y Postura de Seguridad
- Adapter inputs are treated as untrusted and validated fail-closed before recorder append. F:crates/recorder/src/validation.rs L46-L83
recorder-sdkpreserves the same fail-closed adapter/provider boundaries by composing existingLocalRecorderAdapterandRecorderEvidenceProviderimplementations rather than redefining validation or policy behavior. F:crates/recorder-sdk/src/builder.rsrecorder-client(Python) treats sidecar responses as untrusted, validates them against generated contract projections, fails closed on unknown error codes or schema drift, and preserves idempotency keys across retries for mutating operations. F:sdk/python/src/recorder_client/client.py F:sdk/python/src/recorder_client/client_common.py F:sdk/python/src/recorder_client/sync_client.py F:sdk/python/src/recorder_client/async_client.py F:sdk/python/src/recorder_client/errors.py- Core identity constructors fail closed on blank/control-character strings,
and
KeyIdenforces canonical lowercase SHA-256 hex shape. String-backed identity types now also enforce explicit max-length policies and constructor-validated serde deserialization. F:crates/recorder-core/src/identity.rs L31-L55 F:crates/recorder-core/src/identity.rs L37-L47 F:crates/recorder-core/src/identity.rs L516-L647 - Envelope-layer attachment and segment constructors reject empty/blank/control
text inputs for
content_typeandrecorder_id, enforce max-length bounds, and validate serde deserialize boundaries through constructors. F:crates/recorder-envelope/src/attachment.rs L35-L44 F:crates/recorder-envelope/src/attachment.rs L79-L96 F:crates/recorder-envelope/src/segment.rs L42-L56 F:crates/recorder-envelope/src/segment.rs L91-L118 - Store append enforces chain continuity and rejects writes to sealed segments. F:crates/recorder-store/src/traits.rs L47-L66
- Verifier and provider library boundaries enforce bounded-work fail-closed policies to protect embedding wrappers from unbounded in-memory workloads. F:crates/recorder/src/verifier.rs L73-L246 F:crates/recorder/src/evidence_provider.rs L52-L300
- SQLite dispatch now uses bounded queue admission and deterministic saturation rejection rather than unbounded operation buffering. F:crates/recorder-store/src/sqlite/connection.rs L42-L136
- Sidecar HTTP boundary enforces request identity, optional bearer-token auth
with constant-time compare, protocol/header/body bounds, bounded queue and
active concurrency gates, timeout fail-closed behavior, required
tenant/recorder mutating request identity checks, and persistent stream-scoped
idempotency replay/conflict controls in the single-writer ingest path.
Probe routes (
/health,/startup,/ready) are explicitly exempt from those request gates so orchestration liveness/startup/readiness signals remain observable under stress. F:crates/recorder-sidecar/src/middleware/request_id.rs L15-L30 F:crates/recorder-sidecar/src/middleware/auth.rs L21-L84 F:crates/recorder-sidecar/src/middleware/bounds.rs L21-L172 F:crates/recorder-sidecar/src/ingest.rs - Sidecar container packaging is first-party in-repo and keeps the same fail-closed config + token-hardening model under distroless nonroot runtime with explicit UID/GID mapping plus secret-source-to-tmpfs token bootstrap before sidecar startup. F:docker/sidecar/Dockerfile F:docker/sidecar/docker-compose.yml F:docker/sidecar/config/sidecar.toml F:crates/recorder-sidecar/src/bin/docker_bootstrap.rs
- Hash equality uses constant-time comparison in core, verifier, and stores. F:crates/recorder-core/src/hash.rs L102-L174 F:crates/recorder/src/verifier.rs L29-L52 F:crates/recorder-store/src/sqlite/envelope_ops.rs L62-L79
Modelo de Determinismo
El determinismo se logra combinando:
- codificación JCS canónica para datos hashables,
- estructuras de ordenamiento deterministas (
BTreeMap, vectores ordenados), - lógica de hash estable y de selector,
- verificación reproducible sin conexión.
F:crates/recorder-envelope/src/encoding.rs L13-L31 F:crates/recorder/src/bundle_builder.rs L27-L30 F:crates/recorder/src/bundle_builder.rs L466-L477
Alcance de Implementación Actual
Implementado ahora:
- Ciclo de vida completo de grabación local y verificación de paquetes.
- Backend de almacenamiento SQLite/en memoria.
recorder-sdksingle-crate facade for Rust consumers, including deterministic valores predeterminados del constructor y accesores directos de adaptador/proveedor/almacen.recorder-clientPython SDK with generated contract constants, typed sync/async clientes sidecar y pruebas de unidad/sistema.- Operaciones impulsadas por CLI y salida localizada.
- CLI attachment-aware recording (
record-with-attachments) with bounded file Grabación consciente de archivos adjuntos en CLI (record-with-attachments) con entradas de archivo limitado y adjuntos en línea. - CLI Decision Gate fixture ingest command path wired to production adapter Ruta del comando de ingestión del accesorio de la puerta de decisión de CLI conectada a la implementación del adaptador de producción.
- CLI OTel fixture ingest command path wired to production adapter implementación.
- System-test harness with registry-driven coverage, including OpenClaw gateway/CLI mock-flow and Decision Gate runpack-flow adapter-ingest pruebas de integración.
- CLI expansion system-tests for recorder-id shape validation parity,
attachment-recording fail-closed boundaries, auto-seal duration/combined
lifecycle behavior, query JSON pagination/limit guardrails, and Decision Gate
Rutas de éxito/fallo estricto de
ingest-fixtureen la CLI. - Production Decision Gate adapter crate
(
crates/recorder-decision-gate-adapter) with deterministic MCP/runpack mapping, runpack-integrity strict/anomaly policy handling, and transcript controles de límites de redacción. - Production OTel adapter crate (
crates/recorder-otel-adapter) with deterministic span/log mapping, strict identifier/time parsing, and bounded/redacted controles de carga útil. - OpenClaw integration harness mapping policy with deterministic sensitive-field Política de mapeo del arnés de integración de OpenClaw con redacción determinista de campos sensibles y manejo de carga útil acotada para pruebas de acoplamiento impulsadas por fixture.
- Decision Gate integration system-tests wired to the production adapter crate, covering signed/unsigned lanes, root-hash and manifest-integrity mismatch Sistema de pruebas de integración de Decision Gate conectado a la caja del adaptador de producción, cubriendo comportamientos de fallo cerrado por desajuste de carriles firmados/no firmados, raíz-hash y manifiesto-integridad.
- Sidecar Docker operator profile (
docker/sidecar) and system-test coverage for Dockerfile/Compose hardening plus containerized probe/record/query flujo. - Sidecar readiness semantics now support fail-closed dependency modes: La semántica de disponibilidad del sidecar ahora admite modos de dependencia que fallan en cerrado: disponibilidad solo de almacenamiento y disponibilidad de almacenamiento + control empresarial.
Postura actual de rendimiento/estrés:
- P2 stress/performance system-test expansion is now implemented with
deterministic artifact-first lanes:
performance::stress_concurrent_append_query_sqlite,performance::performance_bundle_materialization_smoke, andsidecar_perf::sidecar_perf_smoke_generates_gate_report(regression lane), plussidecar_capacity::sidecar_capacity_sweep_generates_report(carril de capacidad). - CI now runs a profile matrix (
testinformational +releasegated) on the pinned perf runner viascripts/ci/perf_matrix.pyand enforces release thresholds/regression policy viascripts/ci/perf_gate.pyandsystem-tests/perf_baselines/linux_x86_64_ci.json. - Sidecar ingest
10ktarget is defined only on the capacity lane metricmax_sustainable_rpsand is enforced only for pinned runner policy class (linux_x86_64_pinned); the regression lane remains a deterministic fixed señal anti-regresión de carga de trabajo. - Sidecar perf throughput/latency lanes now run against persistent keep-alive HTTP client pools to avoid connection-churn distortion in write-path mediciones.
- Bundle-path metric
bundle_build_envelopes_per_secremains a separate core métrica de materialización de paquete y no es un proxy de capacidad de ingestión.
Elementos de endurecimiento a nivel de sistema recientemente cerrados:
- Provider listing/fetch now uses stable bundle IDs and fetch-by-ID segment selectors. F:crates/recorder/src/evidence_provider.rs L125-L229
- Recorder verifier/provider now enforce explicit bounded-work policy and reject oversized workloads fail-closed at library boundaries. F:crates/recorder/src/verifier.rs L73-L246 F:crates/recorder/src/evidence_provider.rs L52-L300
- SQLite operation dispatch now applies bounded queue backpressure with deterministic saturation failure behavior. F:crates/recorder-store/src/sqlite/connection.rs L42-L203
- Contract generation now overlays inferred schemas with authoritative runtime constraints and fail-closes on missing constraint pointer targets. F:crates/recorder-contract/src/lib.rs L565-L742
- Sidecar contract-generation/projection integration is now implemented with
generated API artifacts (
openapi/errors/enums/examples/compat) and fail-closed contract gates in CI orchestration under current semántica de la política de pre-lanzamiento para/v1. - Sidecar compatibility surface now includes OTel ingest route
(
POST /v1/ingest/otel) with generated contract baseline and strict media-type gate alignment from shared policy source. F:crates/recorder-contract/src/lib.rs F:scripts/ci/sidecar_contract_gates.py F:scripts/ci/generate_all.sh - Verifier phase 6 now executes trust-root signature checks with policy evaluation (skip warning only when no trust root is provided). F:crates/recorder/src/verifier.rs L509-L651
- Startup read-back verification now executes from
RecorderEngine::newwhenstartup_verification_depth > 0. F:crates/recorder/src/engine.rs L154-L197 F:crates/recorder/src/engine.rs L380-L543
Referencia Cruzada Archivo por Archivo
| Área | Archivo | Notas |
|---|---|---|
| Límites del espacio de trabajo | Cargo.toml | Membresía del crate y postura de lint. |
| Contratos centrales | crates/recorder-core/src/lib.rs | Primitivas y rasgos de dominio compartidos. |
| Modelo de sobre | crates/recorder-envelope/src/lib.rs | Modelo de datos de evidencia y contratos. |
| Tiempo de ejecución del grabador | crates/recorder/src/engine.rs | Ingesta, progresión de la cadena, ciclo de vida del segmento. |
| Fachada del SDK de Rust | crates/recorder-sdk/src/lib.rs, crates/recorder-sdk/src/builder.rs | Constructor ergonómico y fachada unificada de grabador/proveedor para consumidores de Rust. |
| Fachada del SDK de Python | sdk/python/src/recorder_client/client.py, sdk/python/src/recorder_client/client_common.py, sdk/python/src/recorder_client/sync_client.py, sdk/python/src/recorder_client/async_client.py, sdk/python/src/recorder_client/models.py, sdk/python/src/recorder_client/errors.py | Superficie de cliente sidecar síncrono/asíncrono impulsada por contratos generados con validación estricta de cierre en fallo y mapeo de errores tipados. |
| Paquete + verificador | crates/recorder/src/bundle_builder.rs | Resolución de selectores y materialización de paquetes. |
| Verificación de paquetes | crates/recorder/src/verifier.rs | Verificación fuera de línea en 7 fases. |
| Rasgos de almacenamiento | crates/recorder-store/src/traits.rs | Invariantes del contrato de persistencia. |
| Límite de cola de SQLite | crates/recorder-store/src/sqlite/connection.rs | Comportamiento de despacho asíncrono a síncrono limitado y cierre en fallo de saturación. |
| Superficie de CLI | crates/recorder-cli/src/main.rs, crates/recorder-cli/src/commands.rs, crates/recorder-cli/src/support.rs | Arquitectura de comando operativo dividida en despacho, controladores de comandos y ayudantes de tiempo de ejecución/utilidad. |
| Servicio sidecar | crates/recorder-sidecar/src/lib.rs, crates/recorder-sidecar/src/server.rs | Tiempo de ejecución HTTP, pila de middleware y ciclo de vida del transporte. |
| Configuración/seguridad del sidecar | crates/recorder-sidecar-config/src/config.rs, crates/recorder-sidecar-config/src/validation.rs, crates/recorder-sidecar/src/middleware/auth.rs, crates/recorder-sidecar/src/middleware/bounds.rs, crates/recorder-sidecar/src/ingest.rs | Validación de configuración de cierre en fallo y autoridad de proyección más controles de autenticación/límites del sidecar y comportamiento de idempotencia de ingesta de un solo escritor. |
| Empaquetado de contenedor sidecar | docker/sidecar/Dockerfile, docker/sidecar/docker-compose.yml, docker/sidecar/config/sidecar.toml, crates/recorder-sidecar/src/bin/docker_bootstrap.rs | Perfil de construcción/tiempo de ejecución de contenedor de primera parte alineado con las invariantes de validación/seguridad del sidecar, incluyendo el arranque de token de fuente secreta en la ruta de tiempo de ejecución tmpfs endurecida. |
| Generador de contratos | crates/recorder-contract/src/lib.rs, crates/recorder-contract/src/sidecar_api.rs, crates/recorder-contract/src/sidecar_api/openapi.rs, crates/recorder-contract/src/sidecar_api/artifacts.rs, crates/recorder-contract/src/sidecar_api/specs/mod.rs | Autoridad de artefacto generado determinista + comprobaciones de deriva. |
| Pruebas del sistema | system-tests/README.md | Contrato de validación de extremo a extremo. |
| Adaptador de Decision Gate | crates/recorder-decision-gate-adapter/src/adapter.rs | Implementación de mapeo/redacción/política de integridad de producción para la ingesta de Decision Gate. |
| Adaptador OTel | crates/recorder-otel-adapter/src/adapter.rs | Implementación de mapeo/redacción/política de límites de producción para la ingesta de JSON de OTel. |
| Pruebas de integración de OpenClaw | system-tests/tests/suites/integration_openclaw.rs | Verificación de ingesta de adaptador impulsada por fixture para acoplamiento de flujo de trabajo externo. |
| Arquitectura de integración de OpenClaw | Docs/architecture/recorder_openclaw_integration_architecture.md | Contrato de mapeo y redacción/política de carga útil limitada versionada. |
| Pruebas de integración de Decision Gate | system-tests/tests/suites/integration_decision_gate.rs | Verificación de ingesta de flujo de MCP impulsada por fixture para acoplamiento de plano de control. |
| Arquitectura de integración de Decision Gate | Docs/architecture/recorder_decision_gate_integration_architecture.md | Mapeo de flujo de MCP versionado, política de integridad de runpack y contrato de redacción/límites de transcripción. |
| Arquitectura de integración de OTel | Docs/architecture/recorder_otel_integration_architecture.md | Contrato de mapeo de JSON de OTel versionado, IDs/eventos deterministas y cableado de ingesta de sidecar/CLI. |