DOMAIN:EVIDENCE_AUTOMATION¶

OWNER: julian
UPDATED: 2026-03-24
SCOPE: automated compliance evidence generation, collection, and review
SERVES: julian (compliance officer), amber (auditor)
ALSO_USED_BY: boris (DB evidence), marta (change evidence), ron (monitoring evidence), rutger (infra evidence)

OVERVIEW¶

PRINCIPLE: compliance evidence should be a byproduct of normal operations, not a separate activity.
GE_ADVANTAGE: multi-agent architecture naturally produces structured evidence — every action is logged, attributed, and timestamped.
GOAL: 80%+ of SOC 2 and ISO 27001 evidence collected automatically, reducing audit preparation to days, not weeks.

EVIDENCE_SSOT: PostgreSQL (structured, queryable)
EVIDENCE_HUMAN_READABLE: wiki brain (narrative context)
EVIDENCE_IMMUTABLE: git history (code changes, config changes)

EVIDENCE_PRODUCER_MAP¶

ACCESS_CONTROLS (CC6, A.8.2, A.8.3, A.8.5)¶

Evidence	Producer Agent	Source System	Format	Frequency
RBAC role bindings	rutger	k8s API	YAML manifests	On change + quarterly snapshot
WebAuthn enrollment records	julian	admin-ui DB	JSON records	On enrollment change
Vault access policies	rutger	Vault API	HCL/JSON	On change + quarterly snapshot
Database permission matrix	boris	PostgreSQL	SQL grants dump	Monthly
Network policy manifests	rutger	k8s API	YAML manifests	On change + quarterly snapshot
Access review records	julian	admin-ui DB	Structured records	Quarterly
Agent registry (role definitions)	jaap	AGENT-REGISTRY.json	JSON	On change
Redis ACL configuration	rutger	Redis	ACL dump	Monthly
Agent commissioning records	julian	admin-ui DB	Structured records	On onboarding
Agent decommission records	julian	admin-ui DB	Structured records	On decommission

AUTOMATION_LEVEL: HIGH
- k8s manifests extracted automatically via kubectl
- Vault policies queryable via API
- PostgreSQL grants queryable via information_schema
- Agent registry is version-controlled JSON
MANUAL_COMPONENTS: access review sign-off, commissioning approval

CHANGE_MANAGEMENT (CC8, A.8.32)¶

Evidence	Producer Agent	Source System	Format	Frequency
Pull request records	marta	GitHub API	JSON	Continuous
Code review records	koen	GitHub API	JSON (review comments)	Per PR
Test results per PR	marije	CI/CD pipeline	JSON/JUnit XML	Per PR
Adversarial test reports	ashley	executor output	Structured report	Per deployment
Merge approvals	marta	GitHub API	JSON (approval records)	Per merge
Deployment logs	rutger	k8s API	Container logs	Per deployment
Emergency change records	mira	incident DB	Structured records	Per emergency
DAG execution records	faye/sytske	orchestrator DB	JSON	Per work package
Blast radius assessments	all devs	PR descriptions	Markdown	Per shared interface change
Specification documents	anna	wiki/DB	Structured spec	Per feature

AUTOMATION_LEVEL: HIGH
- GitHub API provides PR, review, and merge data automatically
- CI/CD produces test results as pipeline artifacts
- DAG tracked in PostgreSQL automatically
- PTY capture records all agent activity
MANUAL_COMPONENTS: specification review sign-off, emergency change post-facto review

MARTA_SPECIFIC_EVIDENCE:
- Total PRs merged per period
- PRs merged with/without required approvals
- Average time from PR creation to merge
- PRs with failing tests that were merged (should be zero)
- Emergency merges (count, justification documented)
- Branch protection rule configuration
FORMAT: monthly summary report generated from GitHub API data

VULNERABILITY_MANAGEMENT (A.8.7, A.8.8)¶

Evidence	Producer Agent	Source System	Format	Frequency
Container image scan results	rutger	Trivy	JSON/SARIF	Per image build
Code security scan results	koen	Semgrep	JSON/SARIF	Per PR
Dependency vulnerability reports	rutger	Trivy/SBOM tools	JSON	Weekly
Patch tracking records	rutger	admin-ui DB	Structured records	Continuous
CIS benchmark results	rutger	kube-bench	JSON	Monthly
SBOM manifests	rutger	syft/cyclonedx	CycloneDX JSON	Per release
Vulnerability remediation records	koen/rutger	admin-ui DB	Structured records	Per finding

AUTOMATION_LEVEL: VERY_HIGH
- Trivy runs automatically in CI/CD
- Semgrep runs automatically on every PR
- kube-bench scheduled as k8s CronJob
- SBOM generated during build process
MANUAL_COMPONENTS: vulnerability triage decisions, risk acceptance documentation

INCIDENT_MANAGEMENT (CC7, A.5.24-A.5.27)¶

Evidence	Producer Agent	Source System	Format	Frequency
Incident records	mira	PostgreSQL	Structured records	Per incident
Incident timelines	mira	PostgreSQL	Timestamped events	Per incident
SLA compliance data	mira	PostgreSQL (calculated)	Metrics	Monthly summary
Post-mortem documents	mira	wiki	Markdown	Per SEV1/SEV2
Action item tracking	mira	PostgreSQL	Structured records	Per finding
Alert configuration	ron	Grafana/Loki	JSON	On change
Monitoring dashboards	ron	Grafana	Dashboard JSON	On change
Escalation records	ron	Redis/PostgreSQL	Event logs	Per escalation
Health check results	ron	k8s-health-dump.sh	JSON	Every 1 minute

AUTOMATION_LEVEL: HIGH
- Incident records created automatically when incident acknowledged
- Timelines populated from timestamped actions during response
- SLA metrics calculated from timestamps
- Health checks run on cron
MANUAL_COMPONENTS: post-mortem authoring, action item definition

MONITORING_AND_LOGGING (A.8.15, A.8.16)¶

Evidence	Producer Agent	Source System	Format	Frequency
k8s audit logs	rutger	k8s API server	JSON	Continuous
Application logs	all agents	Loki	Structured log entries	Continuous
PTY capture transcripts	all agents	pty_capture.py	Text with timestamps	Per session
Cost gate enforcement logs	all agents	cost_gate.py	JSON	Per enforcement event
Agent session summaries	eltjo	session_summarizer	Structured summaries	Per session
Cross-session patterns	eltjo	knowledge_synthesizer	Pattern records	6-hourly
Alert firing records	ron	Grafana	JSON	Per alert
Redis stream metrics	rutger	Redis INFO	Metrics	Hourly

AUTOMATION_LEVEL: VERY_HIGH
- All logging is automated and continuous
- PTY capture runs for every agent session
- Session summarization is inline (automatic)
- Alert records captured by Grafana
MANUAL_COMPONENTS: log review for investigation, pattern validation

CONFIGURATION_MANAGEMENT (A.8.9)¶

Evidence	Producer Agent	Source System	Format	Frequency
Config file version history	jaap	git	Git log	Continuous
k8s manifest state	rutger	k8s API	YAML	On change + monthly
Drift detection results	jaap	SSOT verification	Report	Weekly
CIS benchmark compliance	rutger	kube-bench	JSON	Monthly
Port allocation registry	jaap	config/ports.yaml	YAML	On change
Agent execution limits	jaap	config/agent-execution.yaml	YAML	On change

AUTOMATION_LEVEL: HIGH
- Git provides immutable history
- k8s state queryable via API
- Drift detection scriptable
MANUAL_COMPONENTS: configuration review decisions

Evidence	Producer Agent	Source System	Format	Frequency
Data retention configuration	boris	PostgreSQL config	SQL/Config	On change
Data deletion execution logs	boris	PostgreSQL	Audit records	Per deletion
PII detection scan results	koen	Semgrep	SARIF	Per PR
Data masking configuration	boris	PostgreSQL/app config	Config files	On change
DPA register	julian	admin-ui DB	Structured records	Per client
Breach register	julian	PostgreSQL	Structured records	Per breach
DPIA documents	julian	wiki/DB	Documents	Per project
DSR handling records	julian	admin-ui DB	Structured records	Per request
Cookie consent audit results	julian	project audit	Report	Per project
ROPA (records of processing)	julian	admin-ui DB	Structured records	Annual

AUTOMATION_LEVEL: MEDIUM
- Retention enforcement automated
- PII detection automated in CI/CD
- Deletion logging automated
MANUAL_COMPONENTS: DPA negotiation, DPIA authoring, DSR fulfillment, breach assessment

BUSINESS_CONTINUITY (A.5.29, A.5.30)¶

Evidence	Producer Agent	Source System	Format	Frequency
HA configuration	rutger	k8s API	YAML (replicas, PDB)	On change
Backup execution logs	boris	PostgreSQL backup job	Logs	Daily
Backup restoration test results	boris	Test execution	Report	Quarterly
Failover test results	rutger	Test execution	Report	Quarterly
Recovery time measurements	rutger	Test execution	Metrics	Per test
BIA documentation	julian	wiki	Document	Annual

AUTOMATION_LEVEL: MEDIUM
- Backups automated
- HA configuration declarative (k8s)
MANUAL_COMPONENTS: restoration testing, failover testing, BIA authoring

EVIDENCE_STORAGE_ARCHITECTURE¶

POSTGRESQL (SSOT)¶

TABLES:
- compliance_evidence — structured evidence records with control references
- audit_reports — internal audit reports
- corrective_actions — CAR tracking
- management_reviews — management review records
- incidents — incident records with timelines
- change_records — change management evidence (fed from GitHub webhook)
- access_reviews — quarterly access review records
- data_breaches — breach register (all breaches, not just notified)
- dpias — data protection impact assessments
- vulnerability_findings — scan results with remediation status
- session_learnings — learning extraction from agent sessions

SCHEMA_PRINCIPLES:
- Every record has: created_at, created_by, control_ref (ISO/SOC2 tag)
- Immutable records (update creates new version, old version retained)
- Foreign keys to agent registry for attribution
- Indexes on control_ref for audit queries

GIT (IMMUTABLE HISTORY)¶

TRACKED:
- All code changes (PR attribution, review records)
- Configuration changes (ports.yaml, agent-execution.yaml, etc.)
- Policy documents (constitution, CODEBASE-STANDARDS.md)
- k8s manifests (RBAC, network policies, deployments)

RETENTION: indefinite (git history never pruned for compliance-relevant repos)

WIKI (HUMAN-READABLE)¶

CONTENT:
- Policy narratives and procedures
- Post-mortem reports
- DPIA documents
- Management review minutes
- Audit report summaries
- Domain knowledge pages (this file)

PURPOSE: auditor-friendly narrative that contextualizes the structured data

EVIDENCE_REVIEW_SCHEDULE¶

CONTINUOUS (automated)¶

Trivy scans: every container build
Semgrep scans: every PR
Cost gate enforcement: every agent session
Health checks: every minute
PTY capture: every agent session
Redis stream monitoring: continuous

DAILY¶

Backup execution verification (boris)
Alert review and triage (ron)
Open incident status update (mira)

WEEKLY¶

Vulnerability triage (koen, rutger)
Drift detection review (jaap)
Cost trend review (julian)

MONTHLY¶

SLA compliance report generation (ron)
Database permission review (boris)
Change management summary (marta)
SBOM refresh (rutger)

QUARTERLY¶

Internal audit (amber) — see audit-procedures.md
Access review (julian, rutger)
Backup restoration test (boris)
Failover test (rutger)
Risk register review (julian)

ANNUALLY¶

Management review (dirk-jan, julian, amber)
Vendor assessment (julian)
Policy review and update (julian)
BIA update (julian)
ROPA update (julian)
DPIA review for ongoing projects (julian)

SOC2_CONTINUOUS_MONITORING¶

TAG:SOC2 TAG:CONTINUOUS_MONITORING

CONCEPT¶

SOC 2 Type II requires controls to be effective THROUGHOUT the examination period (not just at point of audit).
SOLUTION: automated continuous monitoring that detects control failures in near-real-time.

MONITORING_CONTROLS¶

CONTROL_CC6_ACCESS:
- MONITOR: k8s RBAC changes (k8s audit log)
- ALERT: unauthorized role binding created
- ALERT: privileged access granted outside commissioning process
- ALERT: WebAuthn authentication failures (potential unauthorized access attempts)
- FREQUENCY: continuous
- AGENT: ron (detection), julian (response)

CONTROL_CC7_OPERATIONS:
- MONITOR: system health metrics (Grafana)
- ALERT: service degradation beyond threshold
- ALERT: anomalous error rates
- ALERT: cost gate enforcement triggered (potential abuse)
- FREQUENCY: continuous
- AGENT: ron (detection), mira (response)

CONTROL_CC8_CHANGES:
- MONITOR: GitHub merge events
- ALERT: merge to main without required approvals
- ALERT: branch protection rule modification
- ALERT: container deployed without image scan
- FREQUENCY: continuous
- AGENT: marta (detection), julian (response)

CONTROL_VULNERABILITY:
- MONITOR: Trivy scan results
- ALERT: critical vulnerability in production image
- ALERT: high vulnerability unpatched beyond SLA (7 days)
- ALERT: dependency with known exploit
- FREQUENCY: per build + weekly full scan
- AGENT: rutger (detection), koen (code remediation)

CONTROL_DATA_PROTECTION:
- MONITOR: Semgrep PII detection
- ALERT: PII detected in logs or error output
- ALERT: unmasked data in non-production environment
- ALERT: data retention policy violation (data beyond retention period)
- FREQUENCY: per PR + monthly sweep
- AGENT: koen (code detection), boris (data detection)

CONTROL_AVAILABILITY:
- MONITOR: uptime and response time metrics
- ALERT: SLA breach (availability below target)
- ALERT: backup failure
- ALERT: replica count below minimum
- FREQUENCY: continuous
- AGENT: ron (detection), rutger (response)

CONTROL_FAILURE_RESPONSE¶

SEVERITY_CRITICAL (control bypass or complete failure):
1. Ron detects and alerts immediately
2. Julian assesses compliance impact
3. Corrective action initiated within 4 hours
4. Document in compliance_evidence table as control exception
5. Amber includes in next audit scope

SEVERITY_HIGH (control weakness or degradation):
1. Ron detects and creates alert
2. Julian reviews within 24 hours
3. Corrective action initiated within 7 days
4. Document as monitoring finding

SEVERITY_MEDIUM (minor deviation):
1. Detected during scheduled review
2. Tracked as improvement item
3. Address within 30 days

EVIDENCE_AUTOMATION_PIPELINE¶

ARCHITECTURE¶

Source Systems                    Collection                Storage              Review  

k8s API     ─┐  
GitHub API  ─┤  
Trivy       ─┤    audit-evidence-collect.sh     PostgreSQL    ──→  Amber  
Semgrep     ─┼──→  (scheduled + event-driven) ──→ (SSOT)          (quarterly)  
Grafana     ─┤  
Loki        ─┤                                     Wiki      ──→  Julian  
Redis       ─┤                                  (narrative)       (monthly)  
PTY capture ─┘  
                                                  Git        ──→  Jaap  
                                                (immutable)       (weekly)

COLLECTION_SCRIPT¶

SCRIPT: scripts/audit-evidence-collect.sh
PURPOSE: aggregate evidence from all source systems into PostgreSQL
SCHEDULE: daily automated run + on-demand for audit preparation
ACTIONS:
1. Query k8s API for current state (RBAC, network policies, deployments)
2. Query GitHub API for PR/merge/review data since last run
3. Collect Trivy scan results from CI/CD artifacts
4. Collect Semgrep results from CI/CD artifacts
5. Snapshot Vault access policies
6. Snapshot Redis ACL configuration
7. Calculate SLA metrics from incident data
8. Insert all evidence into PostgreSQL with control_ref tags

EVIDENCE_QUERY_EXAMPLES¶

FOR_AUDITOR (Amber):

-- All evidence for a specific control in audit period  
SELECT * FROM compliance_evidence  
WHERE control_ref = 'CC8.1'  
AND created_at BETWEEN '2026-01-01' AND '2026-03-31'  
ORDER BY created_at;  

-- Open corrective actions past due  
SELECT * FROM corrective_actions  
WHERE status NOT IN ('verified', 'closed')  
AND due_date < CURRENT_DATE;  

-- Change management summary for period  
SELECT  
  COUNT(*) as total_changes,  
  COUNT(CASE WHEN review_approved THEN 1 END) as approved_changes,  
  COUNT(CASE WHEN emergency THEN 1 END) as emergency_changes  
FROM change_records  
WHERE merged_at BETWEEN '2026-01-01' AND '2026-03-31';

FOR_JULIAN (compliance reporting):

-- Control coverage: which controls have evidence, which don't  
SELECT DISTINCT control_ref,  
  COUNT(*) as evidence_count,  
  MAX(created_at) as latest_evidence  
FROM compliance_evidence  
GROUP BY control_ref  
ORDER BY latest_evidence;  

-- Breach register for DPA reporting  
SELECT * FROM data_breaches  
WHERE detected_at >= '2025-01-01'  
ORDER BY detected_at DESC;

AUTOMATION_MATURITY_ROADMAP¶

CURRENT_STATE: ~60% automated
TARGET_STATE: 80%+ automated

PHASE_1 (current):
- Trivy, Semgrep, kube-bench automated
- PTY capture automated
- Cost gate automated
- Git history available
GAPS: evidence aggregation manual, SLA calculation manual, report generation manual

PHASE_2 (next):
- Build audit-evidence-collect.sh script
- Automate SLA metric calculation
- Automate change management summary generation
- Build compliance dashboard in admin-ui
TARGET: 75% automated

PHASE_3 (future):
- Automated evidence freshness monitoring (alert if evidence stale)
- Automated audit preparation package generation
- Integration with external audit platforms
- Automated control effectiveness scoring
TARGET: 85% automated

CONTROL_TO_EVIDENCE_MATRIX (COMPLETE)¶

Control	Description	Evidence Type	Producer	Auto?
CC1.1	Integrity/ethics	Constitution, violations	julian, amber	Partial
CC1.2	Management oversight	Commission records, review minutes	amber, dirk-jan	Manual
CC1.3	Org structure	AGENT-REGISTRY.json	julian	Auto
CC1.5	Accountability	DAG records, PTY captures, cost reports	all agents	Auto
CC2.1	Quality information	DB records, wiki content	annegreet, eltjo	Auto
CC2.2	Internal comms	Constitution ack, Redis records	julian	Auto
CC3.2	Risk identification	Vuln reports, threat intel	julian, annegreet	Partial
CC4.1	Control evaluation	Scan results, audit reports	ron, amber	Auto/Manual
CC5.2	Tech controls	RBAC, network policies, Vault	rutger	Auto
CC6.1	Infra access	RBAC bindings, WebAuthn, Vault	rutger, julian	Auto
CC6.2	Registration	Commissioning records	julian	Manual
CC6.3	Access removal	Decommission records, revocation logs	julian, rutger	Partial
CC6.4	Access restriction	Network policies, DB permissions	boris, rutger	Auto
CC6.5	Boundary protection	Service manifests, ingress config	rutger	Auto
CC6.6	Encryption in transit	TLS certs, SSL config	rutger	Auto
CC6.7	Credential mgmt	Vault config, secret scans	rutger	Auto
CC6.8	Threat protection	Trivy, Semgrep, adversarial tests	ashley, koen, rutger	Auto
CC7.1	Change detection	Git log, k8s audit, SSOT checks	jaap, ron	Auto
CC7.2	Anomaly monitoring	Alert config, dashboards	ron	Auto
CC7.3	Event evaluation	Assessment logs	ron, mira	Partial
CC7.4	Incident response	Incident records, timelines	mira	Auto
CC7.5	Recovery	Recovery docs, test results	rutger, mira	Partial
CC8.1	Change authorization	PR records, merge approvals	marta, koen	Auto
CC8.2	Change testing	Test results, adversarial reports	marije, ashley	Auto
CC8.3	Change documentation	Git log, PR descriptions	marta	Auto
CC8.4	Emergency changes	Incident records, hotfix logs	mira	Partial
CC9.1	Vendor risk	Assessments, DPAs	julian	Manual
A1.1	Performance monitoring	Dashboards, SLA reports	ron	Auto
A1.2	Availability recovery	PDB config, failover results	rutger	Partial
A1.3	Recovery testing	Test results	rutger, boris	Manual
A.8.9	Config management	Git history, kube-bench	jaap, rutger	Auto
A.8.10	Info deletion	Retention config, deletion logs	boris	Auto
A.8.11	Data masking	Masking config, PII scans	boris, koen	Auto
A.8.12	DLP	Secret scans, network policies	ron, rutger	Auto
A.8.15	Logging	Log config, audit trails	ron, rutger	Auto
A.8.28	Secure coding	Semgrep, review records	koen	Auto

READ_ALSO: audit-procedures.md, iso27001-controls.md, soc2-criteria.md, gdpr-implementation.md