apophis-fastify/docs/attic/TEST_AUDIT_REPORT.md

APOPHIS Test Quality Audit Report

Date: 2026-04-29 Scope: 55 test files, ~20,450 lines Auditors: 3 parallel subworkers (CLI tests, Domain/Core tests, Feature tests)

---

Executive Summary

Category	Count	Lines	Verdict
CLI Tests	18 files	~9,209 lines	10 KEEP, 3 MERGE, 4 REFACTOR, 1 DELETE
Domain/Core Tests	11 files	~4,500 lines	8 KEEP, 1 MERGE, 2 REFACTOR
Feature Tests	26 files	~6,741 lines	20 KEEP, 2 MERGE, 4 REFACTOR, 3 DELETE
Total	55 files	~20,450 lines	38 KEEP, 6 MERGE, 10 REFACTOR, 4 DELETE

Key Findings:

• 4 test files test non-production helpers (cascade-validator, hypermedia-validator, etc.)
• 6 files have significant overlap with other tests
• 10 files need refactoring (temp app approach broken, implementation testing, weak assertions)
• 38 files provide unique, valuable coverage

---

Critical Issues (Fix First)

1. Broken Test Approach: verify-ux.test.ts

• Status: 16 of 20 tests FAIL (80% failure rate)
• Root cause: Creates temp app.js files that aren't valid Fastify apps
• Impact: Unreliable regression protection
• Fix: Switch to fixture apps (src/cli/__fixtures__/) or create new fixtures

2. Duplicate Tests: integration.test.ts

• Status: 3 pairs of duplicate/near-duplicate tests (6 tests)
• Impact: Wasted CI time, no added coverage
• Fix: Remove duplicates

3. Non-Production Helpers: cascade-validator.test.ts, hypermedia-validator.test.ts

• Status: Test helpers that were merged into test files, never imported by production code
• Impact: Test maintenance burden for dead code
• Fix: Delete (production coverage exists in relationships.test.ts)

4. Inline Copies: deduplication.test.ts

• Status: Contains stale copies of deduplicatePetit/deduplicateStateful
• Impact: Tests don't exercise actual production code
• Fix: Import from runner-utils.ts instead

---

CLI Test Audit (18 files)

KEEP (10 files)

File	Tests	Value	Why
docs-smoke.test.ts	4	Unique	Only test verifying documentation accuracy
goldens.test.ts	9	High	Guards CLI output against accidental changes
init.test.ts	17	Unique	Only deep init coverage
latency.test.ts	5	Unique	Performance regression guards
migrate-reliability.test.ts	20	Unique	Canonical migrate test, 80% coverage
observe-safety.test.ts	20	Unique	Only policy engine + observe integration
packaging.test.ts	15	Unique	Only test of built binary
qualify-signal.test.ts	16	Unique	Only artifact structure validation
renderers.test.ts	18	Unique	Only renderer function tests
replay-integrity.test.ts	10	Unique	Only replay loader/schema tests

MERGE (3 files)

File	Target	Reason
core.test.ts	dispatch.test.ts	Tests same CLI entrypoint, weaker assertions
migrate.test.ts	migrate-reliability.test.ts	Subset coverage, 15 tests vs 20
observe.test.ts	observe-safety.test.ts	Keep fixture-based tests only

REFACTOR (4 files)

File	Issue	Fix
acceptance.test.ts	8 tests fail due to fixture instability	Use main() entrypoint, drop failing tests
config-validation.test.ts	271 tests, many permutations	Collapse to ~50 parameterized tests
doctor-consistency.test.ts	5 tests fail (temp apps not valid)	Use fixture apps instead
verify-ux.test.ts	16 of 20 tests fail	Switch to fixture apps

DELETE (after merge)

• core.test.ts → merged into dispatch
• migrate.test.ts → merged into migrate-reliability
• observe.test.ts → merged into observe-safety

---

Domain/Core Test Audit (11 files)

KEEP (8 files)

File	Tests	Value
domain.test.ts	45	Foundational classification rules
formula.test.ts	~85	Core parser/evaluator, property tests
extension.test.ts	36	Registry/framework, no overlap
infrastructure.test.ts	15	ScopeRegistry, CleanupManager, HookValidator
error-context.test.ts	24	Core contract validation
error-suggestions.test.ts	31	Exhaustive suggestion branches
cross-operation-support.test.ts	8	Only integration tests for previous()
protocol-extensions.test.ts	22	Built-in extensions

MERGE (1 file)

File	Target	Reason
examples.test.ts	integration.test.ts	Redundant smoke tests

REFACTOR (2 files)

File	Issue	Fix
integration.test.ts	6 duplicate/near-duplicate tests	Remove duplicates
success-metrics.test.ts	Arbitrary thresholds, covered elsewhere	Delete (assertions in error-context + integration)

---

Feature Test Audit (26 files)

KEEP (20 files)

File	Tests	Value
cache-hints.test.ts	7	Cache invalidation patterns
counterexample.test.ts	17	Failure analysis + formatting
debug-mode.test.ts	2	Debug logging toggle
incremental.test.ts	12	Hash determinism
incremental/cache.test.ts	7	Cache API round-trip
invariant-registry.test.ts	5	Invariant resolution
outbound-interceptor.test.ts	16	Chaos application
outbound-runtime.test.ts	10	Outbound registry + mocks
outbound-stateful.test.ts	7	Stateful mock CRUD
production-safety.test.ts	4	Production guards
regex-guard.test.ts	13	ReDoS protection
relationships.test.ts	9	Production relationship predicates
resource-inference.test.ts	13	Schema-driven identity
route-matcher.test.ts	17	URL pattern matching
scenario-runner.test.ts	6	Scenario capture/rebind/cookies
schema-to-arbitrary.test.ts	33	Schema-to-fast-check (property tests)
scope-isolation.test.ts	4	Scope filtering
serverless.test.ts	3	Serverless compatibility
stateful-runner.test.ts	6	Stateful test execution
tap-formatter.test.ts	15	TAP output formatting

MERGE (2 files)

File	Target	Reason
format-diff.test.ts	counterexample.test.ts	Only 4 tests, same module
seeded-rng.test.ts	schema-to-arbitrary.test.ts	5 tests, RNG core to generation

REFACTOR (4 files)

File	Issue	Fix
deduplication.test.ts	Stale copies of production code	Import from runner-utils.ts
incremental/cache.test.ts	Weak "persists to disk" test	Fix or remove
counterexample.test.ts	Growing file (224L)	Split if exceeds 250L
tap-formatter.test.ts	Same module as counterexample	Consider unified formatters.test.ts

DELETE (4 files)

File	Reason	Coverage Moves To
cascade-validator.test.ts	Tests non-production helpers	relationships.test.ts
hypermedia-validator.test.ts	Tests non-production helpers	relationships.test.ts
gap-fixes.test.ts	Runtime hooks → infrastructure, chaos → outbound-interceptor	infrastructure.test.ts, outbound-interceptor.test.ts
success-metrics.test.ts	Arbitrary metrics, covered elsewhere	error-context.test.ts, integration.test.ts

---

Action Plan

Phase A: Fix Broken Tests (Week 1)

1. Refactor verify-ux.test.ts - Switch to fixture apps
2. Refactor doctor-consistency.test.ts - Use fixture apps for failing tests
3. Refactor acceptance.test.ts - Remove failing tests, use main() entrypoint
4. Remove duplicates from integration.test.ts - 6 tests

Phase B: Delete Dead Tests (Week 1)

1. Delete cascade-validator.test.ts
2. Delete hypermedia-validator.test.ts
3. Delete gap-fixes.test.ts (after moving valuable tests)
4. Delete success-metrics.test.ts

Phase C: Merge Overlapping Tests (Week 2)

1. Merge core.test.ts → dispatch.test.ts
2. Merge migrate.test.ts → migrate-reliability.test.ts
3. Merge observe.test.ts → observe-safety.test.ts
4. Merge examples.test.ts → integration.test.ts
5. Merge format-diff.test.ts → counterexample.test.ts
6. Merge seeded-rng.test.ts → schema-to-arbitrary.test.ts

Phase D: Refactor Implementation Tests (Week 2)

1. Refactor deduplication.test.ts - Use real imports
2. Refactor config-validation.test.ts - Parameterize permutations
3. Fix incremental/cache.test.ts - Strengthen or remove weak test

---

Impact Projection

Metric	Current	After	Change
Test files	55	~45	-10 (-18%)
Test lines	~20,450	~18,000	-2,450 (-12%)
Failing tests	~20	0	-20 (100%)
Duplicate tests	~15	0	-15 (100%)
Non-production tests	4 files	0	-4 (100%)

Coverage target: Retain or move the useful assertions before deleting overlapping tests.

---

Test Quality Principles Applied

1. Behavior over implementation - Tests should verify observable behavior, not internal structure
2. Fixtures over temp files - Use stable fixture apps instead of generating temp app.js files
3. Parameterized over permutations - One test with multiple inputs beats 10 identical tests
4. Production over helpers - Test production code, not test-only helpers
5. Independence - Each test should create its own context, not depend on global state

---

Report generated from static analysis of all 55 test files. No code changes made.