fix: correct documented vs actual behavior discrepancies from subworker audit

- Fix verify.md --changed exit code (0 → 2)
- Add 'deep' as alias for 'thorough' in generation profile resolution
- Fix PLUGIN_CONTRACTS_SPEC status: Partially implemented (registry done, runner pending)
- Fix OUTBOUND_CONTRACT_MOCKING_SPEC status: Implemented (Phase 1)
- Fix cli.md environment matrix to match actual code granularity
- Fix chaos.md: document delay handler is currently a no-op
- Fix getting-started.md warning: note APOPHIS does not proactively detect nondeterminism
- Add variants section to getting-started.md
- Build: clean | Tests: 849 pass, 0 fail
This commit is contained in:
John Dvorak
2026-04-30 11:50:39 -07:00
parent bf7376b5ad
commit dc7a4205ec
7 changed files with 56 additions and 15 deletions
+4 -1
View File
@@ -1,6 +1,9 @@
## Outbound Contract-Driven Mocking Spec ## Outbound Contract-Driven Mocking Spec
Status: Proposed Status: Implemented (Phase 1)
Phase 1 (implemented): Schema parsing (`x-outbound`), mock runtime, imperative API (`enableOutboundMocks`, `getOutboundCalls`), fetch patching.
Phase 2 (pending): APOSTL extensions `outbound_calls(this)` and `outbound_last(this)` for contract assertions.
Date: 2026-04-27 Date: 2026-04-27
This document supersedes Arbiter's local draft at `~/Business/workspace/Arbiter/docs/APOPHIS_OUTBOUND_MOCK_PROPOSAL.md` and its interim adapter at `~/Business/workspace/Arbiter/src/server/server/services/StripeFetchAdapter.js`. This document supersedes Arbiter's local draft at `~/Business/workspace/Arbiter/docs/APOPHIS_OUTBOUND_MOCK_PROPOSAL.md` and its interim adapter at `~/Business/workspace/Arbiter/src/server/server/services/StripeFetchAdapter.js`.
+5 -1
View File
@@ -1,6 +1,10 @@
# APOPHIS Plugin Contract System Specification # APOPHIS Plugin Contract System Specification
## Status: Implemented ## Status: Partially implemented
- Registry, types, and registration API: **implemented**
- Runner integration (merging plugin contracts into route execution): **pending**
- Built-in contracts for `@fastify/auth`, `@fastify/compress`, `@fastify/cors`, `@fastify/rate-limit`: **registered but not yet applied**
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins. **Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
+2
View File
@@ -30,6 +30,8 @@ timeout_occurred(this) == false
response_time(this) < 1000 response_time(this) < 1000
``` ```
**Note**: Delay events are generated by the chaos arbitrary but the inbound delay handler is currently a no-op. Use this for timeout contract documentation; actual delay injection requires the outbound delay strategy or a custom handler.
### Error ### Error
Forces HTTP status codes. Tests error-handling contracts: Forces HTTP status codes. Tests error-handling contracts:
+7 -5
View File
@@ -255,10 +255,12 @@ apophis replay --artifact reports/apophis/failure-*.json
|---|---|---|---|---| |---|---|---|---|---|
| `verify` | enabled | enabled | optional | optional, usually off | | `verify` | enabled | enabled | optional | optional, usually off |
| `observe` | optional | optional | enabled | enabled | | `observe` | optional | optional | enabled | enabled |
| `qualify: scenario` | enabled | enabled | enabled with allowlist | disabled by default | | `qualify` | enabled | enabled | optional | disabled by default |
| `qualify: stateful` | enabled | enabled | synthetic-only | disabled by default | | `chaos` | enabled | enabled | optional | disabled by default |
| `qualify: chaos` | enabled | enabled | canary-only | disabled by default |
| outbound mocks | enabled | enabled | allowlisted only | disabled by default |
| runtime throw-on-violation | optional | optional | exceptional | disabled by default | | runtime throw-on-violation | optional | optional | exceptional | disabled by default |
Operational rule: Production must never inherit qualify capabilities accidentally from a generic config file. Notes:
- `qualify` is gated as a whole. The code does not distinguish scenario, stateful, and chaos sub-modes in environment policy.
- `chaos` on protected routes requires `allowChaosOnProtected: true`.
- `observe` blocking requires `allowBlocking: true`.
- Production must never inherit qualify capabilities accidentally from a generic config file.
+26 -1
View File
@@ -50,7 +50,7 @@ app.post('/users', {
}); });
``` ```
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead. > **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead. APOPHIS does not proactively detect nondeterministic handlers; it warns only when a replay diverges from the original run.
## Step 4: Run Verify ## Step 4: Run Verify
@@ -105,6 +105,31 @@ Fix the bug in your handler. Re-run verify. The failure should now pass.
- Add observe mode for runtime drift detection: see [observe.md](observe.md) - Add observe mode for runtime drift detection: see [observe.md](observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md) - Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
## Variants
Test the same route with different headers or content types:
```javascript
await fastify.apophis.contract({
variants: [
{ name: 'json', headers: { accept: 'application/json' } },
{ name: 'xml', headers: { accept: 'application/xml' } }
]
})
```
Or declare variants in the route schema:
```javascript
app.get('/users', {
schema: {
'x-variants': [
{ name: 'json', headers: { accept: 'application/json' } }
]
}
})
```
## Config Reference ## Config Reference
For the full configuration reference, see [CLI Reference](cli.md). For the full configuration reference, see [CLI Reference](cli.md).
+1 -1
View File
@@ -81,7 +81,7 @@ Run only routes modified in the current git branch:
apophis verify --profile ci --changed apophis verify --profile ci --changed
``` ```
If no routes changed, exits 0 with a message. If no routes changed, exits 2 with a message.
## Failure Output Format ## Failure Output Format
+11 -6
View File
@@ -10,7 +10,12 @@ export class GenerationProfileResolutionError extends Error {
} }
function isBuiltInProfile(value: string): value is ResolvedGenerationProfile { function isBuiltInProfile(value: string): value is ResolvedGenerationProfile {
return value === 'quick' || value === 'standard' || value === 'thorough' return value === 'quick' || value === 'standard' || value === 'thorough' || value === 'deep'
}
function normalizeProfile(value: string): ResolvedGenerationProfile {
if (value === 'deep') return 'thorough'
return value as ResolvedGenerationProfile
} }
export function resolveGenerationProfileOverride( export function resolveGenerationProfileOverride(
@@ -22,13 +27,13 @@ export function resolveGenerationProfileOverride(
} }
if (isBuiltInProfile(rawProfile)) { if (isBuiltInProfile(rawProfile)) {
return rawProfile return normalizeProfile(rawProfile)
} }
const aliases = config.generationProfiles const aliases = config.generationProfiles
if (!aliases) { if (!aliases) {
throw new GenerationProfileResolutionError( throw new GenerationProfileResolutionError(
`Unknown generation profile "${rawProfile}". Use one of: quick, standard, thorough, or define an alias in config.generationProfiles.`, `Unknown generation profile "${rawProfile}". Use one of: quick, standard, deep, or define an alias in config.generationProfiles.`,
) )
} }
@@ -36,16 +41,16 @@ export function resolveGenerationProfileOverride(
if (!alias) { if (!alias) {
const available = Object.keys(aliases).join(', ') || 'none' const available = Object.keys(aliases).join(', ') || 'none'
throw new GenerationProfileResolutionError( throw new GenerationProfileResolutionError(
`Unknown generation profile "${rawProfile}". Built-ins: quick, standard, thorough. Config aliases: ${available}.`, `Unknown generation profile "${rawProfile}". Built-ins: quick, standard, deep. Config aliases: ${available}.`,
) )
} }
const target = typeof alias === 'string' ? alias : alias.base const target = typeof alias === 'string' ? alias : alias.base
if (!isBuiltInProfile(target)) { if (!isBuiltInProfile(target)) {
throw new GenerationProfileResolutionError( throw new GenerationProfileResolutionError(
`Invalid generation profile alias "${rawProfile}". Alias must resolve to quick, standard, or thorough.`, `Invalid generation profile alias "${rawProfile}". Alias must resolve to quick, standard, or deep.`,
) )
} }
return target return normalizeProfile(target)
} }