apophis/apophis-fastify
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public release of Apophis — invariant-driven automated API testing

Browse Source
This commit is contained in:
John Dvorak
2026-03-10 00:00:00 -07:00
parent d278c4b105
commit 3ac1daf7e9
82 changed files with 3902 additions and 1098 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/attic/OUTBOUND_CONTRACT_MOCKING_SPEC.md
+519
View File
@@ -0,0 +1,519 @@
## Outbound Contract-Driven Mocking Spec
Status: Implemented (Phase 1)
Phase 1 (implemented): Schema parsing (`x-outbound`), mock runtime, imperative API (`enableOutboundMocks`, `getOutboundCalls`), fetch patching.
Phase 2 (pending): APOSTL extensions `outbound_calls(this)` and `outbound_last(this)` for contract assertions.
Date: 2026-04-27
This document supersedes Arbiter's local draft at `~/Business/workspace/Arbiter/docs/APOPHIS_OUTBOUND_MOCK_PROPOSAL.md` and its interim adapter at `~/Business/workspace/Arbiter/src/server/server/services/StripeFetchAdapter.js`.
The direction in that proposal is correct: routes should be able to declare the contracts and expectations of their outbound dependencies, and APOPHIS should use those declarations to generate mocks, inject dependency-layer chaos, and support both contract testing and imperative E2E testing.
This spec keeps that idea small and consistent with the runtime paths APOPHIS already has.
## Goals
1. Let routes declare outbound dependency contracts once and reuse them anywhere.
2. Generate contract-conformant outbound mock responses from JSON Schema.
3. Apply chaos at the dependency layer, before application code receives the response.
4. Record outbound calls so tests and contracts can inspect them.
5. Work in both APOPHIS contract tests and imperative E2E tests.
6. Reuse existing chaos, fast-check, and flake-detection infrastructure.
7. Avoid service-specific adapters and avoid a second testing engine.
## Non-Goals
1. No Stripe-specific or service-specific code in APOPHIS.
2. No second DSL for outbound expectations.
3. No new backward-compatibility layer for old chaos config.
4. No static JS analysis in this cut. The design must enable it later, not implement it now.
## Parsimony Rules
1. One schema annotation: `x-outbound`.
2. One shared registry: `outboundContracts`.
3. One runtime owner: `OutboundMockRuntime`.
4. One fetch interception path: reuse `wrapFetch()` and `createOutboundInterceptor()` instead of inventing another chaos stack.
5. One property-generation engine: reuse `convertSchema()` for dependency responses instead of creating a second generator pipeline.
6. One seeded randomness model: derive outbound mock randomness from the same test seed via sub-seeds, never `Date.now()`.
7. No service adapters in core. If an application wants an adapter, it should be a thin local wrapper over fetch, not the core abstraction.
## Core Design
### 1. Shared outbound contracts are registered once
Add a plugin-level registry:
```ts
await fastify.register(apophis, {
outboundContracts: {
'stripe.paymentIntents.create': {
target: 'https://api.stripe.com/v1/payment_intents',
method: 'POST',
request: { ...json schema... },
response: {
200: { ...json schema... },
402: { ...json schema... },
429: { ...json schema... }
},
chaos: {
error: {
probability: 0.02,
responses: [
{ statusCode: 429, headers: { 'retry-after': '60' } },
{ statusCode: 503, body: { error: { type: 'api_error' } } }
]
}
}
}
}
})
```
### 2. Routes reference or inline outbound contracts with one annotation
Do not add `x-outbound-uses` and `x-outbound-contracts` as two separate concepts. Use one annotation:
```ts
schema: {
'x-outbound': [
'stripe.paymentIntents.create',
{
ref: 'stripe.customers.retrieve',
chaos: {
error: {
probability: 1,
responses: [{ statusCode: 404, body: { error: { type: 'invalid_request_error' } } }]
}
}
},
{
name: 'audit.events.write',
target: 'https://audit.internal/v1/events',
method: 'POST',
request: { ...json schema... },
response: { 202: { ...json schema... } }
}
],
'x-ensures': [
'if response_code(this) == 200 then response_body(this).paid == true else true'
]
}
```
Why this shape:
1. One-off dependencies can be inline.
2. Shared dependencies can be referenced by name.
3. Route-local chaos overrides are possible without duplicating the shared contract.
4. We do not create a second metadata system just to support references.
### 3. APOPHIS owns the outbound runtime in test mode
Contract tests and stateful tests should automatically install outbound mocking when a route declares `x-outbound`.
Imperative E2E tests should be able to opt in manually:
```ts
await fastify.apophis.enableOutboundMocks({
contracts: ['stripe.paymentIntents.create'],
mode: 'property',
overrides: {
'stripe.paymentIntents.create': {
forceStatus: 402,
body: { error: { type: 'card_error', code: 'card_declined' } }
}
}
})
// normal app-level test code here
const calls = fastify.apophis.getOutboundCalls('stripe.paymentIntents.create')
await fastify.apophis.disableOutboundMocks()
```
This is the right place for E2E support. It keeps imperative tests imperative while letting APOPHIS provide deterministic dependency behavior.
### 4. Outbound expectations should reuse APOSTL, not invent a second DSL
Do not add a new outbound assertion language.
Expose outbound call facts through a built-in extension surface so existing `x-ensures` and `x-requires` can talk about dependency behavior.
Target shape:
```ts
'outbound_calls(this).stripe.paymentIntents.create.count == 1'
'outbound_last(this).stripe.paymentIntents.create.response.statusCode == 402'
'if outbound_last(this).stripe.paymentIntents.create.response.statusCode == 402 then response_code == 400 else true'
```
This keeps all behavioral expectations in APOSTL.
Implementation note: contract names should be dot-separated identifiers so they naturally project into accessor paths.
### 5. Property-based testing should run on both sides
Today we generate route inputs from request schemas. We should also be able to generate dependency outputs from outbound response schemas.
That means APOPHIS can test:
1. many valid caller requests to the route
2. many valid dependency responses allowed by the outbound contract
3. whether the route still satisfies its own postconditions
This is not a second property engine. It is an augmentation of the existing command generation and execution flow.
## Runtime Model
For a single route execution under contract testing:
1. runner resolves the route's `x-outbound` bindings
2. bindings are normalized against the shared registry
3. an `OutboundMockRuntime` is installed for the duration of that request execution
4. `globalThis.fetch` is wrapped in test mode for the duration of execution
5. outbound calls matching a resolved contract return generated or overridden responses
6. the existing outbound chaos layer decorates those responses
7. call traces are recorded
8. after the request completes, fetch is restored
9. the recorded outbound facts are attached to the eval context for formulas and diagnostics
For imperative E2E:
1. test calls `enableOutboundMocks()`
2. runtime installs fetch patch once
3. test drives the app normally
4. test inspects `getOutboundCalls()`
5. test calls `disableOutboundMocks()`
## Public API Changes
### `ApophisOptions`
Add shared outbound contract registration:
```ts
readonly outboundContracts?: Record<string, OutboundContractSpec>
```
### `TestConfig`
Add runner-level outbound mock control:
```ts
readonly outboundMocks?: false | {
readonly mode?: 'example' | 'property'
readonly contracts?: readonly string[]
readonly overrides?: Record<string, {
readonly forceStatus?: number
readonly headers?: Record<string, string>
readonly body?: unknown
}>
readonly unmatched?: 'error' | 'passthrough'
}
```
Notes:
1. `false` disables outbound mocking even if routes declare `x-outbound`.
2. `mode: 'example'` returns one contract-conformant response per dependency and is the default.
3. `mode: 'property'` samples across documented dependency responses.
4. `unmatched: 'error'` should be the default in test mode to prevent accidental real network access.
### `ApophisDecorations`
Add imperative test helpers:
```ts
readonly registerOutboundContracts: (contracts: Record<string, OutboundContractSpec>) => void
readonly enableOutboundMocks: (opts?: TestConfig['outboundMocks']) => Promise<void>
readonly disableOutboundMocks: () => Promise<void>
readonly getOutboundCalls: (name?: string) => ReadonlyArray<OutboundCallRecord>
```
## Concrete File Plan
Line numbers below are current as of 2026-04-27 and should be rechecked before editing.
### 1. `src/types.ts`
Current anchors:
1. `RouteContract`: lines 28-42
2. `TestConfig`: lines 256-264
3. `ChaosConfig`: lines 308-355
4. `ApophisOptions`: lines 498-519
5. `ApophisDecorations`: lines 612-627
Modify:
1. Add `outbound?: readonly OutboundBinding[]` to `RouteContract` at lines 28-42.
2. Add `OutboundContractSpec`, `OutboundBinding`, `ResolvedOutboundContract`, and `OutboundCallRecord` near existing outbound chaos types at lines 266-356.
3. Add `outboundMocks?: false | { ... }` to `TestConfig` at lines 256-264.
4. Add `outboundContracts?: Record<string, OutboundContractSpec>` to `ApophisOptions` at lines 498-519.
5. Add `registerOutboundContracts`, `enableOutboundMocks`, `disableOutboundMocks`, and `getOutboundCalls` to `ApophisDecorations` at lines 612-627.
Keep it parsimonious:
1. Do not add a second chaos config type.
2. Do not add separate inline-vs-reference types if a discriminated union on `x-outbound` handles both.
3. Keep `OutboundContractSpec` JSON-Schema-centric so `convertSchema()` can reuse it directly.
### 2. `src/domain/contract.ts`
Current anchor: lines 35-95 extract `x-category`, `x-requires`, `x-ensures`, and `x-timeout`.
Modify:
1. Parse `schema['x-outbound']`.
2. Normalize string refs and inline objects into `RouteContract.outbound`.
3. Preserve current caching behavior in `contractCache`.
Keep it parsimonious:
1. Normalize shape here once.
2. Do not resolve references here because this module does not own plugin-level registries.
3. Do not add outbound-specific execution logic here.
### 3. `src/plugin/index.ts`
Current anchor: lines 29-102 own plugin setup, registries, route capture, and decorations.
Modify:
1. Instantiate an `OutboundContractRegistry` and `OutboundMockRuntime` next to existing registries.
2. Register `opts.outboundContracts` during plugin setup.
3. Add imperative outbound mock decorations.
4. Register a built-in outbound extension that exposes `outbound_calls(this)` and `outbound_last(this)`.
Keep it parsimonious:
1. Do not create a separate plugin or extension package for outbound support.
2. Reuse existing plugin lifecycle instead of bolting on a second orchestrator.
### 4. `src/plugin/contract-builder.ts`
Current anchor: lines 14-29 build the config passed into `runPetitTests()`.
Modify:
1. Pass through `opts.outboundMocks`.
Keep it parsimonious:
1. No logic here beyond forwarding config.
### 5. `src/plugin/stateful-builder.ts`
Current anchor: lines 13-29 build the config passed into `runStatefulTests()`.
Modify:
1. Pass through `opts.outboundMocks`.
2. Keep parity with `contract-builder.ts`.
### 6. `src/test/petit-runner.ts`
Current anchors:
1. lines 237-243 function signature
2. line 322 constructs `EnhancedChaosEngine`
3. lines 342-430 build request and execute the route
4. lines 497-512 attach chaos diagnostics
Modify:
1. Resolve `route.outbound` against the shared registry before execution.
2. Install `OutboundMockRuntime` around the single route execution.
3. If `chaosEngine.executeWithChaos()` returns `outboundInterceptor`, compose it into the runtime instead of inventing a second path.
4. Attach outbound call trace into diagnostics and eval context.
5. In property mode, expand outbound response scenarios using `convertSchema(..., { context: 'response' })` and deterministic seeds.
Keep it parsimonious:
1. Do not create a second runner.
2. Do not fork request generation logic; augment the existing execution loop.
3. Reuse the runner seed and `SeededRng` instead of introducing local randomness.
### 7. `src/test/stateful-runner.ts`
Current anchors:
1. line 25 imports legacy `ChaosEngine` from `../quality/chaos.js`
2. line 62 stores `chaosEngine?: ChaosEngine`
3. lines 272-279 execute with chaos
4. line 394 constructs `new ChaosEngine(config.chaos, config.seed)`
Modify:
1. Migrate stateful testing to `EnhancedChaosEngine` from `src/quality/chaos-v2.ts`.
2. Install the same outbound mock runtime used by `petit-runner.ts`.
3. Use the same outbound scenario generation rules so stateful and contract runners do not diverge.
Keep it parsimonious:
1. Do not maintain two chaos stacks.
2. Do not implement outbound mocking twice.
### 8. `src/quality/chaos-v2.ts`
Current anchors:
1. lines 52-125 `wrapFetch()`
2. lines 148-188 seed management and outbound interceptor construction
3. lines 214-274 route execution and outbound interceptor attachment
Modify:
1. Keep `wrapFetch()` as the only fetch interception primitive.
2. Add a small composition helper if needed so `OutboundMockRuntime` can run `mock response -> outbound chaos overlay -> Response`.
3. Keep per-route chaos resolution in `buildOutboundInterceptor()`.
Keep it parsimonious:
1. Do not move mock generation into chaos-v2.
2. Chaos owns chaos, not contract generation.
### 9. `src/quality/chaos-outbound.ts`
Current anchor: lines 36-105 create the pure outbound interceptor.
Modify:
1. No structural redesign required.
2. Ensure the interceptor remains transport-agnostic and can wrap both real fetch and mock responders.
Keep it parsimonious:
1. This file should stay pure.
2. Do not add runtime registry logic here.
### 10. `src/domain/schema-to-arbitrary.ts`
Current anchor: lines 214-217 export `convertSchema()`.
Modify:
1. Reuse `convertSchema(responseSchema, { context: 'response' })` for generated dependency responses.
2. Add a tiny helper for weighted status-code sampling if needed, but do not fork the schema conversion logic.
Keep it parsimonious:
1. No second schema generator.
2. No outbound-specific arbitrary builder unless it is only a thin composition over `convertSchema()`.
### 11. `src/quality/flake.ts`
Current anchor: lines 56-97 derive reruns from a seed.
Modify:
1. Public API can stay unchanged if outbound runtime derives every sub-seed from the rerun seed.
2. If diagnostics are added, include outbound scenario seed in rerun metadata, but do not add a separate flake engine.
Keep it parsimonious:
1. Flake support should come from determinism, not from more feature flags.
### 12. New files
Add only these new files:
1. `src/domain/outbound-contracts.ts`
- normalize, resolve, and validate `x-outbound` bindings against the shared registry
2. `src/infrastructure/outbound-mock-runtime.ts`
- install and restore fetch patch
- record calls
- resolve overrides
- return generated or overridden responses
3. `src/extensions/outbound.ts`
- built-in extension exposing outbound call facts to APOSTL
Do not add service-specific adapters, provider-specific modules, or a separate outbound runner.
## Interaction With Existing Chaos
The order must be:
1. route declares outbound contract
2. runtime resolves contract
3. runtime generates or overrides mock response
4. existing outbound chaos interceptor applies delay/error/dropout if configured
5. application code receives the final dependency response
This keeps chaos at the correct layer and reuses the current outbound chaos implementation.
Do not invert the order by making chaos choose a response before the contract mock runtime runs. Outbound mocking must generate the dependency response first; chaos then mutates or delays that response.
## Interaction With flake detection
This feature must be deterministic under a single seed.
Rules:
1. route command generation already depends on `config.seed`
2. outbound response generation must derive from that same seed
3. per-contract sampling must use stable sub-seeds, e.g. `hashCombine(seed, stableHash(contractName))`
4. route-local chaos and outbound mock generation must not perturb each other beyond their dedicated sub-streams
If these rules hold, `FlakeDetector` needs no public redesign.
## Interaction With property-based testing
Phase 1:
1. `mode: 'example'` uses one generated success response per dependency plus explicit override cases.
2. This is enough to support contract tests and imperative E2E immediately.
Phase 2:
1. `mode: 'property'` samples across every documented outbound status code.
2. For each sampled dependency response, APOPHIS executes the route and checks route postconditions.
3. This gives property-based testing on both sides of the integration boundary.
We should not block phase 1 on phase 2, but the types and runtime must be designed so phase 2 is an additive change, not a rewrite.
## Suggested Test Plan
Add tests in these areas:
1. `src/test/domain.test.ts`
- `extractContract()` parses `x-outbound` string refs
- `extractContract()` parses inline outbound contracts
- route-local chaos overrides are preserved
2. `src/test/outbound-runtime.test.ts`
- generated success response matches schema shape
- override response takes precedence
- unmatched fetch throws by default in test mode
- call recording works
- fetch patch restore is correct
3. `src/test/outbound-interceptor.test.ts`
- existing outbound chaos still works when wrapping a mock executor
4. `src/test/integration.test.ts`
- `fastify.apophis.contract()` with `x-outbound` exercises dependency-layer failures
- `enableOutboundMocks()` supports imperative E2E style
5. `src/test/stateful-runner.test.ts` or new stateful integration tests
- stateful runner uses the same outbound runtime and chaos path
6. `src/test/flake.test.ts` new
- same seed gives same outbound responses and same call trace
- different seeds explore different dependency outputs without nondeterministic drift
## Migration Guidance
For Arbiter:
1. move the Stripe contract definitions out of `src/server/server/services/StripeFetchAdapter.js`
2. register them once via `outboundContracts`
3. change route schemas to use `x-outbound`
4. delete the local adapter after APOPHIS fetch instrumentation is in place
The long-term target is that applications declare outbound behavior through `outboundContracts` and `x-outbound`; provider-specific fetch wrappers remain application-local.
## Deferred, But Enabled By This Design
1. static analysis of whether a route contract can be satisfied for all permitted dependency responses
2. detection of impossible route postconditions before running property tests
3. contract coverage reports across inbound and outbound boundaries
Those features should be built later on top of the normalized outbound contract registry, not by expanding the runtime surface prematurely.