Initial public release of Apophis — invariant-driven automated API testing
This commit is contained in:
John Dvorak
+4
-1
@@ -1,6 +1,9 @@
|
||||
## Outbound Contract-Driven Mocking Spec
|
||||
|
||||
Status: Proposed
|
||||
Status: Implemented (Phase 1)
|
||||
|
||||
Phase 1 (implemented): Schema parsing (`x-outbound`), mock runtime, imperative API (`enableOutboundMocks`, `getOutboundCalls`), fetch patching.
|
||||
Phase 2 (pending): APOSTL extensions `outbound_calls(this)` and `outbound_last(this)` for contract assertions.
|
||||
Date: 2026-04-27
|
||||
|
||||
This document supersedes Arbiter's local draft at `~/Business/workspace/Arbiter/docs/APOPHIS_OUTBOUND_MOCK_PROPOSAL.md` and its interim adapter at `~/Business/workspace/Arbiter/src/server/server/services/StripeFetchAdapter.js`.
|
||||
@@ -1,6 +1,10 @@
|
||||
# APOPHIS Plugin Contract System Specification
|
||||
|
||||
## Status: Active design; target version to be assigned
|
||||
## Status: Partially implemented
|
||||
|
||||
- Registry, types, and registration API: **implemented**
|
||||
- Runner integration (merging plugin contracts into route execution): **pending**
|
||||
- Built-in contracts for `@fastify/auth`, `@fastify/compress`, `@fastify/cors`, `@fastify/rate-limit`: **registered but not yet applied**
|
||||
|
||||
**Note**: Plugin contracts are complementary to Protocol Extensions (see `docs/protocol-extensions-spec.md`). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE); plugin contracts add hook-phase behavioral contracts for Fastify plugins.
|
||||
|
||||
@@ -374,8 +374,10 @@ import { mkdirSync, rmSync } from 'fs'
|
||||
import { tmpdir } from 'os'
|
||||
import { join } from 'path'
|
||||
|
||||
let testCounter = 0
|
||||
|
||||
export function createTestWorkspace() {
|
||||
const dir = join(tmpdir(), `apophis-test-${Date.now()}`)
|
||||
const dir = join(tmpdir(), `apophis-test-${++testCounter}`)
|
||||
mkdirSync(dir, { recursive: true })
|
||||
|
||||
return {
|
||||
@@ -113,7 +113,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
||||
## Operator Resources
|
||||
|
||||
- [Troubleshooting matrix](docs/troubleshooting.md) — Categorized failure classes with resolution steps
|
||||
- [Adoption certification scorecard](docs/adoption-certification-scorecard.md) — Review template for team rollout
|
||||
- [Adoption certification scorecard](adoption-certification-scorecard.md) — Review template for team rollout
|
||||
|
||||
## CTAs
|
||||
|
||||
|
||||
@@ -1,16 +1,18 @@
|
||||
# APOPHIS Protocol Extensions Specification
|
||||
|
||||
## Status: Active design; shipped baseline: v2.x; remaining targets listed per feature
|
||||
## Status: Active design; shipped baseline: v2.0.0; remaining targets listed per feature
|
||||
|
||||
## 1. Overview
|
||||
|
||||
This specification defines protocol-specific extensions for APOPHIS, driven by the Arbiter team's requirements for testing OAuth 2.1, WIMSE S2S, Transaction Tokens (RFC 8693), SPIFFE/SPIRE, and related security protocols.
|
||||
|
||||
APOPHIS is grounded in [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021). Protocol extensions add domain-specific predicates (JWT, X.509, SPIFFE) to the core invariant framework.
|
||||
|
||||
Arbiter maintains 58 protocol conformance test files covering 138 behaviors across 7 specifications. These extensions bridge the gap between declarative APOSTL contracts and the domain-specific predicates required for security protocol validation.
|
||||
|
||||
### 1.1 Current Shipped vs Not-Shipped Snapshot
|
||||
|
||||
**Shipped in v2.x:**
|
||||
**Shipped in v2.0.0:**
|
||||
|
||||
- `contract({ variants })` for multi-header/media negotiation execution.
|
||||
- `fastify.apophis.scenario(...)` for multi-step capture/rebind flows.
|
||||
@@ -166,12 +168,15 @@ jwtExtension({
|
||||
The JWT extension maintains state across a test run:
|
||||
|
||||
```javascript
|
||||
interface JwtExtensionState {
|
||||
/** Track seen JTIs for replay detection */
|
||||
seenJtis: Set<string>
|
||||
/** Cached decoded JWTs */
|
||||
decodedCache: Map<string, DecodedJwt>
|
||||
}
|
||||
/**
|
||||
* JWT extension state across a test run.
|
||||
* @property {Set<string>} seenJtis - Track seen JTIs for replay detection
|
||||
* @property {Map<string, DecodedJwt>} decodedCache - Cached decoded JWTs
|
||||
*/
|
||||
const jwtExtensionState = {
|
||||
seenJtis: new Set(),
|
||||
decodedCache: new Map()
|
||||
};
|
||||
```
|
||||
|
||||
### 3.5 Example Contracts
|
||||
@@ -234,16 +239,19 @@ await fastify.apophis.time.set('2026-04-25T12:00:00Z');
|
||||
### 4.4 Implementation
|
||||
|
||||
```javascript
|
||||
interface TimeControl {
|
||||
/** Advance simulated time by milliseconds */
|
||||
advance(ms: number): void
|
||||
/** Set simulated time to specific timestamp */
|
||||
set(isoString: string): void
|
||||
/** Get current simulated time */
|
||||
now(): number
|
||||
/** Reset to real time */
|
||||
reset(): void
|
||||
}
|
||||
/**
|
||||
* Time control for deterministic testing.
|
||||
* @property {function(number): void} advance - Advance simulated time by milliseconds
|
||||
* @property {function(string): void} set - Set simulated time to specific ISO timestamp
|
||||
* @property {function(): number} now - Get current simulated time
|
||||
* @property {function(): void} reset - Reset to real time
|
||||
*/
|
||||
const timeControl = {
|
||||
advance(ms) { /* ... */ },
|
||||
set(isoString) { /* ... */ },
|
||||
now() { return Date.now(); },
|
||||
reset() { /* ... */ }
|
||||
};
|
||||
```
|
||||
|
||||
The `now()` predicate returns simulated time when time mocking is enabled, or the host wall clock outside deterministic test mode. Deterministic runs must inject or freeze time.
|
||||
@@ -288,11 +296,17 @@ previous(observer).jwt_claims(this).jti # last observer's JWT ID
|
||||
Extension state tracks tokens across requests:
|
||||
|
||||
```javascript
|
||||
interface StatefulExtensionState {
|
||||
seenTokens: Set<string>
|
||||
consumedTokens: Set<string>
|
||||
categoryHistory: Map<string, EvalContext> // category -> last context
|
||||
}
|
||||
/**
|
||||
* Stateful extension state tracking tokens across requests.
|
||||
* @property {Set<string>} seenTokens - Tokens observed in previous requests
|
||||
* @property {Set<string>} consumedTokens - Tokens that have been consumed
|
||||
* @property {Map<string, EvalContext>} categoryHistory - category -> last context
|
||||
*/
|
||||
const statefulExtensionState = {
|
||||
seenTokens: new Set(),
|
||||
consumedTokens: new Set(),
|
||||
categoryHistory: new Map()
|
||||
};
|
||||
```
|
||||
|
||||
### 5.4 Example Contracts
|
||||
@@ -522,14 +536,14 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
||||
|
||||
## 14. Implementation Plan
|
||||
|
||||
### Phase 1: JWT + Time Control (P0)
|
||||
**Target**: v1.3.0
|
||||
### Phase 1: JWT + Time Control (P0) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/jwt.ts` — JWT extension implementation
|
||||
- `src/extensions/time.ts` — Time control extension
|
||||
- `src/extensions/stateful.ts` — Stateful predicates extension
|
||||
- `src/test/jwt-extension.test.ts` — JWT tests
|
||||
- `src/test/time-extension.test.ts` — Time control tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
- `src/test/cli/protocol-conformance-p2.test.ts` — Protocol conformance tests
|
||||
|
||||
**Tests**:
|
||||
- Decode Base64URL claims without verification
|
||||
@@ -539,27 +553,25 @@ We acknowledge these are too complex or inappropriate for Apophis:
|
||||
- `now()` predicate with mocked time
|
||||
- `apophis.time.advance()` in stateful tests
|
||||
|
||||
### Phase 2: X.509 + SPIFFE (P1)
|
||||
**Target**: v1.3.1
|
||||
### Phase 2: X.509 + SPIFFE (P1) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/x509.ts` — X.509 extension
|
||||
- `src/extensions/spiffe.ts` — SPIFFE extension
|
||||
- `src/test/x509-extension.test.ts` — X.509 tests
|
||||
- `src/test/spiffe-extension.test.ts` — SPIFFE tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
### Phase 3: Token Hash + HTTP Signature (P2)
|
||||
**Target**: v1.3.2
|
||||
### Phase 3: Token Hash + HTTP Signature (P2) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/token-hash.ts` — Token hash extension
|
||||
- `src/extensions/http-signature.ts` — HTTP signature extension
|
||||
- `src/test/token-hash-extension.test.ts` — Token hash tests
|
||||
- `src/test/http-signature-extension.test.ts` — HTTP signature tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
### Phase 4: Request Context (P2)
|
||||
**Target**: v1.3.3
|
||||
### Phase 4: Request Context (P2) — Shipped in v2.0.0
|
||||
**Status**: Complete
|
||||
**Files**:
|
||||
- `src/extensions/request-context.ts` — Request context predicates
|
||||
- `src/test/request-context-extension.test.ts` — Request context tests
|
||||
- `src/test/protocol-extensions.test.ts` — Protocol extension tests
|
||||
|
||||
---
|
||||
|
||||
@@ -135,7 +135,7 @@ fastify.get('/wimse/wit', {
|
||||
})
|
||||
```
|
||||
|
||||
See `docs/protocol-extensions-spec.md` for full JWT extension configuration.
|
||||
See `docs/attic/protocol-extensions-spec.md` for full JWT extension configuration.
|
||||
|
||||
---
|
||||
|
||||
@@ -144,7 +144,7 @@ See `docs/protocol-extensions-spec.md` for full JWT extension configuration.
|
||||
`getToken` runs per request. Handle refresh inline:
|
||||
|
||||
```javascript
|
||||
let cachedToken: string | null = null
|
||||
let cachedToken = null
|
||||
|
||||
const auth = createAuthExtension({
|
||||
name: 'jwt-with-refresh',
|
||||
|
||||
@@ -28,10 +28,10 @@ Each entry is keyed by a hash of the route's path, method, and schema. If the sc
|
||||
|
||||
| Environment | Cache | Reason |
|
||||
|-------------|-------|--------|
|
||||
| `production` | Disabled | No file I/O, no cache hits needed |
|
||||
| `test` | Disabled | Tests should be deterministic, no cache pollution |
|
||||
| `development` | Enabled | Speeds up iterative testing |
|
||||
| default | Enabled | Backward compatible |
|
||||
| `production` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||
| `test` | Enabled by default | Set `APOPHIS_DISABLE_CACHE=1` to opt-out |
|
||||
| `development` | Enabled by default | Speeds up iterative testing |
|
||||
| default | Enabled by default | Backward compatible |
|
||||
|
||||
## Cache Invalidation
|
||||
|
||||
|
||||
+38
-34
@@ -2,19 +2,20 @@
|
||||
|
||||
Inject controlled failures into contract tests to validate resilience guarantees.
|
||||
|
||||
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
|
||||
|
||||
## Usage
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
const result = await fastify.apophis.contract({
|
||||
depth: 'standard',
|
||||
runs: 50,
|
||||
chaos: {
|
||||
probability: 0.1, // 10% of requests get chaos
|
||||
delay: { probability: 1, minMs: 100, maxMs: 500 },
|
||||
error: { probability: 1, statusCode: 503 },
|
||||
dropout: { probability: 1 },
|
||||
corruption: { probability: 1 },
|
||||
delay: { probability: 0.1, minMs: 100, maxMs: 500 },
|
||||
error: { probability: 0.1, statusCode: 503 },
|
||||
dropout: { probability: 0.05 },
|
||||
corruption: { probability: 0.1 },
|
||||
},
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
## Event Types
|
||||
@@ -24,16 +25,18 @@ const result = await fastify.apophis.contract({
|
||||
Adds artificial latency. Tests timeout contracts:
|
||||
|
||||
```apostl
|
||||
timeout_occurred(this) == false
|
||||
response_time(this) < 1000
|
||||
```
|
||||
|
||||
**Note**: Delay events are generated by the chaos arbitrary but the inbound delay handler is currently a no-op. Use this for timeout contract documentation; actual delay injection requires the outbound delay strategy or a custom handler.
|
||||
|
||||
### Error
|
||||
|
||||
Forces HTTP status codes. Tests error-handling contracts:
|
||||
|
||||
```apostl
|
||||
if status:503 then response_body(this).retry_after != null
|
||||
// Behavioral: when the service is unavailable, the client receives a valid retry signal
|
||||
if status:503 then response_headers(this).retry-after > 0
|
||||
```
|
||||
|
||||
### Dropout
|
||||
@@ -41,7 +44,8 @@ if status:503 then response_body(this).retry_after != null
|
||||
Simulates network failure (status 0). Tests fallback contracts:
|
||||
|
||||
```apostl
|
||||
status:200 || status:0
|
||||
// Behavioral: partial failure must still return previously cached data
|
||||
if status:0 then response_body(this).cached_data == previous(response_body(GET /cache/{request_params(this).key}))
|
||||
```
|
||||
|
||||
### Corruption
|
||||
@@ -49,38 +53,39 @@ status:200 || status:0
|
||||
Mutates response bodies. Tests parsing robustness:
|
||||
|
||||
```apostl
|
||||
response_body(this).id != null
|
||||
// Behavioral: corrupted requests maintain traceability for debugging
|
||||
if status:400 then response_body(this).request_id == request_headers(this).x-request-id
|
||||
```
|
||||
|
||||
## Content-Type Aware Corruption
|
||||
## Corruption Strategies
|
||||
|
||||
Built-in strategies for common formats:
|
||||
Built-in strategies are content-type agnostic:
|
||||
|
||||
| Content-Type | Strategy | Effect |
|
||||
|-------------|----------|--------|
|
||||
| `application/json` | Truncate or null field | Removes fields or sets random field to null |
|
||||
| `application/x-ndjson` | Chunk corrupt | Corrupts one NDJSON chunk |
|
||||
| `text/event-stream` | Event corrupt | Adds malformed SSE line |
|
||||
| `multipart/form-data` | Field corrupt | Replaces field with corrupted data |
|
||||
| `text/plain` | Truncate | Cuts string in half |
|
||||
| Strategy | Effect |
|
||||
|----------|--------|
|
||||
| `truncate` | Cuts response body short |
|
||||
| `malformed` | Invalidates structural boundaries (e.g., unclosed JSON, bad headers) |
|
||||
| `field-corrupt` | Replaces a random field value with corrupted data |
|
||||
|
||||
Extension strategies can add content-type-specific behavior if needed.
|
||||
|
||||
## Custom Corruption via Extensions
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
const myExtension = {
|
||||
name: 'custom-corrupt',
|
||||
corruptionStrategies: {
|
||||
'application/vnd.api+json': (data) => ({
|
||||
...data as object,
|
||||
...data,
|
||||
corrupted: true,
|
||||
}),
|
||||
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
||||
},
|
||||
}
|
||||
};
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [myExtension],
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
||||
@@ -90,7 +95,7 @@ Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`
|
||||
Low-level contract chaos APIs require `NODE_ENV=test`. For CLI qualification, environment policy controls whether chaos gates may run.
|
||||
|
||||
```
|
||||
Error: Chaos mode is only available in test environment.
|
||||
Error: chaos is only available in test environment. Set NODE_ENV=test to enable quality features.
|
||||
```
|
||||
|
||||
## Interpreting Results
|
||||
@@ -100,7 +105,7 @@ Failed tests include chaos events in diagnostics:
|
||||
```json
|
||||
{
|
||||
"statusCode": 503,
|
||||
"error": "Contract violation: status:200",
|
||||
"error": "Contract violation: if status:503 then response_headers(this).retry-after > 0",
|
||||
"chaosEvents": [
|
||||
{
|
||||
"type": "error",
|
||||
@@ -118,26 +123,25 @@ Failed tests include chaos events in diagnostics:
|
||||
|
||||
1. **Start small**: `probability: 0.05` (5% of requests)
|
||||
2. **Test one failure mode at a time**: Comment out other chaos types
|
||||
3. **Verify contracts handle chaos**: `if status:503 then response_body(this).error != null`
|
||||
3. **Verify contracts handle chaos**: `if status:503 then response_code(GET /health) == 200`
|
||||
4. **Use seeds for reproducibility**: `seed: 42` makes chaos deterministic
|
||||
|
||||
## Example: Testing Retry Logic
|
||||
|
||||
```typescript
|
||||
```javascript
|
||||
fastify.get('/data', {
|
||||
schema: {
|
||||
'x-ensures': [
|
||||
'if status:503 then response_headers(this).retry-after != null',
|
||||
'if status:503 then response_headers(this).retry-after > 0',
|
||||
'redirect_count(this) <= 3',
|
||||
],
|
||||
},
|
||||
}, handler)
|
||||
}, handler);
|
||||
|
||||
// Test
|
||||
const result = await fastify.apophis.contract({
|
||||
chaos: {
|
||||
probability: 0.2,
|
||||
error: { probability: 1, statusCode: 503 },
|
||||
error: { probability: 0.2, statusCode: 503 },
|
||||
},
|
||||
})
|
||||
});
|
||||
```
|
||||
|
||||
+39
-23
@@ -10,15 +10,16 @@ Every command accepts these flags:
|
||||
|---|---|---|
|
||||
| `--config <path>` | Config file path | Auto-detect |
|
||||
| `--profile <name>` | Profile name from config | First profile |
|
||||
| `--generation-profile <name>` | Generation budget profile (built-in or config alias) | Depth-derived |
|
||||
| `--cwd <path>` | Working directory override | `process.cwd()` |
|
||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | `human` |
|
||||
| `--color <mode>` | Color mode: `auto`, `always`, `never` | `auto` |
|
||||
| `--quiet` | Suppress non-error output | false |
|
||||
| `--verbose` | Enable verbose logging | false |
|
||||
| `--artifact-dir <path>` | Directory for artifact output | `reports/apophis/` |
|
||||
| `--artifact-dir <path>` | Directory for artifact output. Artifacts written on failure or when explicitly configured. | `reports/apophis/` |
|
||||
| `--workspace` | Run supported commands across workspace packages | false |
|
||||
|
||||
Note: `json-summary` and `ndjson-summary` are only supported by `verify` and `qualify` commands.
|
||||
|
||||
## Commands
|
||||
|
||||
### `apophis init`
|
||||
@@ -37,8 +38,8 @@ apophis init --preset safe-ci
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `--preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||
| `--force` | Overwrite existing files |
|
||||
| `-p, --preset <name>` | Preset name: `safe-ci`, `platform-observe`, `llm-safe`, `protocol-lab` |
|
||||
| `-f, --force` | Overwrite existing files |
|
||||
| `--noninteractive` | Skip all prompts, require explicit flags |
|
||||
|
||||
**Examples:**
|
||||
@@ -60,10 +61,10 @@ apophis verify --profile quick --routes "POST /users"
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `--profile <name>` | Profile name from config |
|
||||
| `--generation-profile <name>` | Override generation budget for this run |
|
||||
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
|
||||
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
||||
| `--changed` | Filter to git-modified routes only |
|
||||
| `--workspace` | Run across all workspace packages |
|
||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
||||
|
||||
**Examples:**
|
||||
@@ -118,7 +119,6 @@ apophis qualify --profile oauth-nightly --seed 42
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `--profile <name>` | Profile name from config |
|
||||
| `--generation-profile <name>` | Override generation budget for this run |
|
||||
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
||||
|
||||
**Examples:**
|
||||
@@ -126,18 +126,6 @@ apophis qualify --profile oauth-nightly --seed 42
|
||||
```bash
|
||||
apophis qualify --profile oauth-nightly --seed 42
|
||||
apophis qualify --profile lifecycle-deep
|
||||
apophis qualify --profile oauth-nightly --generation-profile quick
|
||||
```
|
||||
|
||||
You can define aliases in config:
|
||||
|
||||
```js
|
||||
export default {
|
||||
generationProfiles: {
|
||||
pr: 'quick',
|
||||
nightly: { base: 'thorough' },
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### `apophis replay`
|
||||
@@ -171,6 +159,7 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|
||||
|---|---|
|
||||
| `--mode <mode>` | Filter checks to a specific mode |
|
||||
| `--strict` | Treat warnings as failures |
|
||||
| `--workspace` | Run across all workspace packages |
|
||||
|
||||
**Checks:**
|
||||
|
||||
@@ -210,6 +199,31 @@ apophis migrate --dry-run
|
||||
apophis migrate --write
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### CI workflow with machine output
|
||||
```bash
|
||||
apophis verify --profile ci --format json-summary --artifact-dir reports/apophis
|
||||
```
|
||||
|
||||
### Monorepo workspace verification
|
||||
```bash
|
||||
apophis verify --workspace --profile quick
|
||||
apophis doctor --workspace
|
||||
```
|
||||
|
||||
### Replay a failure
|
||||
```bash
|
||||
apophis replay --artifact reports/apophis/failure-*.json
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `--changed` requires a git repository
|
||||
- `migrate` defaults to `--dry-run` (safe by default)
|
||||
- `--workspace` is fully implemented by `verify` and `doctor`. `observe` and `qualify` accept the flag but run in the current package only.
|
||||
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
@@ -226,10 +240,12 @@ apophis migrate --write
|
||||
|---|---|---|---|---|
|
||||
| `verify` | enabled | enabled | optional | optional, usually off |
|
||||
| `observe` | optional | optional | enabled | enabled |
|
||||
| `qualify: scenario` | enabled | enabled | enabled with allowlist | disabled by default |
|
||||
| `qualify: stateful` | enabled | enabled | synthetic-only | disabled by default |
|
||||
| `qualify: chaos` | enabled | enabled | canary-only | disabled by default |
|
||||
| outbound mocks | enabled | enabled | allowlisted only | disabled by default |
|
||||
| `qualify` | enabled | enabled | optional | disabled by default |
|
||||
| `chaos` | enabled | enabled | optional | disabled by default |
|
||||
| runtime throw-on-violation | optional | optional | exceptional | disabled by default |
|
||||
|
||||
Operational rule: Production must never inherit qualify capabilities accidentally from a generic config file.
|
||||
Notes:
|
||||
- `qualify` is gated as a whole. The code does not distinguish scenario, stateful, and chaos sub-modes in environment policy.
|
||||
- `chaos` on protected routes requires `allowChaosOnProtected: true`.
|
||||
- `observe` blocking requires `allowBlocking: true`.
|
||||
- Production must never inherit qualify capabilities accidentally from a generic config file.
|
||||
|
||||
+44
-29
@@ -1,24 +1,29 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from 'apophis-fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
await fastify.register(apophisPlugin, {
|
||||
runtime: 'error', // Validate contracts on every request
|
||||
cleanup: true, // Auto-cleanup resources on exit
|
||||
runtime: 'error',
|
||||
cleanup: true,
|
||||
})
|
||||
|
||||
// In-memory store for demo
|
||||
const users = new Map<string, { id: string; email: string; name: string }>()
|
||||
|
||||
// CREATE — constructor
|
||||
// Behavioral: the created user must be retrievable.
|
||||
// Note: we do not write 'status:201' or 'response_body(this).id != null'.
|
||||
// The schema already validates status codes and required fields.
|
||||
// Contracts should test behavior across operations, not structure.
|
||||
fastify.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
'x-ensures': [
|
||||
'status:201',
|
||||
'response_body(this).id != null',
|
||||
'response_body(this).email == request_body(this).email',
|
||||
// Round-trip: the server returns exactly what we sent (no mutation, no drops)
|
||||
'response_body(this) == request_body(this)',
|
||||
// Cross-route: the created user must be retrievable
|
||||
'response_code(GET /users/{response_body(this).id}) == 200',
|
||||
],
|
||||
body: {
|
||||
type: 'object',
|
||||
@@ -40,7 +45,7 @@ fastify.post('/users', {
|
||||
}
|
||||
}
|
||||
}, async (req, reply) => {
|
||||
const id = `usr-${Date.now()}`
|
||||
const id = `usr-${crypto.createHash('sha256').update(req.body.email).digest('hex').slice(0, 8)}`
|
||||
const user = { id, email: req.body.email, name: req.body.name }
|
||||
users.set(id, user)
|
||||
reply.status(201)
|
||||
@@ -48,19 +53,21 @@ fastify.post('/users', {
|
||||
})
|
||||
|
||||
// READ — observer
|
||||
// Behavioral: the returned user must match the requested id.
|
||||
fastify.get('/users/:id', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-requires': ['users:id'],
|
||||
'x-requires': [
|
||||
// Precondition: the user must exist for this read to be valid
|
||||
'response_code(GET /users/{request_params(this).id}) == 200'
|
||||
],
|
||||
'x-ensures': [
|
||||
'status:200',
|
||||
// The returned id must match the requested id (no mix-up)
|
||||
'response_body(this).id == request_params(this).id',
|
||||
],
|
||||
params: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string' }
|
||||
},
|
||||
properties: { id: { type: 'string' } },
|
||||
required: ['id']
|
||||
},
|
||||
response: {
|
||||
@@ -83,19 +90,21 @@ fastify.get('/users/:id', {
|
||||
})
|
||||
|
||||
// UPDATE — mutator
|
||||
// Behavioral: after update, the change must be visible on read.
|
||||
fastify.put('/users/:id', {
|
||||
schema: {
|
||||
'x-category': 'mutator',
|
||||
'x-requires': ['users:id'],
|
||||
'x-requires': [
|
||||
// The user must exist before updating
|
||||
'response_code(GET /users/{request_params(this).id}) == 200'
|
||||
],
|
||||
'x-ensures': [
|
||||
'status:200',
|
||||
'response_body(this).id == request_params(this).id',
|
||||
// Cross-route: after update, reading the user shows the new data
|
||||
'response_body(GET /users/{request_params(this).id}).email == request_body(this).email',
|
||||
],
|
||||
params: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string' }
|
||||
},
|
||||
properties: { id: { type: 'string' } },
|
||||
required: ['id']
|
||||
},
|
||||
body: {
|
||||
@@ -131,34 +140,40 @@ fastify.put('/users/:id', {
|
||||
})
|
||||
|
||||
// DELETE — destructor
|
||||
// Behavioral: after deletion, the user must no longer exist.
|
||||
fastify.delete('/users/:id', {
|
||||
schema: {
|
||||
'x-category': 'destructor',
|
||||
'x-requires': ['users:id'],
|
||||
'x-ensures': ['status:204'],
|
||||
'x-requires': [
|
||||
// The user must exist before deleting
|
||||
'response_code(GET /users/{request_params(this).id}) == 200'
|
||||
],
|
||||
'x-ensures': [
|
||||
// After deletion, the user is gone
|
||||
'response_code(GET /users/{request_params(this).id}) == 404',
|
||||
// The deleted user data is returned (matches pre-deletion read)
|
||||
'response_body(this) == previous(response_body(GET /users/{request_params(this).id}))',
|
||||
],
|
||||
params: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string' }
|
||||
},
|
||||
properties: { id: { type: 'string' } },
|
||||
required: ['id']
|
||||
}
|
||||
}
|
||||
}, async (req, reply) => {
|
||||
const user = users.get(req.params.id)
|
||||
users.delete(req.params.id)
|
||||
reply.status(204)
|
||||
reply.status(200)
|
||||
return user
|
||||
})
|
||||
|
||||
await fastify.ready()
|
||||
|
||||
// Run contract tests (all non-utility routes, property-based)
|
||||
const result = await fastify.apophis.contract({ depth: 'standard' })
|
||||
const result = await fastify.apophis.contract({ runs: 50 })
|
||||
console.log('Contract tests:', result.summary)
|
||||
|
||||
// Run stateful tests (constructor→mutator→destructor sequences)
|
||||
const stateful = await fastify.apophis.stateful({ depth: 'standard', seed: 42 })
|
||||
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
||||
console.log('Stateful tests:', stateful.summary)
|
||||
|
||||
// Validate a single route
|
||||
const check = await fastify.apophis.check('POST', '/users')
|
||||
console.log('POST /users check:', check.ok ? 'PASS' : 'FAIL')
|
||||
|
||||
@@ -6,21 +6,30 @@ const fastify = Fastify()
|
||||
// APOPHIS auto-registers @fastify/swagger
|
||||
await fastify.register(apophisPlugin, {})
|
||||
|
||||
fastify.get('/health', {
|
||||
// Behavioral contract: what you send is what you get back.
|
||||
// This is not a structural test — the schema already validates shape.
|
||||
// This checks that the server does not mutate or drop fields.
|
||||
fastify.post('/echo', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-ensures': ['status:200'],
|
||||
'x-ensures': [
|
||||
'response_body(this) == request_body(this)'
|
||||
],
|
||||
body: {
|
||||
type: 'object',
|
||||
properties: { message: { type: 'string' } }
|
||||
},
|
||||
response: {
|
||||
200: {
|
||||
type: 'object',
|
||||
properties: { status: { type: 'string' } }
|
||||
properties: { message: { type: 'string' } }
|
||||
}
|
||||
}
|
||||
}
|
||||
}, async () => ({ status: 'ok' }))
|
||||
}, async (req) => req.body)
|
||||
|
||||
await fastify.ready()
|
||||
|
||||
// Run contract tests
|
||||
const result = await fastify.apophis.contract({ depth: 'quick' })
|
||||
const result = await fastify.apophis.contract({ runs: 10 })
|
||||
console.log(result.summary)
|
||||
|
||||
@@ -280,8 +280,8 @@ app.get('/users/:id', {
|
||||
type: 'object',
|
||||
properties: { id: { type: 'string' } },
|
||||
'x-ensures': [
|
||||
// Standard APOSTL + extension predicates
|
||||
'status:200',
|
||||
// Behavioral: returned user must match the requested id
|
||||
'response_body(this).id == request_params(this).id',
|
||||
'graph_check(this).user.can_read_user == true',
|
||||
'partial_graph(this).tenant.accessible == true',
|
||||
],
|
||||
|
||||
@@ -284,8 +284,8 @@ fastify.get('/api/resource', {
|
||||
'x-ensures': [
|
||||
'timeout_occurred(this) == false',
|
||||
'redirect_count(this) == 0',
|
||||
'response_code(this) == 200',
|
||||
'response_body(this).id != null',
|
||||
// Behavioral: created resource must be retrievable
|
||||
'response_code(GET /api/resource/{response_body(this).id}) == 200',
|
||||
]
|
||||
}
|
||||
}, handler)
|
||||
|
||||
+179
-106
@@ -2,6 +2,8 @@
|
||||
|
||||
Get from install to your first behavioral bug in 10 minutes.
|
||||
|
||||
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 20.x or 22.x
|
||||
@@ -30,6 +32,8 @@ This creates:
|
||||
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -40,27 +44,20 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
```
|
||||
|
||||
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead. APOPHIS does not proactively detect nondeterministic handlers; it warns only when a replay diverges from the original run.
|
||||
|
||||
## Step 4: Run Verify
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
APOPHIS will:
|
||||
|
||||
1. Discover routes from your Fastify app
|
||||
2. Filter to `POST /users`
|
||||
3. Generate test data from the schema
|
||||
4. Execute the route
|
||||
5. Check the behavioral contract
|
||||
6. Print pass/fail, seed, and replay command
|
||||
|
||||
## Example Failure
|
||||
|
||||
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
|
||||
@@ -75,7 +72,7 @@ Expected
|
||||
response_code(GET /users/{response_body(this).id}) == 200
|
||||
|
||||
Observed
|
||||
GET /users/usr-123 returned 404
|
||||
GET /users/usr-7d865e returned 404
|
||||
|
||||
Why this matters
|
||||
The resource created by POST /users is not retrievable.
|
||||
@@ -97,114 +94,190 @@ apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
||||
|
||||
Fix the bug in your handler. Re-run verify. The failure should now pass.
|
||||
|
||||
## Behavioral vs Structural Contracts
|
||||
|
||||
APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fastify/swagger` already enforce status codes, required fields, and types. Behavioral contracts catch what schemas cannot:
|
||||
|
||||
| Structural (avoid) | Behavioral (prefer) |
|
||||
|---|---|
|
||||
| `status:200` | `response_body(this) == request_body(this)` |
|
||||
| `response_body(this).id != null` | `response_code(GET /users/{response_body(this).id}) == 200` |
|
||||
| `response_body(this).name != null` | `response_body(GET /users/{id}).name == previous(response_body(this).name)` |
|
||||
|
||||
**Good behavioral patterns (from the paper):**
|
||||
- **Constructor precondition**: Resource must not exist before creation
|
||||
```apostl
|
||||
response_code(GET /users/{request_body(this).email}) == 404
|
||||
```
|
||||
- **Round-trip equality**: POST response matches the request body
|
||||
```apostl
|
||||
response_body(this) == request_body(this)
|
||||
```
|
||||
- **Cross-route retrievability**: Creating a resource makes it readable via GET
|
||||
```apostl
|
||||
response_code(GET /users/{response_body(this).id}) == 200
|
||||
```
|
||||
- **State-change verification**: DELETE causes subsequent GET to return 404
|
||||
```apostl
|
||||
response_code(GET /users/{request_params(this).id}) == 404
|
||||
```
|
||||
- **Previous state preservation**: DELETE returns the last known state
|
||||
```apostl
|
||||
response_body(this) == previous(response_body(GET /users/{request_params(this).id}))
|
||||
```
|
||||
- **Invariant over collections**: All resources satisfy a cross-resource constraint
|
||||
```apostl
|
||||
for t in response_body(GET /tournaments) :-
|
||||
response_body(GET /tournaments/{t.id}/players).length <= t.capacity
|
||||
```
|
||||
|
||||
**Anti-patterns to avoid:**
|
||||
- Checking status codes (handled by schema validation)
|
||||
- Checking field existence (handled by schema validation)
|
||||
- Checking field types (handled by schema validation)
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
||||
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
||||
- Run all routes: `apophis verify --profile quick`
|
||||
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
||||
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md)
|
||||
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md)
|
||||
- Requires a git repository.
|
||||
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
||||
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
||||
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
|
||||
|
||||
## Variants
|
||||
|
||||
Test the same route with different headers or content types:
|
||||
|
||||
```javascript
|
||||
await fastify.apophis.contract({
|
||||
variants: [
|
||||
{ name: 'json', headers: { accept: 'application/json' } },
|
||||
{ name: 'xml', headers: { accept: 'application/xml' } }
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
Or declare variants in the route schema:
|
||||
|
||||
```javascript
|
||||
app.get('/users', {
|
||||
schema: {
|
||||
'x-variants': [
|
||||
{ name: 'json', headers: { accept: 'application/json' } }
|
||||
]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Plugin Options
|
||||
|
||||
When registering the APOPHIS plugin, you can pass these options:
|
||||
|
||||
```javascript
|
||||
await fastify.register(apophis, {
|
||||
// Swagger config passthrough (if @fastify/swagger is not already registered)
|
||||
swagger: { openapi: { info: { title: 'API', version: '1.0.0' } } },
|
||||
|
||||
// Runtime contract validation hooks: 'off', 'warn', or 'error'
|
||||
// Only active in non-production environments
|
||||
runtime: 'warn',
|
||||
|
||||
// Automatically clean up tracked resources after tests
|
||||
cleanup: true,
|
||||
|
||||
// Global timeout in milliseconds for all requests
|
||||
timeout: 5000,
|
||||
|
||||
// Tenant isolation scopes
|
||||
scopes: {
|
||||
tenant1: { headers: { 'x-tenant-id': '1' } },
|
||||
tenant2: { headers: { 'x-tenant-id': '2' } },
|
||||
},
|
||||
|
||||
// Auth and protocol extensions
|
||||
extensions: [jwtAuth, apiKeyAuth],
|
||||
|
||||
// Plugin hook-phase contracts
|
||||
pluginContracts: {
|
||||
'rate-limit': { appliesTo: 'POST /users', ensures: ['status != 429'] },
|
||||
},
|
||||
|
||||
// Outbound dependency contracts
|
||||
outboundContracts: {
|
||||
'payment-api': {
|
||||
target: 'https://payments.example.com',
|
||||
method: 'POST',
|
||||
response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Schema Annotations
|
||||
|
||||
APOPHIS reads these OpenAPI schema extensions:
|
||||
|
||||
| Annotation | Location | Description |
|
||||
|---|---|---|
|
||||
| `x-category` | Top-level | Route classification: `constructor`, `mutator`, `observer`, `destructor`, `utility` |
|
||||
| `x-ensures` | Top-level or `response[statusCode]` | Post-condition contracts (APOSTL formulas) |
|
||||
| `x-requires` | Top-level or `response[statusCode]` | Pre-condition contracts (APOSTL formulas) |
|
||||
| `x-variants` | Top-level | Request variants for content-type negotiation or feature flags |
|
||||
| `x-timeout` | Top-level or `response[statusCode]` | Per-route timeout in milliseconds |
|
||||
| `x-outbound` | Top-level | Outbound dependency contracts for this route |
|
||||
| `x-streaming` | Top-level | Mark route as streaming (populates `chunks` and `streamDurationMs` in eval context) |
|
||||
| `x-validate-runtime` | Top-level or `response[statusCode]` | Toggle runtime validation for this route (default: true) |
|
||||
| `x-extension-config` | Top-level | Per-route config for extensions (e.g., `{ jwt: { verify: false } }`) |
|
||||
|
||||
Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations take precedence for that status code.
|
||||
|
||||
## Programmatic API
|
||||
|
||||
After registration, `fastify.apophis` provides:
|
||||
|
||||
```javascript
|
||||
// Run contract tests for all routes
|
||||
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })
|
||||
|
||||
// Run stateful tests
|
||||
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
||||
|
||||
// Run a single scenario
|
||||
const scenario = await fastify.apophis.scenario({
|
||||
name: 'oauth-basic',
|
||||
steps: [...]
|
||||
})
|
||||
|
||||
// Check a single route
|
||||
const result = await fastify.apophis.check('GET', '/users/:id')
|
||||
|
||||
// Get enriched OpenAPI spec with contract metadata
|
||||
const spec = fastify.apophis.spec()
|
||||
|
||||
// Clean up tracked resources
|
||||
await fastify.apophis.cleanup()
|
||||
|
||||
// Test-only utilities (NODE_ENV=test only)
|
||||
fastify.apophis.test.registerPluginContracts('name', spec)
|
||||
fastify.apophis.test.registerOutboundContracts({ ... })
|
||||
fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
|
||||
fastify.apophis.test.disableOutboundMocks()
|
||||
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
|
||||
```
|
||||
|
||||
## Config Reference
|
||||
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'verify',
|
||||
profile: 'quick',
|
||||
profiles: {
|
||||
quick: {
|
||||
name: 'quick',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
},
|
||||
ci: {
|
||||
name: 'ci',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: []
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
name: 'safe-ci',
|
||||
depth: 'quick',
|
||||
timeout: 5000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
observe: false
|
||||
}
|
||||
},
|
||||
environments: {
|
||||
local: {
|
||||
name: 'local',
|
||||
allowVerify: true,
|
||||
allowObserve: true,
|
||||
allowQualify: false,
|
||||
allowChaos: false,
|
||||
allowBlocking: true,
|
||||
requireSink: false
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
For the full configuration reference, see [CLI Reference](cli.md).
|
||||
|
||||
## Monorepo Workspaces
|
||||
|
||||
APOPHIS supports workspace-wide operations with the `--workspace` flag.
|
||||
|
||||
### Root package.json scripts
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"apophis:verify": "apophis verify --workspace --profile quick",
|
||||
"apophis:doctor": "apophis doctor --workspace",
|
||||
"apophis:qualify": "apophis qualify --workspace --profile ci"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Workspace fan-out
|
||||
|
||||
Run verify across all packages:
|
||||
Use `--workspace` to run verify or doctor across all packages:
|
||||
|
||||
```bash
|
||||
apophis verify --workspace --profile quick --format json
|
||||
```
|
||||
|
||||
Output is package-attributed:
|
||||
|
||||
```json
|
||||
{
|
||||
"exitCode": 0,
|
||||
"runs": [
|
||||
{
|
||||
"package": "api",
|
||||
"cwd": "/repo/packages/api",
|
||||
"artifact": { ... }
|
||||
},
|
||||
{
|
||||
"package": "web",
|
||||
"cwd": "/repo/packages/web",
|
||||
"artifact": { ... }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Supported commands
|
||||
|
||||
- `apophis verify --workspace`
|
||||
- `apophis doctor --workspace`
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | Success |
|
||||
| 1 | Behavioral / qualification failure |
|
||||
| 2 | Usage, config, or environment safety violation |
|
||||
| 3 | Internal APOPHIS error |
|
||||
| 130 | Interrupted (SIGINT) |
|
||||
See [CLI Reference](cli.md) for workspace output format and exit codes.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
|
||||
|
||||
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
|
||||
|
||||
## Why APOPHIS Is Good for LLM-Generated Services
|
||||
|
||||
Coding agents benefit from:
|
||||
@@ -18,10 +20,10 @@ Use `apophis init` with a preset:
|
||||
|
||||
| Preset | Use Case |
|
||||
|---|---|
|
||||
| `safe-ci` | General CI-safe setup |
|
||||
| `llm-safe` | Ultra-minimal for LLM-generated code |
|
||||
| `platform-observe` | Observe-mode policy and runtime drift reporting |
|
||||
| `protocol-lab` | Multi-step flows and stateful testing |
|
||||
| `safe-ci` | Minimal CI-safe preset (default) |
|
||||
| `llm-safe` | Minimal preset for LLM-generated codebases |
|
||||
| `platform-observe` | Production-ready with observe mode |
|
||||
| `protocol-lab` | Multi-step flow and stateful testing |
|
||||
|
||||
```bash
|
||||
apophis init --preset llm-safe
|
||||
@@ -84,7 +86,6 @@ export default {
|
||||
presets: {
|
||||
'llm-safe': {
|
||||
name: 'llm-safe',
|
||||
depth: 'quick',
|
||||
timeout: 3000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
@@ -108,6 +109,8 @@ export default {
|
||||
### Route Template with Behavioral Contract
|
||||
|
||||
```javascript
|
||||
import crypto from 'crypto';
|
||||
|
||||
app.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
@@ -134,7 +137,7 @@ app.post('/users', {
|
||||
}
|
||||
}, async (request, reply) => {
|
||||
const { name } = request.body;
|
||||
const id = `usr-${Date.now()}`;
|
||||
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
||||
reply.status(201);
|
||||
return { id, name };
|
||||
});
|
||||
|
||||
+50
-4
@@ -2,6 +2,8 @@
|
||||
|
||||
Runtime visibility and drift detection without blocking by default.
|
||||
|
||||
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
|
||||
|
||||
## What Observe Does
|
||||
|
||||
`apophis observe` validates your runtime observe configuration:
|
||||
@@ -65,14 +67,38 @@ profiles: {
|
||||
}
|
||||
```
|
||||
|
||||
The `platform-observe` preset enables sampling at the preset level. Fine-tune per route with `x-observe-sampling` in your route schema.
|
||||
The `platform-observe` preset enables sampling. Configure the rate explicitly:
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'staging-observe': {
|
||||
mode: 'observe',
|
||||
preset: 'platform-observe',
|
||||
routes: [],
|
||||
sampling: 1.0 // 100% of requests observed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Staging vs Production
|
||||
|
||||
| Environment | Blocking | Sampling | Sink Required |
|
||||
|---|---|---|---|
|
||||
| Staging | No (default) | 10% | Yes |
|
||||
| Production | No (default) | 1% | Yes |
|
||||
| Staging | No (default) | 100% | Yes |
|
||||
| Production | No (default) | 100% | Yes |
|
||||
|
||||
Default is `1.0` (100%). Configure lower rates for production explicitly:
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'prod-observe': {
|
||||
mode: 'observe',
|
||||
preset: 'platform-observe',
|
||||
routes: [],
|
||||
sampling: 0.1 // 10% of requests observed
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## `--check-config` Flag
|
||||
|
||||
@@ -109,7 +135,6 @@ export default {
|
||||
presets: {
|
||||
'platform-observe': {
|
||||
name: 'platform-observe',
|
||||
depth: 'standard',
|
||||
timeout: 10000,
|
||||
parallel: true,
|
||||
chaos: false,
|
||||
@@ -138,3 +163,24 @@ export default {
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Sink Endpoint Configuration
|
||||
|
||||
Configure the reporting sink endpoint in your observe config:
|
||||
|
||||
```javascript
|
||||
observe: {
|
||||
sink: {
|
||||
endpoint: 'http://collector.internal:4318'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Monorepo Validation
|
||||
|
||||
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
|
||||
|
||||
## Mode Mismatch
|
||||
|
||||
Profiles configured for `verify` mode will be rejected by `apophis observe`. Only profiles with `mode: 'observe'` are valid.
|
||||
```
|
||||
|
||||
+5
-5
@@ -23,8 +23,8 @@ BENCH_RUNS=12 BENCH_WARMUP=3 npm run benchmark:cli
|
||||
# Increase inner-loop work for micro-benchmarks
|
||||
BENCH_INNER_ITERS=5000 npm run benchmark:hot
|
||||
|
||||
# Benchmark generation profile matrix
|
||||
BENCH_GENERATION_PROFILES=quick,standard,thorough npm run benchmark:all
|
||||
# Benchmark with varying test counts
|
||||
BENCH_RUNS=10,50,200 npm run benchmark:all
|
||||
```
|
||||
|
||||
## Capture CPU Profile for Qualify
|
||||
@@ -41,10 +41,10 @@ This writes Chrome-compatible CPU profiles to `.profiles/qualify.cpuprofile` and
|
||||
- CLI benchmark uses spawned `node dist/cli/index.js` commands so startup costs are included.
|
||||
- Hot path benchmark runs in-process for lower-noise function-level comparisons.
|
||||
- Use fixed `--seed` for qualify benchmarks to keep runs deterministic.
|
||||
- Generation now adapts to depth: `quick` favors bounded payload generation speed, `thorough` keeps broader generation.
|
||||
- Schema generation uses fixed defaults (string≤128, array≤10) regardless of run count.
|
||||
|
||||
You can override generation per run:
|
||||
You can override runs per preset:
|
||||
|
||||
```bash
|
||||
apophis qualify --profile oauth-nightly --generation-profile quick --seed 42
|
||||
apophis qualify --profile oauth-nightly --seed 42
|
||||
```
|
||||
|
||||
+74
-34
@@ -2,6 +2,8 @@
|
||||
|
||||
Run scenario, stateful, and chaos checks against non-production Fastify services.
|
||||
|
||||
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
|
||||
|
||||
## What Qualify Does
|
||||
|
||||
`apophis qualify` runs deeper testing than verify:
|
||||
@@ -49,6 +51,40 @@ profiles: {
|
||||
}
|
||||
```
|
||||
|
||||
## Scenario Definitions
|
||||
|
||||
Scenarios are multi-step flows with capture and rebind:
|
||||
|
||||
```javascript
|
||||
await fastify.apophis.scenario({
|
||||
name: 'oauth-basic',
|
||||
steps: [
|
||||
{
|
||||
name: 'authorize',
|
||||
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
|
||||
// Behavioral: state parameter round-trips for CSRF protection
|
||||
expect: ['response_payload(this).state == request_query(this).state'],
|
||||
capture: { code: 'response_payload(this).code' }
|
||||
},
|
||||
{
|
||||
name: 'token',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: '/oauth/token',
|
||||
form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
|
||||
},
|
||||
// Behavioral: issued token preserves the requested scope
|
||||
expect: ['response_payload(this).scope == request_body(this).scope']
|
||||
}
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
Scenario behavior:
|
||||
1. Cookie jar persists `Set-Cookie` values across steps.
|
||||
2. Step-level `headers.cookie` overrides jar values for that step.
|
||||
3. `form` sends `application/x-www-form-urlencoded` payloads.
|
||||
|
||||
## Stateful Testing
|
||||
|
||||
Stateful tests generate sequences of operations and track resources:
|
||||
@@ -58,7 +94,18 @@ Stateful tests generate sequences of operations and track resources:
|
||||
3. **Observer**: Read resources (GET)
|
||||
4. **Destructor**: Remove resources (DELETE)
|
||||
|
||||
APOPHIS automatically tracks created resources and cleans them up after testing.
|
||||
APOPHIS tracks created resources and runs cleanup after test completion.
|
||||
|
||||
Run stateful tests via the API:
|
||||
|
||||
```javascript
|
||||
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
||||
console.log('Stateful tests:', stateful.summary)
|
||||
```
|
||||
|
||||
## Route Transparency
|
||||
|
||||
Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.
|
||||
|
||||
## Chaos and Adversity
|
||||
|
||||
@@ -67,7 +114,9 @@ Chaos testing injects controlled failures:
|
||||
- **Delay**: Slow responses
|
||||
- **Error**: Return error status codes
|
||||
- **Dropout**: Connection failures
|
||||
- **Corruption**: Malformed response bodies
|
||||
- **Truncate**: Truncated response bodies
|
||||
- **Malformed**: Invalid JSON or content-type
|
||||
- **Field-corrupt**: Random field mutation in response objects
|
||||
|
||||
Configure chaos in your preset:
|
||||
|
||||
@@ -84,36 +133,6 @@ presets: {
|
||||
}
|
||||
```
|
||||
|
||||
## Profile Examples
|
||||
|
||||
### oauth-nightly
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'oauth-nightly': {
|
||||
name: 'oauth-nightly',
|
||||
mode: 'qualify',
|
||||
preset: 'protocol-lab',
|
||||
routes: [],
|
||||
seed: 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### lifecycle-deep
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
'lifecycle-deep': {
|
||||
name: 'lifecycle-deep',
|
||||
mode: 'qualify',
|
||||
preset: 'protocol-lab',
|
||||
routes: [],
|
||||
seed: 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Non-Prod Boundaries
|
||||
|
||||
Qualify mode is gated away from production by default:
|
||||
@@ -122,7 +141,7 @@ Qualify mode is gated away from production by default:
|
||||
|---|---|---|---|
|
||||
| local | enabled | enabled | enabled |
|
||||
| test/CI | enabled | enabled | enabled |
|
||||
| staging | enabled with allowlist | synthetic-only | canary-only |
|
||||
| staging | enabled with allowlist | enabled | blocked on protected routes |
|
||||
| production | disabled by default | disabled by default | disabled by default |
|
||||
|
||||
## Machine Output for CI
|
||||
@@ -186,7 +205,7 @@ export default {
|
||||
presets: {
|
||||
'protocol-lab': {
|
||||
name: 'protocol-lab',
|
||||
depth: 'deep',
|
||||
runs: 200,
|
||||
timeout: 15000,
|
||||
parallel: false,
|
||||
chaos: true,
|
||||
@@ -224,3 +243,24 @@ export default {
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Gate Execution Counts
|
||||
|
||||
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
|
||||
|
||||
## Zero-Execution Guardrail
|
||||
|
||||
Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.
|
||||
|
||||
## Test Budget
|
||||
|
||||
The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:
|
||||
|
||||
```javascript
|
||||
presets: {
|
||||
'protocol-lab': {
|
||||
runs: 200,
|
||||
timeout: 15000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
+226
@@ -0,0 +1,226 @@
|
||||
# Quality Engines
|
||||
|
||||
APOPHIS includes three quality engines for advanced testing: chaos injection, flake detection, and mutation testing. All require `NODE_ENV=test`.
|
||||
|
||||
## Chaos Injection
|
||||
|
||||
Inject controlled failures into contract tests to validate resilience guarantees. Chaos events are generated by fast-check alongside test data, making them shrinkable — when a test fails, fast-check finds the minimal chaos event that causes the failure.
|
||||
|
||||
### Usage
|
||||
|
||||
```javascript
|
||||
const result = await fastify.apophis.contract({
|
||||
runs: 50,
|
||||
chaos: {
|
||||
delay: { probability: 0.1, minMs: 100, maxMs: 500 },
|
||||
error: { probability: 0.1, statusCode: 503 },
|
||||
dropout: { probability: 0.05 },
|
||||
corruption: { probability: 0.1 },
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### Event Types
|
||||
|
||||
| Type | Effect | Tests |
|
||||
|------|--------|-------|
|
||||
| `delay` | Artificial latency | `response_time(this) < 1000` |
|
||||
| `error` | Forces HTTP status code | Error-handling contracts |
|
||||
| `dropout` | Network failure (status 0 or 504) | Fallback contracts |
|
||||
| `corruption` | Mutates response bodies | Parsing robustness |
|
||||
|
||||
### Corruption Strategies
|
||||
|
||||
| Strategy | Effect |
|
||||
|----------|--------|
|
||||
| `truncate` | Cuts response body in half |
|
||||
| `malformed` | Returns invalid JSON (`{"broken":`) |
|
||||
| `field-corrupt` | Sets a random field to `null` |
|
||||
|
||||
### Programmatic API
|
||||
|
||||
```javascript
|
||||
import {
|
||||
applyChaosToExecution,
|
||||
createChaosEventArbitrary,
|
||||
formatChaosEvents,
|
||||
} from 'apophis-fastify'
|
||||
|
||||
// Apply pre-generated chaos events to a context
|
||||
const result = applyChaosToExecution(ctx, events)
|
||||
|
||||
// Generate deterministic chaos events
|
||||
const arb = createChaosEventArbitrary(config, contractNames)
|
||||
const events = fc.sample(arb, { numRuns: 1, seed: 42 })[0]
|
||||
|
||||
// Format for diagnostics
|
||||
console.log(formatChaosEvents(events))
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. Start small: `probability: 0.05` (5% of requests)
|
||||
2. Test one failure mode at a time
|
||||
3. Verify contracts handle chaos: `if status:503 then response_code(GET /health) == 200`
|
||||
4. Use seeds for reproducibility: `seed: 42`
|
||||
|
||||
## Flake Detection
|
||||
|
||||
Automatically rerun failing tests with varied seeds to detect non-deterministic contracts. A "flake" is a test that fails on one run but passes on another with the same or different seed.
|
||||
|
||||
### Usage
|
||||
|
||||
```javascript
|
||||
import { FlakeDetector } from 'apophis-fastify'
|
||||
|
||||
const detector = new FlakeDetector({
|
||||
sameSeedReruns: 1, // Rerun with same seed
|
||||
seedVariations: 3, // Try 3 additional seeds
|
||||
})
|
||||
|
||||
const report = await detector.detectFlake(
|
||||
originalFailingResult,
|
||||
async (seed) => {
|
||||
const suite = await fastify.apophis.contract({ seed })
|
||||
return { passed: suite.summary.failed === 0 }
|
||||
},
|
||||
originalSeed
|
||||
)
|
||||
|
||||
if (report.isFlaky) {
|
||||
console.log(`Flaky with ${report.confidence} confidence`)
|
||||
console.log('Reruns:', report.reruns)
|
||||
}
|
||||
```
|
||||
|
||||
### Report Structure
|
||||
|
||||
```javascript
|
||||
{
|
||||
isFlaky: true,
|
||||
confidence: 'high', // 'high' | 'medium' | 'low'
|
||||
reruns: [
|
||||
{ seed: 42, passed: false },
|
||||
{ seed: 43, passed: true },
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Confidence Scoring
|
||||
|
||||
| Pass Rate | Confidence |
|
||||
|-----------|------------|
|
||||
| 0% pass | `high` (deterministic failure) |
|
||||
| < 50% pass | `medium` |
|
||||
| >= 50% pass | `low` (likely flaky) |
|
||||
|
||||
## Mutation Testing
|
||||
|
||||
Measure contract strength by injecting synthetic bugs. A "mutation" is a small change to a contract (e.g., flip `==` to `!=`). If the test suite catches the mutation (fails), the mutation is "killed". If it passes, the mutation "survives" — indicating weak coverage.
|
||||
|
||||
### Usage
|
||||
|
||||
```javascript
|
||||
import { runMutationTesting } from 'apophis-fastify/quality/mutation'
|
||||
|
||||
const report = await runMutationTesting(fastify, {
|
||||
runs: 10,
|
||||
seed: 42,
|
||||
maxMutationsPerContract: 5,
|
||||
routes: ['/items'], // Optional: only test these routes
|
||||
})
|
||||
|
||||
console.log(`Mutation score: ${report.score}%`)
|
||||
console.log(`Killed: ${report.killed}, Survived: ${report.survived}`)
|
||||
console.log('Weak contracts:', report.weakContracts)
|
||||
```
|
||||
|
||||
### Mutation Operators
|
||||
|
||||
| Type | Example |
|
||||
|------|---------|
|
||||
| `flip-operator` | `== 201` → `!= 201` |
|
||||
| `change-number` | `== 200` → `== 201` |
|
||||
| `remove-clause` | `A && B` → `A` |
|
||||
| `negate-boolean` | `== true` → `== false` |
|
||||
| `swap-variable` | `response_body` → `request_body` |
|
||||
| `remove-ensures` | Remove one ensures clause entirely |
|
||||
|
||||
### Report Structure
|
||||
|
||||
```javascript
|
||||
{
|
||||
score: 85, // 0-100
|
||||
killed: 17,
|
||||
survived: 3,
|
||||
durationMs: 4500,
|
||||
weakContracts: ['POST /items'], // Routes where no mutations were killed
|
||||
mutations: [
|
||||
{
|
||||
mutation: {
|
||||
id: 'm0',
|
||||
route: 'POST /items',
|
||||
original: 'response_code(this) == 201',
|
||||
mutated: 'response_code(this) != 201',
|
||||
type: 'flip-operator',
|
||||
},
|
||||
killed: true,
|
||||
durationMs: 120,
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Single Mutation Test
|
||||
|
||||
Test a specific mutation without running the full suite:
|
||||
|
||||
```javascript
|
||||
import { testMutation } from 'apophis-fastify/quality/mutation'
|
||||
|
||||
const killed = await testMutation(fastify, contract, mutation, {
|
||||
runs: 10,
|
||||
seed: 42,
|
||||
})
|
||||
```
|
||||
|
||||
## Environment Guard
|
||||
|
||||
All quality engines require `NODE_ENV=test`:
|
||||
|
||||
```
|
||||
Error: chaos is only available in test environment.
|
||||
Set NODE_ENV=test to enable quality features.
|
||||
```
|
||||
|
||||
This prevents accidental execution in production or development.
|
||||
|
||||
## Integration Example
|
||||
|
||||
Run all three engines in a CI pipeline:
|
||||
|
||||
```javascript
|
||||
// 1. Standard contract tests
|
||||
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })
|
||||
|
||||
// 2. Chaos tests
|
||||
const chaosSuite = await fastify.apophis.contract({
|
||||
runs: 50,
|
||||
seed: 42,
|
||||
chaos: { error: { probability: 0.1, statusCode: 503 } },
|
||||
})
|
||||
|
||||
// 3. Flake detection on failures
|
||||
for (const test of suite.tests.filter(t => !t.ok)) {
|
||||
const report = await detector.detectFlake(test, rerunFn, 42)
|
||||
if (report.isFlaky) {
|
||||
console.warn(`Flaky test detected: ${test.name}`)
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Mutation testing
|
||||
const mutationReport = await runMutationTesting(fastify, { runs: 10 })
|
||||
if (mutationReport.score < 80) {
|
||||
console.warn(`Low mutation score: ${mutationReport.score}%`)
|
||||
}
|
||||
```
|
||||
@@ -31,7 +31,7 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
||||
|
||||
**Symptoms**
|
||||
- `Unexpected token` in formula output
|
||||
- `Unterminated string` in x-ensures clause
|
||||
- `Unterminated string literal` in x-ensures clause
|
||||
- `Missing this` in operation call
|
||||
|
||||
**Resolution**
|
||||
@@ -88,12 +88,12 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
||||
**Symptoms**
|
||||
- `Plugin decorator already added`
|
||||
- `Duplicate route registration`
|
||||
- `No behavioral contracts found`
|
||||
- `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`
|
||||
|
||||
**Resolution**
|
||||
1. Ensure the APOPHIS plugin is registered exactly once in the Fastify app.
|
||||
2. Check for multiple imports or plugin registrations in test vs production entry points.
|
||||
3. If `No behavioral contracts found`, add `x-ensures` or `x-requires` to route schemas.
|
||||
3. If `No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.`, add `x-ensures` or `x-requires` to route schemas.
|
||||
4. Run `apophis doctor` to verify route discovery matches expectations.
|
||||
|
||||
**Prevention**
|
||||
@@ -150,13 +150,13 @@ Every failure produces an artifact JSON file. Use it for deep triage:
|
||||
|
||||
```bash
|
||||
# Inspect the artifact
|
||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures[0]'
|
||||
cat reports/apophis/failure-<timestamp>.json | jq '.failures[0]'
|
||||
|
||||
# Replay the exact failure
|
||||
apophis replay --artifact reports/apophis/verify-<timestamp>.json
|
||||
apophis replay --artifact reports/apophis/failure-<timestamp>.json
|
||||
|
||||
# Filter by error category
|
||||
cat reports/apophis/verify-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||
cat reports/apophis/failure-<timestamp>.json | jq '.failures | map(select(.category == "runtime"))'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+42
-36
@@ -2,15 +2,7 @@
|
||||
|
||||
Deterministic contract verification for CI and local development.
|
||||
|
||||
## What Verify Does
|
||||
|
||||
`apophis verify` runs behavioral contracts against your Fastify routes:
|
||||
|
||||
1. Discovers routes from your Fastify app
|
||||
2. Filters routes by profile config and CLI flags
|
||||
3. Generates test data from JSON Schema
|
||||
4. Executes routes and checks `x-ensures` contracts
|
||||
5. Reports pass/fail with deterministic seed and replay command
|
||||
APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
|
||||
|
||||
## When to Use It
|
||||
|
||||
@@ -79,6 +71,8 @@ apophis verify --routes "POST /users/*"
|
||||
apophis verify --profile quick
|
||||
```
|
||||
|
||||
`*` and `?` wildcards are supported in `--routes`.
|
||||
|
||||
## `--changed` Flag
|
||||
|
||||
Run only routes modified in the current git branch:
|
||||
@@ -87,7 +81,7 @@ Run only routes modified in the current git branch:
|
||||
apophis verify --profile ci --changed
|
||||
```
|
||||
|
||||
If no routes changed, exits 0 with a message.
|
||||
If no routes changed, exits 2 with a message.
|
||||
|
||||
## Failure Output Format
|
||||
|
||||
@@ -126,6 +120,8 @@ Next
|
||||
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
||||
```
|
||||
|
||||
Nondeterminism warnings appear in output when the same seed produces different results across runs. This indicates stateful behavior in your application that contracts cannot control.
|
||||
|
||||
## Machine Output for CI
|
||||
|
||||
Use concise formats to reduce log volume in large verify runs:
|
||||
@@ -137,6 +133,7 @@ Use concise formats to reduce log volume in large verify runs:
|
||||
|
||||
```bash
|
||||
# Extract only failed routes from full ndjson
|
||||
# Note: route.failed events are only emitted for failures, not passed routes
|
||||
apophis verify --profile quick --format ndjson | jq 'select(.type == "route.failed")'
|
||||
|
||||
# Write artifact to disk and parse the file instead of stdout
|
||||
@@ -149,7 +146,7 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
||||
|---|---|
|
||||
| 0 | All contracts passed |
|
||||
| 1 | One or more behavioral contracts failed |
|
||||
| 2 | Config error or no routes matched |
|
||||
| 2 | Config error, no routes matched, no contracts found, or not a git repo |
|
||||
| 3 | Internal APOPHIS error |
|
||||
| 130 | Interrupted (SIGINT) |
|
||||
|
||||
@@ -158,42 +155,51 @@ apophis verify --profile quick --format json --artifact-dir reports/apophis
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'verify',
|
||||
profile: 'quick',
|
||||
profiles: {
|
||||
quick: {
|
||||
name: 'quick',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
},
|
||||
ci: {
|
||||
name: 'ci',
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: []
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
name: 'safe-ci',
|
||||
depth: 'quick',
|
||||
timeout: 5000,
|
||||
parallel: false,
|
||||
chaos: false,
|
||||
observe: false
|
||||
}
|
||||
},
|
||||
environments: {
|
||||
local: {
|
||||
name: 'local',
|
||||
allowVerify: true,
|
||||
allowObserve: true,
|
||||
allowQualify: false,
|
||||
allowChaos: false,
|
||||
allowBlocking: true,
|
||||
requireSink: false
|
||||
runs: 10,
|
||||
timeout: 5000
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
For the full config schema, see [CLI Reference](cli.md).
|
||||
|
||||
## Workspace Support
|
||||
|
||||
Run verify across all packages in a monorepo workspace:
|
||||
|
||||
```bash
|
||||
apophis verify --workspace --profile quick --format json
|
||||
```
|
||||
|
||||
Output includes per-package pass/fail summaries. Fails if any package fails.
|
||||
|
||||
## Test Budget
|
||||
|
||||
The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
quick: {
|
||||
mode: 'verify',
|
||||
preset: 'safe-ci',
|
||||
routes: ['POST /users']
|
||||
}
|
||||
},
|
||||
presets: {
|
||||
'safe-ci': {
|
||||
runs: 10,
|
||||
timeout: 5000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user