apophis/apophis-fastify
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public release of Apophis — invariant-driven automated API testing

Browse Source
This commit is contained in:
John Dvorak
2026-03-10 00:00:00 -07:00
parent d278c4b105
commit 3ac1daf7e9
82 changed files with 3902 additions and 1098 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/getting-started.md
+179 -106
View File
@@ -2,6 +2,8 @@
Get from install to your first behavioral bug in 10 minutes.
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
## Prerequisites
- Node.js 20.x or 22.x
@@ -30,6 +32,8 @@ This creates:
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
```javascript
import crypto from 'crypto';
app.post('/users', {
schema: {
'x-category': 'constructor',
@@ -40,27 +44,20 @@ app.post('/users', {
}
}, async (request, reply) => {
const { name } = request.body;
const id = `usr-${Date.now()}`;
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201);
return { id, name };
});
```
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead. APOPHIS does not proactively detect nondeterministic handlers; it warns only when a replay diverges from the original run.
## Step 4: Run Verify
```bash
apophis verify --profile quick --routes "POST /users"
```
APOPHIS will:
1. Discover routes from your Fastify app
2. Filter to `POST /users`
3. Generate test data from the schema
4. Execute the route
5. Check the behavioral contract
6. Print pass/fail, seed, and replay command
## Example Failure
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
@@ -75,7 +72,7 @@ Expected
response_code(GET /users/{response_body(this).id}) == 200
Observed
GET /users/usr-123 returned 404
GET /users/usr-7d865e returned 404
Why this matters
The resource created by POST /users is not retrievable.
@@ -97,114 +94,190 @@ apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
Fix the bug in your handler. Re-run verify. The failure should now pass.
## Behavioral vs Structural Contracts
APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fastify/swagger` already enforce status codes, required fields, and types. Behavioral contracts catch what schemas cannot:
| Structural (avoid) | Behavioral (prefer) |
|---|---|
| `status:200` | `response_body(this) == request_body(this)` |
| `response_body(this).id != null` | `response_code(GET /users/{response_body(this).id}) == 200` |
| `response_body(this).name != null` | `response_body(GET /users/{id}).name == previous(response_body(this).name)` |
**Good behavioral patterns (from the paper):**
- **Constructor precondition**: Resource must not exist before creation
```apostl
response_code(GET /users/{request_body(this).email}) == 404
```
- **Round-trip equality**: POST response matches the request body
```apostl
response_body(this) == request_body(this)
```
- **Cross-route retrievability**: Creating a resource makes it readable via GET
```apostl
response_code(GET /users/{response_body(this).id}) == 200
```
- **State-change verification**: DELETE causes subsequent GET to return 404
```apostl
response_code(GET /users/{request_params(this).id}) == 404
```
- **Previous state preservation**: DELETE returns the last known state
```apostl
response_body(this) == previous(response_body(GET /users/{request_params(this).id}))
```
- **Invariant over collections**: All resources satisfy a cross-resource constraint
```apostl
for t in response_body(GET /tournaments) :-
response_body(GET /tournaments/{t.id}/players).length <= t.capacity
```
**Anti-patterns to avoid:**
- Checking status codes (handled by schema validation)
- Checking field existence (handled by schema validation)
- Checking field types (handled by schema validation)
## Next Steps
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
- Run all routes: `apophis verify --profile quick`
- Run only changed routes in CI: `apophis verify --profile ci --changed`
- Add observe mode for runtime drift detection: see [docs/observe.md](docs/observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [docs/qualify.md](docs/qualify.md)
- Requires a git repository.
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
## Variants
Test the same route with different headers or content types:
```javascript
await fastify.apophis.contract({
variants: [
{ name: 'json', headers: { accept: 'application/json' } },
{ name: 'xml', headers: { accept: 'application/xml' } }
]
})
```
Or declare variants in the route schema:
```javascript
app.get('/users', {
schema: {
'x-variants': [
{ name: 'json', headers: { accept: 'application/json' } }
]
}
})
```
## Plugin Options
When registering the APOPHIS plugin, you can pass these options:
```javascript
await fastify.register(apophis, {
// Swagger config passthrough (if @fastify/swagger is not already registered)
swagger: { openapi: { info: { title: 'API', version: '1.0.0' } } },
// Runtime contract validation hooks: 'off', 'warn', or 'error'
// Only active in non-production environments
runtime: 'warn',
// Automatically clean up tracked resources after tests
cleanup: true,
// Global timeout in milliseconds for all requests
timeout: 5000,
// Tenant isolation scopes
scopes: {
tenant1: { headers: { 'x-tenant-id': '1' } },
tenant2: { headers: { 'x-tenant-id': '2' } },
},
// Auth and protocol extensions
extensions: [jwtAuth, apiKeyAuth],
// Plugin hook-phase contracts
pluginContracts: {
'rate-limit': { appliesTo: 'POST /users', ensures: ['status != 429'] },
},
// Outbound dependency contracts
outboundContracts: {
'payment-api': {
target: 'https://payments.example.com',
method: 'POST',
response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
}
}
})
```
## Schema Annotations
APOPHIS reads these OpenAPI schema extensions:
| Annotation | Location | Description |
|---|---|---|
| `x-category` | Top-level | Route classification: `constructor`, `mutator`, `observer`, `destructor`, `utility` |
| `x-ensures` | Top-level or `response[statusCode]` | Post-condition contracts (APOSTL formulas) |
| `x-requires` | Top-level or `response[statusCode]` | Pre-condition contracts (APOSTL formulas) |
| `x-variants` | Top-level | Request variants for content-type negotiation or feature flags |
| `x-timeout` | Top-level or `response[statusCode]` | Per-route timeout in milliseconds |
| `x-outbound` | Top-level | Outbound dependency contracts for this route |
| `x-streaming` | Top-level | Mark route as streaming (populates `chunks` and `streamDurationMs` in eval context) |
| `x-validate-runtime` | Top-level or `response[statusCode]` | Toggle runtime validation for this route (default: true) |
| `x-extension-config` | Top-level | Per-route config for extensions (e.g., `{ jwt: { verify: false } }`) |
Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations take precedence for that status code.
## Programmatic API
After registration, `fastify.apophis` provides:
```javascript
// Run contract tests for all routes
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })
// Run stateful tests
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
// Run a single scenario
const scenario = await fastify.apophis.scenario({
name: 'oauth-basic',
steps: [...]
})
// Check a single route
const result = await fastify.apophis.check('GET', '/users/:id')
// Get enriched OpenAPI spec with contract metadata
const spec = fastify.apophis.spec()
// Clean up tracked resources
await fastify.apophis.cleanup()
// Test-only utilities (NODE_ENV=test only)
fastify.apophis.test.registerPluginContracts('name', spec)
fastify.apophis.test.registerOutboundContracts({ ... })
fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
fastify.apophis.test.disableOutboundMocks()
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
```
## Config Reference
```javascript
// apophis.config.js
export default {
mode: 'verify',
profile: 'quick',
profiles: {
quick: {
name: 'quick',
mode: 'verify',
preset: 'safe-ci',
routes: ['POST /users']
},
ci: {
name: 'ci',
mode: 'verify',
preset: 'safe-ci',
routes: []
}
},
presets: {
'safe-ci': {
name: 'safe-ci',
depth: 'quick',
timeout: 5000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: true,
allowQualify: false,
allowChaos: false,
allowBlocking: true,
requireSink: false
}
}
};
```
For the full configuration reference, see [CLI Reference](cli.md).
## Monorepo Workspaces
APOPHIS supports workspace-wide operations with the `--workspace` flag.
### Root package.json scripts
```json
{
"scripts": {
"apophis:verify": "apophis verify --workspace --profile quick",
"apophis:doctor": "apophis doctor --workspace",
"apophis:qualify": "apophis qualify --workspace --profile ci"
}
}
```
### Workspace fan-out
Run verify across all packages:
Use `--workspace` to run verify or doctor across all packages:
```bash
apophis verify --workspace --profile quick --format json
```
Output is package-attributed:
```json
{
"exitCode": 0,
"runs": [
{
"package": "api",
"cwd": "/repo/packages/api",
"artifact": { ... }
},
{
"package": "web",
"cwd": "/repo/packages/web",
"artifact": { ... }
}
]
}
```
### Supported commands
- `apophis verify --workspace`
- `apophis doctor --workspace`
## Exit Codes
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Behavioral / qualification failure |
| 2 | Usage, config, or environment safety violation |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |
See [CLI Reference](cli.md) for workspace output format and exit codes.