284 lines
8.8 KiB
Markdown
284 lines
8.8 KiB
Markdown
# Getting Started with APOPHIS
|
|
|
|
Get from install to your first behavioral bug in 10 minutes.
|
|
|
|
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js 20.x or 22.x
|
|
- A Fastify app with `@fastify/swagger` registered
|
|
|
|
## Step 1: Install
|
|
|
|
```bash
|
|
npm install apophis-fastify fastify @fastify/swagger
|
|
```
|
|
|
|
## Step 2: Scaffold
|
|
|
|
```bash
|
|
apophis init --preset safe-ci
|
|
```
|
|
|
|
This creates:
|
|
|
|
- `apophis.config.js` — config with a `quick` profile
|
|
- `APOPHIS.md` — preset-specific guidance
|
|
- Package script: `npm run apophis:verify`
|
|
|
|
## Step 3: Add One Behavioral Contract
|
|
|
|
Pick one important route. Add an `x-ensures` clause that checks behavior across operations:
|
|
|
|
```javascript
|
|
import crypto from 'crypto';
|
|
|
|
app.post('/users', {
|
|
schema: {
|
|
'x-category': 'constructor',
|
|
'x-ensures': [
|
|
// BEHAVIORAL: Creating a user must make it retrievable
|
|
'response_code(GET /users/{response_body(this).id}) == 200'
|
|
]
|
|
}
|
|
}, async (request, reply) => {
|
|
const { name } = request.body;
|
|
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
|
|
reply.status(201);
|
|
return { id, name };
|
|
});
|
|
```
|
|
|
|
> **Warning:** Using `Date.now()` or `Math.random()` in handlers breaks determinism and replay. Use a stable function of the input instead. APOPHIS does not proactively detect nondeterministic handlers; it warns only when a replay diverges from the original run.
|
|
|
|
## Step 4: Run Verify
|
|
|
|
```bash
|
|
apophis verify --profile quick --routes "POST /users"
|
|
```
|
|
|
|
## Example Failure
|
|
|
|
If your `GET /users/:id` handler has a bug (always returns 404), APOPHIS catches it:
|
|
|
|
```text
|
|
Contract violation
|
|
POST /users
|
|
Profile: quick
|
|
Seed: 42
|
|
|
|
Expected
|
|
response_code(GET /users/{response_body(this).id}) == 200
|
|
|
|
Observed
|
|
GET /users/usr-7d865e returned 404
|
|
|
|
Why this matters
|
|
The resource created by POST /users is not retrievable.
|
|
|
|
Replay
|
|
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
|
|
|
Next
|
|
Check the create/read consistency for POST /users and GET /users/{id}.
|
|
```
|
|
|
|
## Step 5: Replay and Fix
|
|
|
|
Copy the replay command and run it:
|
|
|
|
```bash
|
|
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
|
```
|
|
|
|
Fix the bug in your handler. Re-run verify. The failure should now pass.
|
|
|
|
## Behavioral vs Structural Contracts
|
|
|
|
APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fastify/swagger` already enforce status codes, required fields, and types. Behavioral contracts catch what schemas cannot:
|
|
|
|
| Structural (avoid) | Behavioral (prefer) |
|
|
|---|---|
|
|
| `status:200` | `response_body(this) == request_body(this)` |
|
|
| `response_body(this).id != null` | `response_code(GET /users/{response_body(this).id}) == 200` |
|
|
| `response_body(this).name != null` | `response_body(GET /users/{id}).name == previous(response_body(this).name)` |
|
|
|
|
**Good behavioral patterns (from the paper):**
|
|
- **Constructor precondition**: Resource must not exist before creation
|
|
```apostl
|
|
response_code(GET /users/{request_body(this).email}) == 404
|
|
```
|
|
- **Round-trip equality**: POST response matches the request body
|
|
```apostl
|
|
response_body(this) == request_body(this)
|
|
```
|
|
- **Cross-route retrievability**: Creating a resource makes it readable via GET
|
|
```apostl
|
|
response_code(GET /users/{response_body(this).id}) == 200
|
|
```
|
|
- **State-change verification**: DELETE causes subsequent GET to return 404
|
|
```apostl
|
|
response_code(GET /users/{request_params(this).id}) == 404
|
|
```
|
|
- **Previous state preservation**: DELETE returns the last known state
|
|
```apostl
|
|
response_body(this) == previous(response_body(GET /users/{request_params(this).id}))
|
|
```
|
|
- **Invariant over collections**: All resources satisfy a cross-resource constraint
|
|
```apostl
|
|
for t in response_body(GET /tournaments) :-
|
|
response_body(GET /tournaments/{t.id}/players).length <= t.capacity
|
|
```
|
|
|
|
**Anti-patterns to avoid:**
|
|
- Checking status codes (handled by schema validation)
|
|
- Checking field existence (handled by schema validation)
|
|
- Checking field types (handled by schema validation)
|
|
|
|
## Next Steps
|
|
|
|
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
|
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
|
- Run all routes: `apophis verify --profile quick`
|
|
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
|
- Requires a git repository.
|
|
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
|
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
|
- Add qualify mode for scenario, stateful, and chaos checks: see [qualify.md](qualify.md)
|
|
|
|
## Variants
|
|
|
|
Test the same route with different headers or content types:
|
|
|
|
```javascript
|
|
await fastify.apophis.contract({
|
|
variants: [
|
|
{ name: 'json', headers: { accept: 'application/json' } },
|
|
{ name: 'xml', headers: { accept: 'application/xml' } }
|
|
]
|
|
})
|
|
```
|
|
|
|
Or declare variants in the route schema:
|
|
|
|
```javascript
|
|
app.get('/users', {
|
|
schema: {
|
|
'x-variants': [
|
|
{ name: 'json', headers: { accept: 'application/json' } }
|
|
]
|
|
}
|
|
})
|
|
```
|
|
|
|
## Plugin Options
|
|
|
|
When registering the APOPHIS plugin, you can pass these options:
|
|
|
|
```javascript
|
|
await fastify.register(apophis, {
|
|
// Swagger config passthrough (if @fastify/swagger is not already registered)
|
|
swagger: { openapi: { info: { title: 'API', version: '1.0.0' } } },
|
|
|
|
// Runtime contract validation hooks: 'off', 'warn', or 'error'
|
|
// Only active in non-production environments
|
|
runtime: 'warn',
|
|
|
|
// Automatically clean up tracked resources after tests
|
|
cleanup: true,
|
|
|
|
// Global timeout in milliseconds for all requests
|
|
timeout: 5000,
|
|
|
|
// Tenant isolation scopes
|
|
scopes: {
|
|
tenant1: { headers: { 'x-tenant-id': '1' } },
|
|
tenant2: { headers: { 'x-tenant-id': '2' } },
|
|
},
|
|
|
|
// Auth and protocol extensions
|
|
extensions: [jwtAuth, apiKeyAuth],
|
|
|
|
// Plugin hook-phase contracts
|
|
pluginContracts: {
|
|
'rate-limit': { appliesTo: 'POST /users', ensures: ['status != 429'] },
|
|
},
|
|
|
|
// Outbound dependency contracts
|
|
outboundContracts: {
|
|
'payment-api': {
|
|
target: 'https://payments.example.com',
|
|
method: 'POST',
|
|
response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
|
|
}
|
|
}
|
|
})
|
|
```
|
|
|
|
## Schema Annotations
|
|
|
|
APOPHIS reads these OpenAPI schema extensions:
|
|
|
|
| Annotation | Location | Description |
|
|
|---|---|---|
|
|
| `x-category` | Top-level | Route classification: `constructor`, `mutator`, `observer`, `destructor`, `utility` |
|
|
| `x-ensures` | Top-level or `response[statusCode]` | Post-condition contracts (APOSTL formulas) |
|
|
| `x-requires` | Top-level or `response[statusCode]` | Pre-condition contracts (APOSTL formulas) |
|
|
| `x-variants` | Top-level | Request variants for content-type negotiation or feature flags |
|
|
| `x-timeout` | Top-level or `response[statusCode]` | Per-route timeout in milliseconds |
|
|
| `x-outbound` | Top-level | Outbound dependency contracts for this route |
|
|
| `x-streaming` | Top-level | Mark route as streaming (populates `chunks` and `streamDurationMs` in eval context) |
|
|
| `x-validate-runtime` | Top-level or `response[statusCode]` | Toggle runtime validation for this route (default: true) |
|
|
| `x-extension-config` | Top-level | Per-route config for extensions (e.g., `{ jwt: { verify: false } }`) |
|
|
|
|
Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations take precedence for that status code.
|
|
|
|
## Programmatic API
|
|
|
|
After registration, `fastify.apophis` provides:
|
|
|
|
```javascript
|
|
// Run contract tests for all routes
|
|
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })
|
|
|
|
// Run stateful tests
|
|
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
|
|
|
// Run a single scenario
|
|
const scenario = await fastify.apophis.scenario({
|
|
name: 'oauth-basic',
|
|
steps: [...]
|
|
})
|
|
|
|
// Check a single route
|
|
const result = await fastify.apophis.check('GET', '/users/:id')
|
|
|
|
// Get enriched OpenAPI spec with contract metadata
|
|
const spec = fastify.apophis.spec()
|
|
|
|
// Clean up tracked resources
|
|
await fastify.apophis.cleanup()
|
|
|
|
// Test-only utilities (NODE_ENV=test only)
|
|
fastify.apophis.test.registerPluginContracts('name', spec)
|
|
fastify.apophis.test.registerOutboundContracts({ ... })
|
|
fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
|
|
fastify.apophis.test.disableOutboundMocks()
|
|
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
|
|
```
|
|
|
|
## Config Reference
|
|
|
|
For the full configuration reference, see [CLI Reference](cli.md).
|
|
|
|
## Monorepo Workspaces
|
|
|
|
Use `--workspace` to run verify or doctor across all packages:
|
|
|
|
```bash
|
|
apophis verify --workspace --profile quick --format json
|
|
```
|
|
|
|
See [CLI Reference](cli.md) for workspace output format and exit codes.
|