(mess) Stuffing commit.
This commit is contained in:
@@ -122,6 +122,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
|||||||
- [Verify Mode](docs/verify.md) — Deterministic contract verification
|
- [Verify Mode](docs/verify.md) — Deterministic contract verification
|
||||||
- [Observe Mode](docs/observe.md) — Runtime visibility and drift detection
|
- [Observe Mode](docs/observe.md) — Runtime visibility and drift detection
|
||||||
- [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos
|
- [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos
|
||||||
|
- [Quality Engines](docs/quality.md) — Chaos injection, flake detection, mutation testing
|
||||||
- [Performance](docs/performance.md) — Repeatable benchmarks and CPU profiling
|
- [Performance](docs/performance.md) — Repeatable benchmarks and CPU profiling
|
||||||
- [LLM-Safe Adoption](docs/llm-safe-adoption.md) — Scaffolds and CI guards
|
- [LLM-Safe Adoption](docs/llm-safe-adoption.md) — Scaffolds and CI guards
|
||||||
- [Protocol Extensions](docs/attic/protocol-extensions-spec.md) — JWT, X.509, SPIFFE, WIMSE
|
- [Protocol Extensions](docs/attic/protocol-extensions-spec.md) — JWT, X.509, SPIFFE, WIMSE
|
||||||
|
|||||||
@@ -1,51 +1,86 @@
|
|||||||
---
|
---
|
||||||
name: apophis-fastify
|
name: apophis-fastify
|
||||||
description: Use this skill when adding or improving APOPHIS contract-driven testing for Fastify APIs: route schemas, APOSTL x-requires/x-ensures formulas, property and stateful checks, replayable failures, runtime observe hooks, variants, scenarios, and operator-facing adoption guidance.
|
description: Use this skill when adding or improving APOPHIS contract-driven testing for Fastify APIs. This tool finds real implementation bugs—resources that appear to create but cannot be retrieved, updates that silently fail to persist, deletions that leave data visible, cross-tenant leakage, and broken state transitions. Use it to encode intended behavior as executable contracts and verify them continuously, not to paper over failures.
|
||||||
---
|
---
|
||||||
|
|
||||||
# apophis-fastify
|
# apophis-fastify
|
||||||
|
|
||||||
APOPHIS verifies API behavior across operations, state changes, protocol flows, and dependencies. Use it when schema validation is not enough to answer whether an endpoint did the right thing.
|
APOPHIS finds real API behavior bugs that schema validation misses. It verifies that operations cause the state changes, isolation guarantees, and dependency interactions the service actually depends on.
|
||||||
|
|
||||||
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
|
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
|
||||||
|
|
||||||
|
## Core Purpose
|
||||||
|
|
||||||
|
APOPHIS is a correctness tool, not a coverage checkbox. Its job is to find where implementation diverges from intended behavior.
|
||||||
|
|
||||||
|
**What APOPHIS finds:**
|
||||||
|
- POST creates a resource that GET cannot retrieve
|
||||||
|
- PUT updates a field but subsequent GET returns stale data
|
||||||
|
- DELETE removes a resource but it still appears in listings
|
||||||
|
- Cross-tenant data leakage through shared collections
|
||||||
|
- Auth boundaries that return success instead of 401/403
|
||||||
|
- Error paths that violate declared retry semantics
|
||||||
|
- State transitions that skip required steps
|
||||||
|
|
||||||
|
**What APOPHIS does NOT do:**
|
||||||
|
- Prove the whole system correct (it finds counterexamples, not proofs)
|
||||||
|
- Replace integration tests, security review, or domain judgment
|
||||||
|
- Validate JSON Schema (Fastify + Swagger already do this)
|
||||||
|
|
||||||
|
**Critical rule for LLMs:** Do not weaken contracts to make tests pass. A failing contract is a signal that the implementation has a bug or the contract captures wrong intent. Investigate first. Fix the bug if the contract is correct. Fix the contract only if the implementation is right and the contract is wrong.
|
||||||
|
|
||||||
## When To Use
|
## When To Use
|
||||||
|
|
||||||
Use this skill when the operator asks to:
|
Use this skill when the operator asks to:
|
||||||
|
|
||||||
1. Add contract testing, API behavior checks, property tests, stateful tests, or Fastify route verification.
|
1. Add contract testing, API behavior checks, property tests, stateful tests, or Fastify route verification.
|
||||||
2. Improve confidence beyond JSON Schema validation.
|
2. Find real bugs in create/read/update/delete behavior, auth boundaries, tenant isolation, redirects, timeouts, streaming, or external dependency behavior.
|
||||||
3. Check create/read/update/delete behavior, auth boundaries, tenant isolation, redirects, timeouts, streaming, or external dependency behavior.
|
3. Make an API safer for AI-assisted refactoring, CI, or release qualification.
|
||||||
4. Make an API safer for AI-assisted refactoring, CI, or release qualification.
|
4. Evaluate whether APOPHIS would help a project, even if the operator does not name APOPHIS directly.
|
||||||
5. Evaluate whether APOPHIS would help a project, even if the operator does not name APOPHIS directly.
|
|
||||||
|
|
||||||
Read `README.md`, `docs/getting-started.md`, or command-specific docs only when the task needs details not present here.
|
Read `README.md`, `docs/getting-started.md`, or command-specific docs only when the task needs details not present here.
|
||||||
|
|
||||||
## Operator Explanation
|
## Operator Explanation
|
||||||
|
|
||||||
Describe APOPHIS as replayable behavioral checks for declared API contracts.
|
Describe APOPHIS as a bug-finding tool for API behavior.
|
||||||
|
|
||||||
Short explanation:
|
Short explanation:
|
||||||
|
|
||||||
> APOPHIS turns intended API behavior into executable contracts. It checks whether operations cause the state changes, isolation guarantees, and dependency interactions the service depends on, instead of only checking payload shape.
|
> APOPHIS encodes intended API behavior as executable contracts, then automatically finds cases where the implementation violates that behavior. It catches bugs that schema validation cannot: create-not-retrievable, update-not-persisted, delete-still-visible, cross-tenant leakage, and broken state transitions.
|
||||||
|
|
||||||
Use these points when relevant:
|
Use these points when relevant:
|
||||||
|
|
||||||
1. It catches failures schema validation misses: create-not-retrievable, update-not-persisted, delete-still-visible, cross-tenant leakage, and inconsistent error behavior.
|
1. It finds real bugs schema validation misses: resources that create but cannot be read, updates that do not persist, deletes that leave data visible.
|
||||||
2. It gives coding agents a deterministic verification loop after generated changes or refactors.
|
2. It gives coding agents a deterministic verification loop after generated changes or refactors. Run contracts after every change; failures indicate regressions.
|
||||||
3. It reduces review burden by converting agreed behavior into repeatable checks.
|
3. It prevents silent breakage. When a refactoring breaks a state transition or isolation guarantee, contracts fail immediately with a reproducible seed.
|
||||||
4. It improves CI triage with fixed seeds, replay artifacts, and machine-readable output.
|
4. It improves CI triage with fixed seeds, replay artifacts, and machine-readable output.
|
||||||
5. It supports incremental adoption: start with the highest-risk routes, add high-signal formulas, run, fix, and tighten.
|
5. It supports incremental adoption: start with the highest-risk routes, add high-signal behavioral formulas, run, fix real bugs, and tighten.
|
||||||
|
|
||||||
Do not overclaim:
|
Do not overclaim:
|
||||||
|
|
||||||
1. Do not say APOPHIS proves the whole system correct.
|
1. Do not say APOPHIS proves the whole system correct.
|
||||||
2. Do not say contracts replace integration tests, security review, or domain judgment.
|
2. Do not say contracts replace integration tests, security review, or domain judgment.
|
||||||
3. Say explicitly that schema quality and formula quality determine test quality.
|
3. Say explicitly that behavioral contracts—not structural checks—determine bug-finding quality.
|
||||||
|
|
||||||
Good operator ask:
|
Good operator ask:
|
||||||
|
|
||||||
> I can add APOPHIS to the five highest-risk routes first, encode the expected behavior as contracts, run the verifier, and show concrete failures or confidence gaps. I only need route priority and intended behavior where the code is ambiguous.
|
> I can add APOPHIS to the five highest-risk routes first, encode the expected behavior as contracts, run the verifier, and show concrete failures where implementation diverges from intent. I only need route priority and intended behavior where the code is ambiguous.
|
||||||
|
|
||||||
|
## Failure Discipline
|
||||||
|
|
||||||
|
**Never paper over a failing contract.**
|
||||||
|
|
||||||
|
When APOPHIS reports a contract violation, follow this order:
|
||||||
|
|
||||||
|
1. **Replay the failure** with the reported seed to confirm it is real.
|
||||||
|
2. **Investigate the implementation.** Does the handler actually do what the contract expects? Is there a race condition? A missing database commit? A wrong query?
|
||||||
|
3. **Fix the implementation** if the contract correctly describes intended behavior.
|
||||||
|
4. **Fix the contract** only if investigation proves the implementation is correct and the contract over-constrains or misdescribes behavior.
|
||||||
|
5. **Never** weaken a contract because "the test is flaky" or "it is too strict" without first proving the implementation is correct.
|
||||||
|
|
||||||
|
**If a contract fails intermittently**, that is a bug. Intermittent failures indicate nondeterminism: race conditions, uncommitted transactions, time-dependent logic, or randomness in handlers. Do not remove the contract. Isolate the nondeterminism and fix it.
|
||||||
|
|
||||||
|
**If a contract fails only under chaos**, that is a resilience bug. The service does not handle the failure mode correctly. Fix the handler or the contract's error-path expectations.
|
||||||
|
|
||||||
## Context Discipline
|
## Context Discipline
|
||||||
|
|
||||||
@@ -55,7 +90,7 @@ Treat context as a finite budget.
|
|||||||
2. Prefer targeted file reads and symbol searches over loading whole directories.
|
2. Prefer targeted file reads and symbol searches over loading whole directories.
|
||||||
3. Track routes touched, contracts added, seeds used, failures found, and unresolved domain questions.
|
3. Track routes touched, contracts added, seeds used, failures found, and unresolved domain questions.
|
||||||
4. Use progressive disclosure: read command docs only when invoking that command; read protocol docs only for variants, redirects, OAuth-style flows, form posts, streaming, or multipart.
|
4. Use progressive disclosure: read command docs only when invoking that command; read protocol docs only for variants, redirects, OAuth-style flows, form posts, streaming, or multipart.
|
||||||
5. Run small loops: annotate one route group, run the narrowest verification, fix, then widen.
|
5. Run small loops: annotate one route group, run the narrowest verification, fix real bugs, then widen.
|
||||||
|
|
||||||
## Default Workflow
|
## Default Workflow
|
||||||
|
|
||||||
@@ -69,8 +104,8 @@ When entering a Fastify codebase:
|
|||||||
6. Add `x-category` where auto-categorization could be ambiguous.
|
6. Add `x-category` where auto-categorization could be ambiguous.
|
||||||
7. Add `x-requires` for preconditions and `x-ensures` for postconditions.
|
7. Add `x-requires` for preconditions and `x-ensures` for postconditions.
|
||||||
8. Run a focused APOPHIS check, then broader contract or stateful verification.
|
8. Run a focused APOPHIS check, then broader contract or stateful verification.
|
||||||
9. Fix real behavior failures or tighten weak contracts.
|
9. **Fix real behavior failures or tighten weak contracts.** Do not weaken passing contracts to avoid work.
|
||||||
10. Report what changed, what ran, what failed, and what needs operator judgment.
|
10. Report what changed, what ran, what failed, what bugs were found, and what needs operator judgment.
|
||||||
|
|
||||||
## Fast Start
|
## Fast Start
|
||||||
|
|
||||||
@@ -88,12 +123,15 @@ app.post('/users', {
|
|||||||
schema: {
|
schema: {
|
||||||
'x-category': 'constructor',
|
'x-category': 'constructor',
|
||||||
'x-requires': [
|
'x-requires': [
|
||||||
'request_headers(this).x-tenant-id != null'
|
// Precondition: user must not already exist
|
||||||
|
'response_code(GET /users/{request_body(this).email}) == 404'
|
||||||
],
|
],
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
'status:201',
|
// Behavioral: created resource must be retrievable
|
||||||
'response_body(this).id != null',
|
|
||||||
'response_code(GET /users/{response_body(this).id}) == 200',
|
'response_code(GET /users/{response_body(this).id}) == 200',
|
||||||
|
// Behavioral: round-trip equality
|
||||||
|
'response_body(this) == request_body(this)',
|
||||||
|
// Behavioral: cross-route field persistence
|
||||||
'response_body(GET /users/{response_body(this).id}).email == request_body(this).email'
|
'response_body(GET /users/{response_body(this).id}).email == request_body(this).email'
|
||||||
],
|
],
|
||||||
body: {
|
body: {
|
||||||
@@ -145,25 +183,34 @@ Test-only helpers:
|
|||||||
4. `fastify.apophis.test.disableOutboundMocks()`
|
4. `fastify.apophis.test.disableOutboundMocks()`
|
||||||
5. `fastify.apophis.test.getOutboundCalls(...)`
|
5. `fastify.apophis.test.getOutboundCalls(...)`
|
||||||
|
|
||||||
## Contract Quality
|
## Contract Quality: Behavioral, Not Structural
|
||||||
|
|
||||||
Minimum:
|
**Structural checks are useless.** Fastify + `@fastify/swagger` already enforce status codes, required fields, and types. Behavioral contracts find what schemas cannot.
|
||||||
|
|
||||||
1. Each mutating route has a status expectation.
|
**Minimum behavioral baseline:**
|
||||||
2. Each response with identity has key field non-null checks.
|
|
||||||
|
1. Constructor routes verify cross-route retrievability.
|
||||||
|
2. Mutator routes verify state-change visibility.
|
||||||
|
3. Destructor routes verify unavailability after deletion.
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
status:201
|
// Constructor: resource must be retrievable after creation
|
||||||
response_body(this).id != null
|
response_code(GET /users/{response_body(this).id}) == 200
|
||||||
|
|
||||||
|
// Mutator: changed field must persist
|
||||||
|
response_body(GET /users/{request_params(this).id}).status == request_body(this).status
|
||||||
|
|
||||||
|
// Destructor: deleted resource must not be retrievable
|
||||||
|
response_code(GET /users/{request_params(this).id}) == 404
|
||||||
```
|
```
|
||||||
|
|
||||||
Production baseline:
|
**Production baseline:**
|
||||||
|
|
||||||
1. Constructor routes check that created resources are retrievable.
|
1. Constructor routes check that created resources are retrievable.
|
||||||
2. Mutator routes check that persisted state reflects the mutation.
|
2. Mutator routes check that persisted state reflects the mutation.
|
||||||
3. Destructor routes check that deleted resources are unavailable or marked inactive.
|
3. Destructor routes check that deleted resources are unavailable or marked inactive.
|
||||||
|
|
||||||
High-confidence contracts add:
|
**High-confidence contracts add:**
|
||||||
|
|
||||||
1. Tenant isolation.
|
1. Tenant isolation.
|
||||||
2. Auth and permission behavior.
|
2. Auth and permission behavior.
|
||||||
@@ -176,37 +223,32 @@ High-confidence contracts add:
|
|||||||
|
|
||||||
Constructor routes, such as `POST /collection`:
|
Constructor routes, such as `POST /collection`:
|
||||||
|
|
||||||
1. Response has identity.
|
1. Resource is retrievable after creation.
|
||||||
2. Created resource is retrievable.
|
2. Persisted fields reflect request fields.
|
||||||
3. Persisted fields reflect request fields.
|
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
status:201
|
|
||||||
response_body(this).id != null
|
|
||||||
response_code(GET /items/{response_body(this).id}) == 200
|
response_code(GET /items/{response_body(this).id}) == 200
|
||||||
response_body(GET /items/{response_body(this).id}).name == request_body(this).name
|
response_body(GET /items/{response_body(this).id}).name == request_body(this).name
|
||||||
```
|
```
|
||||||
|
|
||||||
Mutator routes, such as `PUT`, `PATCH`, or action `POST`:
|
Mutator routes, such as `PUT`, `PATCH`, or action `POST`:
|
||||||
|
|
||||||
1. Mutation succeeds with expected code.
|
1. Changed field actually changed and persists.
|
||||||
2. Changed field actually changed.
|
2. Unrelated invariants still hold.
|
||||||
3. Unrelated invariants still hold.
|
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
status:200
|
response_body(GET /items/{request_params(this).id}).status == request_body(this).status
|
||||||
response_body(this).status == request_body(this).status
|
previous(response_body(this).version) < response_body(this).version
|
||||||
response_body(this).updatedAt != null
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Destructor routes:
|
Destructor routes:
|
||||||
|
|
||||||
1. Delete returns expected code.
|
1. Follow-up retrieval fails or shows a domain-specific inactive state.
|
||||||
2. Follow-up retrieval fails or shows a domain-specific inactive state.
|
2. Previous state is preserved if the API returns deleted data.
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
status:204 || status:200
|
|
||||||
response_code(GET /items/{request_params(this).id}) == 404
|
response_code(GET /items/{request_params(this).id}) == 404
|
||||||
|
response_body(this) == previous(response_body(GET /items/{request_params(this).id}))
|
||||||
```
|
```
|
||||||
|
|
||||||
Observer routes:
|
Observer routes:
|
||||||
@@ -258,7 +300,7 @@ Use these patterns when they match the API:
|
|||||||
6. Error consistency: expected error status implies expected error payload fields.
|
6. Error consistency: expected error status implies expected error payload fields.
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
if status:401 then response_body(this).error != null else true
|
if status:401 then response_body(this).error.length > 0 else true
|
||||||
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
|
if request_headers(this).x-tenant-id != null then response_headers(this).x-tenant-id == request_headers(this).x-tenant-id else true
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -316,8 +358,9 @@ await app.apophis.scenario({
|
|||||||
steps: [
|
steps: [
|
||||||
{
|
{
|
||||||
name: 'authorize',
|
name: 'authorize',
|
||||||
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
|
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
|
||||||
expect: ['status:200', 'response_payload(this).code != null'],
|
// Behavioral: state parameter round-trips for CSRF protection
|
||||||
|
expect: ['response_payload(this).state == request_query(this).state'],
|
||||||
capture: { code: 'response_payload(this).code' }
|
capture: { code: 'response_payload(this).code' }
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -325,9 +368,10 @@ await app.apophis.scenario({
|
|||||||
request: {
|
request: {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
url: '/oauth/token',
|
url: '/oauth/token',
|
||||||
form: { grant_type: 'authorization_code', code: '$authorize.code' }
|
form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
|
||||||
},
|
},
|
||||||
expect: ['status:200', 'response_payload(this).access_token != null']
|
// Behavioral: issued token preserves the requested scope
|
||||||
|
expect: ['response_payload(this).scope == request_body(this).scope']
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
})
|
})
|
||||||
@@ -347,27 +391,25 @@ Prefer deterministic verification for CI, regression triage, and AI-generated ch
|
|||||||
1. Capture and reuse seeds from verify and qualify runs.
|
1. Capture and reuse seeds from verify and qualify runs.
|
||||||
2. Use replay artifacts for failure triage before changing production logic.
|
2. Use replay artifacts for failure triage before changing production logic.
|
||||||
3. Preserve route identity as `METHOD /path` in notes and reports.
|
3. Preserve route identity as `METHOD /path` in notes and reports.
|
||||||
4. If a failure is not reproducible, check for source drift, external dependencies, time, randomness, and insufficient cleanup before weakening the contract.
|
4. **If a failure is not reproducible, treat it as a bug, not a flaky test.** Check for source drift, external dependencies, time, randomness, and insufficient cleanup. Do not weaken the contract without proving the implementation is correct.
|
||||||
5. Treat nondeterminism as a quality issue to isolate.
|
5. Treat nondeterminism as a quality issue to isolate and fix.
|
||||||
|
|
||||||
Operator framing:
|
Operator framing:
|
||||||
|
|
||||||
> The failing seed gives us a reproducible behavioral example. I'll replay it first so we can distinguish a real regression from source drift or nondeterministic app state.
|
> The failing seed gives us a reproducible behavioral counterexample. I'll replay it first to confirm the bug, then investigate the implementation before changing anything.
|
||||||
|
|
||||||
## Progressive Complexity
|
## Progressive Complexity
|
||||||
|
|
||||||
Start simple and add depth only where it pays off:
|
Start with behavioral contracts and add depth only where it pays off:
|
||||||
|
|
||||||
**Level 1 — Status and shape**: Every route gets an expected status code and key field existence.
|
**Level 1 — Cross-route behavior**: Every constructor checks retrievability.
|
||||||
```apostl
|
|
||||||
status:201
|
|
||||||
response_body(this).id != null
|
|
||||||
```
|
|
||||||
|
|
||||||
**Level 2 — Cross-route behavior**: Constructors check retrievability; mutators check persistence.
|
|
||||||
```apostl
|
```apostl
|
||||||
response_code(GET /users/{response_body(this).id}) == 200
|
response_code(GET /users/{response_body(this).id}) == 200
|
||||||
response_body(GET /users/{response_body(this).id}).email == request_body(this).email
|
```
|
||||||
|
|
||||||
|
**Level 2 — State persistence**: Mutators check that changes are visible.
|
||||||
|
```apostl
|
||||||
|
response_body(GET /users/{request_params(this).id}).email == request_body(this).email
|
||||||
```
|
```
|
||||||
|
|
||||||
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
|
**Level 3 — Isolation and boundaries**: Tenant, auth, and idempotency checks.
|
||||||
@@ -377,19 +419,21 @@ if request_headers(this).x-tenant-id != null then response_headers(this).x-tenan
|
|||||||
|
|
||||||
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
|
**Level 4 — Protocol and dependency flows**: Variants, scenarios, outbound contracts, and chaos.
|
||||||
|
|
||||||
Add level 2 before level 4. Do not skip level 2 for resource APIs.
|
Add level 1 before level 4. Do not skip level 1 for resource APIs.
|
||||||
|
|
||||||
## Anti-Patterns
|
## Anti-Patterns
|
||||||
|
|
||||||
Do not:
|
Do not:
|
||||||
|
|
||||||
1. Assert only `status:200` everywhere.
|
1. Assert only `status:200` everywhere. Schema validation already checks this.
|
||||||
2. Duplicate JSON Schema checks while ignoring behavior.
|
2. Check `response_body(this).id != null` when the schema already requires `id`.
|
||||||
3. Encode route internals instead of API-observable outcomes.
|
3. Duplicate JSON Schema checks while ignoring cross-route behavior.
|
||||||
4. Ignore delete/retrieve or update/retrieve relationships.
|
4. Encode route internals instead of API-observable outcomes.
|
||||||
5. Treat stateful mode as optional for resource APIs.
|
5. Ignore delete/retrieve or update/retrieve relationships.
|
||||||
6. Ask the operator to review every formula before running; run first when intent is clear, then ask about ambiguous domain behavior.
|
6. Treat stateful mode as optional for resource APIs.
|
||||||
7. Load every doc file before making a small change.
|
7. **Weaken a contract to make a test pass without proving the implementation is correct.**
|
||||||
|
8. Ask the operator to review every formula before running; run first when intent is clear, then ask about ambiguous domain behavior.
|
||||||
|
9. Load every doc file before making a small change.
|
||||||
|
|
||||||
## Verification Commands
|
## Verification Commands
|
||||||
|
|
||||||
@@ -417,5 +461,6 @@ For each route, ask:
|
|||||||
2. What must be true after this call?
|
2. What must be true after this call?
|
||||||
3. What related call should now behave differently?
|
3. What related call should now behave differently?
|
||||||
4. What isolation, security, dependency, or protocol expectation should not regress?
|
4. What isolation, security, dependency, or protocol expectation should not regress?
|
||||||
|
5. If a contract fails, is the implementation wrong or is the contract wrong?
|
||||||
|
|
||||||
Write those expectations as formulas and run them continuously.
|
Write those expectations as behavioral formulas, run them continuously, and treat every failure as a bug to investigate—not an obstacle to remove.
|
||||||
|
|||||||
+14
-14
@@ -10,11 +10,10 @@ Chaos testing applies the invariant-driven verification approach from [Invariant
|
|||||||
const result = await fastify.apophis.contract({
|
const result = await fastify.apophis.contract({
|
||||||
runs: 50,
|
runs: 50,
|
||||||
chaos: {
|
chaos: {
|
||||||
probability: 0.1, // 10% of requests get chaos
|
delay: { probability: 0.1, minMs: 100, maxMs: 500 },
|
||||||
delay: { probability: 1, minMs: 100, maxMs: 500 },
|
error: { probability: 0.1, statusCode: 503 },
|
||||||
error: { probability: 1, statusCode: 503 },
|
dropout: { probability: 0.05 },
|
||||||
dropout: { probability: 1 },
|
corruption: { probability: 0.1 },
|
||||||
corruption: { probability: 1 },
|
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
```
|
```
|
||||||
@@ -26,7 +25,6 @@ const result = await fastify.apophis.contract({
|
|||||||
Adds artificial latency. Tests timeout contracts:
|
Adds artificial latency. Tests timeout contracts:
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
timeout_occurred(this) == false
|
|
||||||
response_time(this) < 1000
|
response_time(this) < 1000
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -37,7 +35,8 @@ response_time(this) < 1000
|
|||||||
Forces HTTP status codes. Tests error-handling contracts:
|
Forces HTTP status codes. Tests error-handling contracts:
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
if status:503 then response_body(this).retry_after != null
|
// Behavioral: when the service is unavailable, the client receives a valid retry signal
|
||||||
|
if status:503 then response_headers(this).retry-after > 0
|
||||||
```
|
```
|
||||||
|
|
||||||
### Dropout
|
### Dropout
|
||||||
@@ -45,7 +44,8 @@ if status:503 then response_body(this).retry_after != null
|
|||||||
Simulates network failure (status 0). Tests fallback contracts:
|
Simulates network failure (status 0). Tests fallback contracts:
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
status:200 || status:0
|
// Behavioral: partial failure must still return previously cached data
|
||||||
|
if status:0 then response_body(this).cached_data == previous(response_body(GET /cache/{request_params(this).key}))
|
||||||
```
|
```
|
||||||
|
|
||||||
### Corruption
|
### Corruption
|
||||||
@@ -53,7 +53,8 @@ status:200 || status:0
|
|||||||
Mutates response bodies. Tests parsing robustness:
|
Mutates response bodies. Tests parsing robustness:
|
||||||
|
|
||||||
```apostl
|
```apostl
|
||||||
response_body(this).id != null
|
// Behavioral: corrupted requests maintain traceability for debugging
|
||||||
|
if status:400 then response_body(this).request_id == request_headers(this).x-request-id
|
||||||
```
|
```
|
||||||
|
|
||||||
## Corruption Strategies
|
## Corruption Strategies
|
||||||
@@ -104,7 +105,7 @@ Failed tests include chaos events in diagnostics:
|
|||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"statusCode": 503,
|
"statusCode": 503,
|
||||||
"error": "Contract violation: status:200",
|
"error": "Contract violation: if status:503 then response_headers(this).retry-after > 0",
|
||||||
"chaosEvents": [
|
"chaosEvents": [
|
||||||
{
|
{
|
||||||
"type": "error",
|
"type": "error",
|
||||||
@@ -122,7 +123,7 @@ Failed tests include chaos events in diagnostics:
|
|||||||
|
|
||||||
1. **Start small**: `probability: 0.05` (5% of requests)
|
1. **Start small**: `probability: 0.05` (5% of requests)
|
||||||
2. **Test one failure mode at a time**: Comment out other chaos types
|
2. **Test one failure mode at a time**: Comment out other chaos types
|
||||||
3. **Verify contracts handle chaos**: `if status:503 then response_body(this).error != null`
|
3. **Verify contracts handle chaos**: `if status:503 then response_code(GET /health) == 200`
|
||||||
4. **Use seeds for reproducibility**: `seed: 42` makes chaos deterministic
|
4. **Use seeds for reproducibility**: `seed: 42` makes chaos deterministic
|
||||||
|
|
||||||
## Example: Testing Retry Logic
|
## Example: Testing Retry Logic
|
||||||
@@ -131,7 +132,7 @@ Failed tests include chaos events in diagnostics:
|
|||||||
fastify.get('/data', {
|
fastify.get('/data', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
'if status:503 then response_headers(this).retry-after != null',
|
'if status:503 then response_headers(this).retry-after > 0',
|
||||||
'redirect_count(this) <= 3',
|
'redirect_count(this) <= 3',
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
@@ -140,8 +141,7 @@ fastify.get('/data', {
|
|||||||
// Test
|
// Test
|
||||||
const result = await fastify.apophis.contract({
|
const result = await fastify.apophis.contract({
|
||||||
chaos: {
|
chaos: {
|
||||||
probability: 0.2,
|
error: { probability: 0.2, statusCode: 503 },
|
||||||
error: { probability: 1, statusCode: 503 },
|
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
```
|
```
|
||||||
|
|||||||
+40
-26
@@ -5,21 +5,25 @@ import crypto from 'crypto'
|
|||||||
const fastify = Fastify()
|
const fastify = Fastify()
|
||||||
|
|
||||||
await fastify.register(apophisPlugin, {
|
await fastify.register(apophisPlugin, {
|
||||||
runtime: 'error', // Validate contracts on every request
|
runtime: 'error',
|
||||||
cleanup: true, // Auto-cleanup resources on exit
|
cleanup: true,
|
||||||
})
|
})
|
||||||
|
|
||||||
// In-memory store for demo
|
|
||||||
const users = new Map<string, { id: string; email: string; name: string }>()
|
const users = new Map<string, { id: string; email: string; name: string }>()
|
||||||
|
|
||||||
// CREATE — constructor
|
// CREATE — constructor
|
||||||
|
// Behavioral: the created user must be retrievable.
|
||||||
|
// Note: we do not write 'status:201' or 'response_body(this).id != null'.
|
||||||
|
// The schema already validates status codes and required fields.
|
||||||
|
// Contracts should test behavior across operations, not structure.
|
||||||
fastify.post('/users', {
|
fastify.post('/users', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'constructor',
|
'x-category': 'constructor',
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
'status:201',
|
// Round-trip: the server returns exactly what we sent (no mutation, no drops)
|
||||||
'response_body(this).id != null',
|
'response_body(this) == request_body(this)',
|
||||||
'response_body(this).email == request_body(this).email',
|
// Cross-route: the created user must be retrievable
|
||||||
|
'response_code(GET /users/{response_body(this).id}) == 200',
|
||||||
],
|
],
|
||||||
body: {
|
body: {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
@@ -49,19 +53,21 @@ fastify.post('/users', {
|
|||||||
})
|
})
|
||||||
|
|
||||||
// READ — observer
|
// READ — observer
|
||||||
|
// Behavioral: the returned user must match the requested id.
|
||||||
fastify.get('/users/:id', {
|
fastify.get('/users/:id', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'observer',
|
'x-category': 'observer',
|
||||||
'x-requires': ['users:id'],
|
'x-requires': [
|
||||||
|
// Precondition: the user must exist for this read to be valid
|
||||||
|
'response_code(GET /users/{request_params(this).id}) == 200'
|
||||||
|
],
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
'status:200',
|
// The returned id must match the requested id (no mix-up)
|
||||||
'response_body(this).id == request_params(this).id',
|
'response_body(this).id == request_params(this).id',
|
||||||
],
|
],
|
||||||
params: {
|
params: {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
properties: {
|
properties: { id: { type: 'string' } },
|
||||||
id: { type: 'string' }
|
|
||||||
},
|
|
||||||
required: ['id']
|
required: ['id']
|
||||||
},
|
},
|
||||||
response: {
|
response: {
|
||||||
@@ -84,19 +90,21 @@ fastify.get('/users/:id', {
|
|||||||
})
|
})
|
||||||
|
|
||||||
// UPDATE — mutator
|
// UPDATE — mutator
|
||||||
|
// Behavioral: after update, the change must be visible on read.
|
||||||
fastify.put('/users/:id', {
|
fastify.put('/users/:id', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'mutator',
|
'x-category': 'mutator',
|
||||||
'x-requires': ['users:id'],
|
'x-requires': [
|
||||||
|
// The user must exist before updating
|
||||||
|
'response_code(GET /users/{request_params(this).id}) == 200'
|
||||||
|
],
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
'status:200',
|
// Cross-route: after update, reading the user shows the new data
|
||||||
'response_body(this).id == request_params(this).id',
|
'response_body(GET /users/{request_params(this).id}).email == request_body(this).email',
|
||||||
],
|
],
|
||||||
params: {
|
params: {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
properties: {
|
properties: { id: { type: 'string' } },
|
||||||
id: { type: 'string' }
|
|
||||||
},
|
|
||||||
required: ['id']
|
required: ['id']
|
||||||
},
|
},
|
||||||
body: {
|
body: {
|
||||||
@@ -132,34 +140,40 @@ fastify.put('/users/:id', {
|
|||||||
})
|
})
|
||||||
|
|
||||||
// DELETE — destructor
|
// DELETE — destructor
|
||||||
|
// Behavioral: after deletion, the user must no longer exist.
|
||||||
fastify.delete('/users/:id', {
|
fastify.delete('/users/:id', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'destructor',
|
'x-category': 'destructor',
|
||||||
'x-requires': ['users:id'],
|
'x-requires': [
|
||||||
'x-ensures': ['status:204'],
|
// The user must exist before deleting
|
||||||
|
'response_code(GET /users/{request_params(this).id}) == 200'
|
||||||
|
],
|
||||||
|
'x-ensures': [
|
||||||
|
// After deletion, the user is gone
|
||||||
|
'response_code(GET /users/{request_params(this).id}) == 404',
|
||||||
|
// The deleted user data is returned (matches pre-deletion read)
|
||||||
|
'response_body(this) == previous(response_body(GET /users/{request_params(this).id}))',
|
||||||
|
],
|
||||||
params: {
|
params: {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
properties: {
|
properties: { id: { type: 'string' } },
|
||||||
id: { type: 'string' }
|
|
||||||
},
|
|
||||||
required: ['id']
|
required: ['id']
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}, async (req, reply) => {
|
}, async (req, reply) => {
|
||||||
|
const user = users.get(req.params.id)
|
||||||
users.delete(req.params.id)
|
users.delete(req.params.id)
|
||||||
reply.status(204)
|
reply.status(200)
|
||||||
|
return user
|
||||||
})
|
})
|
||||||
|
|
||||||
await fastify.ready()
|
await fastify.ready()
|
||||||
|
|
||||||
// Run contract tests (all non-utility routes, property-based)
|
|
||||||
const result = await fastify.apophis.contract({ runs: 50 })
|
const result = await fastify.apophis.contract({ runs: 50 })
|
||||||
console.log('Contract tests:', result.summary)
|
console.log('Contract tests:', result.summary)
|
||||||
|
|
||||||
// Run stateful tests (constructor→mutator→destructor sequences)
|
|
||||||
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
||||||
console.log('Stateful tests:', stateful.summary)
|
console.log('Stateful tests:', stateful.summary)
|
||||||
|
|
||||||
// Validate a single route
|
|
||||||
const check = await fastify.apophis.check('POST', '/users')
|
const check = await fastify.apophis.check('POST', '/users')
|
||||||
console.log('POST /users check:', check.ok ? 'PASS' : 'FAIL')
|
console.log('POST /users check:', check.ok ? 'PASS' : 'FAIL')
|
||||||
|
|||||||
@@ -6,18 +6,27 @@ const fastify = Fastify()
|
|||||||
// APOPHIS auto-registers @fastify/swagger
|
// APOPHIS auto-registers @fastify/swagger
|
||||||
await fastify.register(apophisPlugin, {})
|
await fastify.register(apophisPlugin, {})
|
||||||
|
|
||||||
fastify.get('/health', {
|
// Behavioral contract: what you send is what you get back.
|
||||||
|
// This is not a structural test — the schema already validates shape.
|
||||||
|
// This checks that the server does not mutate or drop fields.
|
||||||
|
fastify.post('/echo', {
|
||||||
schema: {
|
schema: {
|
||||||
'x-category': 'observer',
|
'x-category': 'observer',
|
||||||
'x-ensures': ['status:200'],
|
'x-ensures': [
|
||||||
|
'response_body(this) == request_body(this)'
|
||||||
|
],
|
||||||
|
body: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { message: { type: 'string' } }
|
||||||
|
},
|
||||||
response: {
|
response: {
|
||||||
200: {
|
200: {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
properties: { status: { type: 'string' } }
|
properties: { message: { type: 'string' } }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}, async () => ({ status: 'ok' }))
|
}, async (req) => req.body)
|
||||||
|
|
||||||
await fastify.ready()
|
await fastify.ready()
|
||||||
|
|
||||||
|
|||||||
@@ -280,8 +280,8 @@ app.get('/users/:id', {
|
|||||||
type: 'object',
|
type: 'object',
|
||||||
properties: { id: { type: 'string' } },
|
properties: { id: { type: 'string' } },
|
||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
// Standard APOSTL + extension predicates
|
// Behavioral: returned user must match the requested id
|
||||||
'status:200',
|
'response_body(this).id == request_params(this).id',
|
||||||
'graph_check(this).user.can_read_user == true',
|
'graph_check(this).user.can_read_user == true',
|
||||||
'partial_graph(this).tenant.accessible == true',
|
'partial_graph(this).tenant.accessible == true',
|
||||||
],
|
],
|
||||||
|
|||||||
@@ -284,8 +284,8 @@ fastify.get('/api/resource', {
|
|||||||
'x-ensures': [
|
'x-ensures': [
|
||||||
'timeout_occurred(this) == false',
|
'timeout_occurred(this) == false',
|
||||||
'redirect_count(this) == 0',
|
'redirect_count(this) == 0',
|
||||||
'response_code(this) == 200',
|
// Behavioral: created resource must be retrievable
|
||||||
'response_body(this).id != null',
|
'response_code(GET /api/resource/{response_body(this).id}) == 200',
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}, handler)
|
}, handler)
|
||||||
|
|||||||
@@ -94,6 +94,48 @@ apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
|||||||
|
|
||||||
Fix the bug in your handler. Re-run verify. The failure should now pass.
|
Fix the bug in your handler. Re-run verify. The failure should now pass.
|
||||||
|
|
||||||
|
## Behavioral vs Structural Contracts
|
||||||
|
|
||||||
|
APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fastify/swagger` already enforce status codes, required fields, and types. Behavioral contracts catch what schemas cannot:
|
||||||
|
|
||||||
|
| Structural (avoid) | Behavioral (prefer) |
|
||||||
|
|---|---|
|
||||||
|
| `status:200` | `response_body(this) == request_body(this)` |
|
||||||
|
| `response_body(this).id != null` | `response_code(GET /users/{response_body(this).id}) == 200` |
|
||||||
|
| `response_body(this).name != null` | `response_body(GET /users/{id}).name == previous(response_body(this).name)` |
|
||||||
|
|
||||||
|
**Good behavioral patterns (from the paper):**
|
||||||
|
- **Constructor precondition**: Resource must not exist before creation
|
||||||
|
```apostl
|
||||||
|
response_code(GET /users/{request_body(this).email}) == 404
|
||||||
|
```
|
||||||
|
- **Round-trip equality**: POST response matches the request body
|
||||||
|
```apostl
|
||||||
|
response_body(this) == request_body(this)
|
||||||
|
```
|
||||||
|
- **Cross-route retrievability**: Creating a resource makes it readable via GET
|
||||||
|
```apostl
|
||||||
|
response_code(GET /users/{response_body(this).id}) == 200
|
||||||
|
```
|
||||||
|
- **State-change verification**: DELETE causes subsequent GET to return 404
|
||||||
|
```apostl
|
||||||
|
response_code(GET /users/{request_params(this).id}) == 404
|
||||||
|
```
|
||||||
|
- **Previous state preservation**: DELETE returns the last known state
|
||||||
|
```apostl
|
||||||
|
response_body(this) == previous(response_body(GET /users/{request_params(this).id}))
|
||||||
|
```
|
||||||
|
- **Invariant over collections**: All resources satisfy a cross-resource constraint
|
||||||
|
```apostl
|
||||||
|
for t in response_body(GET /tournaments) :-
|
||||||
|
response_body(GET /tournaments/{t.id}/players).length <= t.capacity
|
||||||
|
```
|
||||||
|
|
||||||
|
**Anti-patterns to avoid:**
|
||||||
|
- Checking status codes (handled by schema validation)
|
||||||
|
- Checking field existence (handled by schema validation)
|
||||||
|
- Checking field types (handled by schema validation)
|
||||||
|
|
||||||
## Next Steps
|
## Next Steps
|
||||||
|
|
||||||
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
||||||
|
|||||||
+6
-4
@@ -61,8 +61,9 @@ await fastify.apophis.scenario({
|
|||||||
steps: [
|
steps: [
|
||||||
{
|
{
|
||||||
name: 'authorize',
|
name: 'authorize',
|
||||||
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
|
request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
|
||||||
expect: ['status:200', 'response_payload(this).code != null'],
|
// Behavioral: state parameter round-trips for CSRF protection
|
||||||
|
expect: ['response_payload(this).state == request_query(this).state'],
|
||||||
capture: { code: 'response_payload(this).code' }
|
capture: { code: 'response_payload(this).code' }
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -70,9 +71,10 @@ await fastify.apophis.scenario({
|
|||||||
request: {
|
request: {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
url: '/oauth/token',
|
url: '/oauth/token',
|
||||||
form: { grant_type: 'authorization_code', code: '$authorize.code' }
|
form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
|
||||||
},
|
},
|
||||||
expect: ['status:200', 'response_payload(this).access_token != null']
|
// Behavioral: issued token preserves the requested scope
|
||||||
|
expect: ['response_payload(this).scope == request_body(this).scope']
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
})
|
})
|
||||||
|
|||||||
+226
@@ -0,0 +1,226 @@
|
|||||||
|
# Quality Engines
|
||||||
|
|
||||||
|
APOPHIS includes three quality engines for advanced testing: chaos injection, flake detection, and mutation testing. All require `NODE_ENV=test`.
|
||||||
|
|
||||||
|
## Chaos Injection
|
||||||
|
|
||||||
|
Inject controlled failures into contract tests to validate resilience guarantees. Chaos events are generated by fast-check alongside test data, making them shrinkable — when a test fails, fast-check finds the minimal chaos event that causes the failure.
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const result = await fastify.apophis.contract({
|
||||||
|
runs: 50,
|
||||||
|
chaos: {
|
||||||
|
delay: { probability: 0.1, minMs: 100, maxMs: 500 },
|
||||||
|
error: { probability: 0.1, statusCode: 503 },
|
||||||
|
dropout: { probability: 0.05 },
|
||||||
|
corruption: { probability: 0.1 },
|
||||||
|
},
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
### Event Types
|
||||||
|
|
||||||
|
| Type | Effect | Tests |
|
||||||
|
|------|--------|-------|
|
||||||
|
| `delay` | Artificial latency | `response_time(this) < 1000` |
|
||||||
|
| `error` | Forces HTTP status code | Error-handling contracts |
|
||||||
|
| `dropout` | Network failure (status 0 or 504) | Fallback contracts |
|
||||||
|
| `corruption` | Mutates response bodies | Parsing robustness |
|
||||||
|
|
||||||
|
### Corruption Strategies
|
||||||
|
|
||||||
|
| Strategy | Effect |
|
||||||
|
|----------|--------|
|
||||||
|
| `truncate` | Cuts response body in half |
|
||||||
|
| `malformed` | Returns invalid JSON (`{"broken":`) |
|
||||||
|
| `field-corrupt` | Sets a random field to `null` |
|
||||||
|
|
||||||
|
### Programmatic API
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import {
|
||||||
|
applyChaosToExecution,
|
||||||
|
createChaosEventArbitrary,
|
||||||
|
formatChaosEvents,
|
||||||
|
} from 'apophis-fastify'
|
||||||
|
|
||||||
|
// Apply pre-generated chaos events to a context
|
||||||
|
const result = applyChaosToExecution(ctx, events)
|
||||||
|
|
||||||
|
// Generate deterministic chaos events
|
||||||
|
const arb = createChaosEventArbitrary(config, contractNames)
|
||||||
|
const events = fc.sample(arb, { numRuns: 1, seed: 42 })[0]
|
||||||
|
|
||||||
|
// Format for diagnostics
|
||||||
|
console.log(formatChaosEvents(events))
|
||||||
|
```
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
1. Start small: `probability: 0.05` (5% of requests)
|
||||||
|
2. Test one failure mode at a time
|
||||||
|
3. Verify contracts handle chaos: `if status:503 then response_code(GET /health) == 200`
|
||||||
|
4. Use seeds for reproducibility: `seed: 42`
|
||||||
|
|
||||||
|
## Flake Detection
|
||||||
|
|
||||||
|
Automatically rerun failing tests with varied seeds to detect non-deterministic contracts. A "flake" is a test that fails on one run but passes on another with the same or different seed.
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import { FlakeDetector } from 'apophis-fastify'
|
||||||
|
|
||||||
|
const detector = new FlakeDetector({
|
||||||
|
sameSeedReruns: 1, // Rerun with same seed
|
||||||
|
seedVariations: 3, // Try 3 additional seeds
|
||||||
|
})
|
||||||
|
|
||||||
|
const report = await detector.detectFlake(
|
||||||
|
originalFailingResult,
|
||||||
|
async (seed) => {
|
||||||
|
const suite = await fastify.apophis.contract({ seed })
|
||||||
|
return { passed: suite.summary.failed === 0 }
|
||||||
|
},
|
||||||
|
originalSeed
|
||||||
|
)
|
||||||
|
|
||||||
|
if (report.isFlaky) {
|
||||||
|
console.log(`Flaky with ${report.confidence} confidence`)
|
||||||
|
console.log('Reruns:', report.reruns)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Report Structure
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
{
|
||||||
|
isFlaky: true,
|
||||||
|
confidence: 'high', // 'high' | 'medium' | 'low'
|
||||||
|
reruns: [
|
||||||
|
{ seed: 42, passed: false },
|
||||||
|
{ seed: 43, passed: true },
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Confidence Scoring
|
||||||
|
|
||||||
|
| Pass Rate | Confidence |
|
||||||
|
|-----------|------------|
|
||||||
|
| 0% pass | `high` (deterministic failure) |
|
||||||
|
| < 50% pass | `medium` |
|
||||||
|
| >= 50% pass | `low` (likely flaky) |
|
||||||
|
|
||||||
|
## Mutation Testing
|
||||||
|
|
||||||
|
Measure contract strength by injecting synthetic bugs. A "mutation" is a small change to a contract (e.g., flip `==` to `!=`). If the test suite catches the mutation (fails), the mutation is "killed". If it passes, the mutation "survives" — indicating weak coverage.
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import { runMutationTesting } from 'apophis-fastify/quality/mutation'
|
||||||
|
|
||||||
|
const report = await runMutationTesting(fastify, {
|
||||||
|
runs: 10,
|
||||||
|
seed: 42,
|
||||||
|
maxMutationsPerContract: 5,
|
||||||
|
routes: ['/items'], // Optional: only test these routes
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(`Mutation score: ${report.score}%`)
|
||||||
|
console.log(`Killed: ${report.killed}, Survived: ${report.survived}`)
|
||||||
|
console.log('Weak contracts:', report.weakContracts)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Mutation Operators
|
||||||
|
|
||||||
|
| Type | Example |
|
||||||
|
|------|---------|
|
||||||
|
| `flip-operator` | `== 201` → `!= 201` |
|
||||||
|
| `change-number` | `== 200` → `== 201` |
|
||||||
|
| `remove-clause` | `A && B` → `A` |
|
||||||
|
| `negate-boolean` | `== true` → `== false` |
|
||||||
|
| `swap-variable` | `response_body` → `request_body` |
|
||||||
|
| `remove-ensures` | Remove one ensures clause entirely |
|
||||||
|
|
||||||
|
### Report Structure
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
{
|
||||||
|
score: 85, // 0-100
|
||||||
|
killed: 17,
|
||||||
|
survived: 3,
|
||||||
|
durationMs: 4500,
|
||||||
|
weakContracts: ['POST /items'], // Routes where no mutations were killed
|
||||||
|
mutations: [
|
||||||
|
{
|
||||||
|
mutation: {
|
||||||
|
id: 'm0',
|
||||||
|
route: 'POST /items',
|
||||||
|
original: 'response_code(this) == 201',
|
||||||
|
mutated: 'response_code(this) != 201',
|
||||||
|
type: 'flip-operator',
|
||||||
|
},
|
||||||
|
killed: true,
|
||||||
|
durationMs: 120,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Single Mutation Test
|
||||||
|
|
||||||
|
Test a specific mutation without running the full suite:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import { testMutation } from 'apophis-fastify/quality/mutation'
|
||||||
|
|
||||||
|
const killed = await testMutation(fastify, contract, mutation, {
|
||||||
|
runs: 10,
|
||||||
|
seed: 42,
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Guard
|
||||||
|
|
||||||
|
All quality engines require `NODE_ENV=test`:
|
||||||
|
|
||||||
|
```
|
||||||
|
Error: chaos is only available in test environment.
|
||||||
|
Set NODE_ENV=test to enable quality features.
|
||||||
|
```
|
||||||
|
|
||||||
|
This prevents accidental execution in production or development.
|
||||||
|
|
||||||
|
## Integration Example
|
||||||
|
|
||||||
|
Run all three engines in a CI pipeline:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// 1. Standard contract tests
|
||||||
|
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })
|
||||||
|
|
||||||
|
// 2. Chaos tests
|
||||||
|
const chaosSuite = await fastify.apophis.contract({
|
||||||
|
runs: 50,
|
||||||
|
seed: 42,
|
||||||
|
chaos: { error: { probability: 0.1, statusCode: 503 } },
|
||||||
|
})
|
||||||
|
|
||||||
|
// 3. Flake detection on failures
|
||||||
|
for (const test of suite.tests.filter(t => !t.ok)) {
|
||||||
|
const report = await detector.detectFlake(test, rerunFn, 42)
|
||||||
|
if (report.isFlaky) {
|
||||||
|
console.warn(`Flaky test detected: ${test.name}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. Mutation testing
|
||||||
|
const mutationReport = await runMutationTesting(fastify, { runs: 10 })
|
||||||
|
if (mutationReport.score < 80) {
|
||||||
|
console.warn(`Low mutation score: ${mutationReport.score}%`)
|
||||||
|
}
|
||||||
|
```
|
||||||
+2170
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,6 @@
|
|||||||
|
|
||||||
|
Mercedes Rodriguez
|
||||||
|
#BOAF9148155679Z
|
||||||
|
1500 - 00251743R
|
||||||
|
2000 - 00361Z903R
|
||||||
|
|
||||||
@@ -285,7 +285,7 @@ function evaluateQuantified(
|
|||||||
bodyFn: (item: unknown) => boolean
|
bodyFn: (item: unknown) => boolean
|
||||||
): boolean {
|
): boolean {
|
||||||
if (!Array.isArray(collection)) {
|
if (!Array.isArray(collection)) {
|
||||||
throw new Error(`Quantified expression requires an array collection, got: ${typeof collection}`)
|
return false
|
||||||
}
|
}
|
||||||
if (quantifier === 'for') {
|
if (quantifier === 'for') {
|
||||||
return collection.every(bodyFn)
|
return collection.every(bodyFn)
|
||||||
@@ -299,7 +299,7 @@ async function evaluateQuantifiedAsync(
|
|||||||
bodyFn: (item: unknown) => Promise<boolean>
|
bodyFn: (item: unknown) => Promise<boolean>
|
||||||
): Promise<boolean> {
|
): Promise<boolean> {
|
||||||
if (!Array.isArray(collection)) {
|
if (!Array.isArray(collection)) {
|
||||||
throw new Error(`Quantified expression requires an array collection, got: ${typeof collection}`)
|
return false
|
||||||
}
|
}
|
||||||
if (quantifier === 'for') {
|
if (quantifier === 'for') {
|
||||||
for (const item of collection) {
|
for (const item of collection) {
|
||||||
|
|||||||
+20
-1
@@ -13,14 +13,33 @@ export default fp(apophisPlugin, {
|
|||||||
|
|
||||||
export * from './types.js'
|
export * from './types.js'
|
||||||
|
|
||||||
// Chaos-v3: Pure chaos application for property-based testing
|
// Quality engines
|
||||||
export {
|
export {
|
||||||
applyChaosToExecution,
|
applyChaosToExecution,
|
||||||
applyChaosToAllResponses,
|
applyChaosToAllResponses,
|
||||||
createChaosEventArbitrary,
|
createChaosEventArbitrary,
|
||||||
extractDelays,
|
extractDelays,
|
||||||
sleep,
|
sleep,
|
||||||
|
hasAppliedChaos,
|
||||||
|
formatChaosEvents,
|
||||||
type ChaosEvent,
|
type ChaosEvent,
|
||||||
type ChaosEventType,
|
type ChaosEventType,
|
||||||
type ChaosApplicationResult,
|
type ChaosApplicationResult,
|
||||||
} from './quality/chaos-v3.js'
|
} from './quality/chaos-v3.js'
|
||||||
|
|
||||||
|
export {
|
||||||
|
FlakeDetector,
|
||||||
|
type FlakeReport,
|
||||||
|
type FlakeRerun,
|
||||||
|
type FlakeOptions,
|
||||||
|
} from './quality/flake.js'
|
||||||
|
|
||||||
|
export {
|
||||||
|
runMutationTesting,
|
||||||
|
testMutation,
|
||||||
|
type Mutation,
|
||||||
|
type MutationType,
|
||||||
|
type MutationResult,
|
||||||
|
type MutationReport,
|
||||||
|
type MutationConfig,
|
||||||
|
} from './quality/mutation.js'
|
||||||
|
|||||||
+11
-5
@@ -256,20 +256,26 @@ export async function runMutationTesting(
|
|||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* Run petit tests with a mutated contract.
|
* Run petit tests with a mutated contract.
|
||||||
* This is a simplified version that tests a single mutated contract.
|
* Injects the mutated contract so the runner uses it instead of discovering from Fastify.
|
||||||
*/
|
*/
|
||||||
async function runPetitTestsWithMutation(
|
async function runPetitTestsWithMutation(
|
||||||
fastify: FastifyInjectInstance,
|
fastify: FastifyInjectInstance,
|
||||||
config: { runs?: number; seed?: number },
|
config: { runs?: number; seed?: number },
|
||||||
mutatedContract: RouteContract
|
mutatedContract: RouteContract
|
||||||
): Promise<TestSuite> {
|
): Promise<TestSuite> {
|
||||||
// For now, run the full suite - the mutated contract will be discovered
|
return runPetitTests(
|
||||||
// In a real implementation, you'd inject the mutated contract into the discovery
|
fastify,
|
||||||
return runPetitTests(fastify, {
|
{
|
||||||
runs: config.runs ?? 10,
|
runs: config.runs ?? 10,
|
||||||
seed: config.seed,
|
seed: config.seed,
|
||||||
routes: [`${mutatedContract.method} ${mutatedContract.path}`],
|
routes: [`${mutatedContract.method} ${mutatedContract.path}`],
|
||||||
})
|
},
|
||||||
|
undefined,
|
||||||
|
undefined,
|
||||||
|
undefined,
|
||||||
|
undefined,
|
||||||
|
[mutatedContract]
|
||||||
|
)
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* Quick mutation test for a single contract formula.
|
* Quick mutation test for a single contract formula.
|
||||||
|
|||||||
@@ -523,11 +523,35 @@ test('evaluate: returns error for missing previous context', () => {
|
|||||||
assert.strictEqual(result.success, false)
|
assert.strictEqual(result.success, false)
|
||||||
assert.ok((result as { success: false; error: string }).error.includes('No previous context'))
|
assert.ok((result as { success: false; error: string }).error.includes('No previous context'))
|
||||||
})
|
})
|
||||||
test('evaluate: returns error for non-array in quantified expression', () => {
|
test('evaluate: returns false for non-array in quantified expression', () => {
|
||||||
const ast = parse('for x in response_code(this): x == 1')
|
const ast = parse('for x in response_code(this): x == 1')
|
||||||
const result = evaluate(ast.ast, makeContext())
|
const result = evaluate(ast.ast, makeContext())
|
||||||
assert.strictEqual(result.success, false)
|
assert.strictEqual(result.success, true)
|
||||||
assert.ok((result as { success: false; error: string }).error.includes('array collection'))
|
assert.strictEqual((result as { success: true; value: unknown }).value, false)
|
||||||
|
})
|
||||||
|
test('evaluate: returns false for undefined collection in quantified expression', () => {
|
||||||
|
const ast = parse('for x in response_body(this): x == 1')
|
||||||
|
const result = evaluate(ast.ast, makeContext({ response: { body: undefined, headers: {}, statusCode: 200, responseTime: 0 } }))
|
||||||
|
assert.strictEqual(result.success, true)
|
||||||
|
assert.strictEqual((result as { success: true; value: unknown }).value, false)
|
||||||
|
})
|
||||||
|
test('evaluate: returns false for error object collection in quantified expression', () => {
|
||||||
|
const ast = parse('for x in response_body(this): x == 1')
|
||||||
|
const result = evaluate(ast.ast, makeContext({ response: { body: { error: 'Chaos error: forced 503' }, headers: {}, statusCode: 200, responseTime: 0 } }))
|
||||||
|
assert.strictEqual(result.success, true)
|
||||||
|
assert.strictEqual((result as { success: true; value: unknown }).value, false)
|
||||||
|
})
|
||||||
|
test('evaluate: quantified nested in conditional handles undefined gracefully', () => {
|
||||||
|
const ast = parse('if status:503 then for x in response_body(this): x == 1 else true')
|
||||||
|
const result = evaluate(ast.ast, makeContext({ response: { statusCode: 503, body: undefined, headers: {}, responseTime: 0 } }))
|
||||||
|
assert.strictEqual(result.success, true)
|
||||||
|
assert.strictEqual((result as { success: true; value: unknown }).value, false)
|
||||||
|
})
|
||||||
|
test('evaluate: exists returns false for non-array collection', () => {
|
||||||
|
const ast = parse('exists x in response_body(this): x == 1')
|
||||||
|
const result = evaluate(ast.ast, makeContext({ response: { body: { error: 'fail' }, headers: {}, statusCode: 200, responseTime: 0 } }))
|
||||||
|
assert.strictEqual(result.success, true)
|
||||||
|
assert.strictEqual((result as { success: true; value: unknown }).value, false)
|
||||||
})
|
})
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
// Unit Tests: Substitutor
|
// Unit Tests: Substitutor
|
||||||
|
|||||||
@@ -105,12 +105,13 @@ export const runPetitTests = async (
|
|||||||
scopeRegistry?: ScopeRegistry,
|
scopeRegistry?: ScopeRegistry,
|
||||||
extensionRegistry?: ExtensionRegistry,
|
extensionRegistry?: ExtensionRegistry,
|
||||||
pluginContractRegistry?: import('../domain/plugin-contracts.js').PluginContractRegistry,
|
pluginContractRegistry?: import('../domain/plugin-contracts.js').PluginContractRegistry,
|
||||||
outboundContractRegistry?: OutboundContractRegistry
|
outboundContractRegistry?: OutboundContractRegistry,
|
||||||
|
overrideContracts?: RouteContract[]
|
||||||
): Promise<TestSuite> => {
|
): Promise<TestSuite> => {
|
||||||
const startTime = Date.now()
|
const startTime = Date.now()
|
||||||
if (extensionRegistry) await extensionRegistry.runSuiteStartHooks(config)
|
if (extensionRegistry) await extensionRegistry.runSuiteStartHooks(config)
|
||||||
|
|
||||||
const allRoutes = discoverRoutes(fastify)
|
const allRoutes = overrideContracts ?? discoverRoutes(fastify)
|
||||||
const { routes, skippedRoutes } = filterPetitRoutes(allRoutes, config)
|
const { routes, skippedRoutes } = filterPetitRoutes(allRoutes, config)
|
||||||
|
|
||||||
// Merge plugin contracts into route contracts
|
// Merge plugin contracts into route contracts
|
||||||
|
|||||||
@@ -0,0 +1,425 @@
|
|||||||
|
/**
|
||||||
|
* Quality Engine Tests — Flake Detection, Mutation Testing, Chaos
|
||||||
|
*/
|
||||||
|
import { test } from 'node:test'
|
||||||
|
import assert from 'node:assert'
|
||||||
|
import Fastify from 'fastify'
|
||||||
|
import swagger from '@fastify/swagger'
|
||||||
|
import apophisPlugin from '../index.js'
|
||||||
|
import { FlakeDetector } from '../quality/flake.js'
|
||||||
|
import { runMutationTesting, testMutation, type Mutation } from '../quality/mutation.js'
|
||||||
|
import * as fc from 'fast-check'
|
||||||
|
import {
|
||||||
|
applyChaosToExecution,
|
||||||
|
applyChaosToAllResponses,
|
||||||
|
applyChaosToDependencyResponse,
|
||||||
|
createChaosEventArbitrary,
|
||||||
|
extractDelays,
|
||||||
|
sleep,
|
||||||
|
hasAppliedChaos,
|
||||||
|
formatChaosEvents,
|
||||||
|
type ChaosEvent,
|
||||||
|
} from '../quality/chaos-v3.js'
|
||||||
|
import type { EvalContext, TestResult, RouteContract } from '../types.js'
|
||||||
|
|
||||||
|
// Quality engines require test environment
|
||||||
|
process.env.NODE_ENV = 'test'
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Helpers
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
function makeCtx(overrides?: Partial<EvalContext['response']>): EvalContext {
|
||||||
|
return {
|
||||||
|
request: { body: {}, headers: {}, query: {}, params: {} },
|
||||||
|
response: { body: { id: '1' }, headers: {}, statusCode: 200, ...overrides },
|
||||||
|
} as EvalContext
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeTestResult(ok: boolean): TestResult {
|
||||||
|
return {
|
||||||
|
ok,
|
||||||
|
name: 'test',
|
||||||
|
id: 1,
|
||||||
|
diagnostics: ok ? {} : { error: 'failed' },
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Chaos Tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test('applyChaosToExecution: no chaos when events are empty', () => {
|
||||||
|
const ctx = makeCtx()
|
||||||
|
const result = applyChaosToExecution(ctx, [])
|
||||||
|
assert.strictEqual(result.applied, false)
|
||||||
|
assert.strictEqual(result.ctx.response.statusCode, 200)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: inbound error changes status code', () => {
|
||||||
|
const ctx = makeCtx()
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'inbound-error', target: 'inbound', statusCode: 503 },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.applied, true)
|
||||||
|
assert.strictEqual(result.ctx.response.statusCode, 503)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: inbound dropout simulates gateway timeout', () => {
|
||||||
|
const ctx = makeCtx()
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'inbound-dropout', target: 'inbound' },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.ctx.response.statusCode, 504)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: inbound corruption truncates response body', () => {
|
||||||
|
const ctx = makeCtx({ body: { id: '1', name: 'Alice', email: 'a@b.com' } })
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'inbound-corruption', target: 'inbound', corruptionStrategy: 'truncate' },
|
||||||
|
])
|
||||||
|
const body = result.ctx.response.body as Record<string, unknown>
|
||||||
|
assert.ok(Object.keys(body).length <= 2)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: inbound corruption with field-corrupt', () => {
|
||||||
|
const ctx = makeCtx({ body: { id: '1', name: 'Alice' } })
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'inbound-corruption', target: 'inbound', corruptionStrategy: 'field-corrupt', corruptionField: 'name' },
|
||||||
|
])
|
||||||
|
const body = result.ctx.response.body as Record<string, unknown>
|
||||||
|
assert.strictEqual(body.name, null)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: inbound corruption malformed', () => {
|
||||||
|
const ctx = makeCtx({ body: { id: '1' } })
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'inbound-corruption', target: 'inbound', corruptionStrategy: 'malformed' },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.ctx.response.body, '{"broken":')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: ignores none events', () => {
|
||||||
|
const ctx = makeCtx()
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'none', target: 'inbound' },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.applied, false)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToExecution: only applies first non-delay event', () => {
|
||||||
|
const ctx = makeCtx()
|
||||||
|
const result = applyChaosToExecution(ctx, [
|
||||||
|
{ type: 'inbound-error', target: 'inbound', statusCode: 503 },
|
||||||
|
{ type: 'inbound-dropout', target: 'inbound' },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.ctx.response.statusCode, 503)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToDependencyResponse: outbound error changes status', () => {
|
||||||
|
const response = { contractName: 'api', statusCode: 200, body: { ok: true } }
|
||||||
|
const result = applyChaosToDependencyResponse(response, [
|
||||||
|
{ type: 'outbound-error', target: 'outbound', contractName: 'api', statusCode: 503 },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.statusCode, 503)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToDependencyResponse: ignores events for other contracts', () => {
|
||||||
|
const response = { contractName: 'api', statusCode: 200, body: {} }
|
||||||
|
const result = applyChaosToDependencyResponse(response, [
|
||||||
|
{ type: 'outbound-error', target: 'outbound', contractName: 'other', statusCode: 503 },
|
||||||
|
])
|
||||||
|
assert.strictEqual(result.statusCode, 200)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('applyChaosToAllResponses: applies chaos to multiple responses', () => {
|
||||||
|
const responses = [
|
||||||
|
{ contractName: 'a', statusCode: 200, body: {} },
|
||||||
|
{ contractName: 'b', statusCode: 200, body: {} },
|
||||||
|
]
|
||||||
|
const events = [
|
||||||
|
{ type: 'outbound-error', target: 'outbound', contractName: 'a', statusCode: 503 },
|
||||||
|
{ type: 'outbound-error', target: 'outbound', contractName: 'b', statusCode: 504 },
|
||||||
|
] as ChaosEvent[]
|
||||||
|
const result = applyChaosToAllResponses(responses, events)
|
||||||
|
assert.strictEqual(result[0]!.statusCode, 503)
|
||||||
|
assert.strictEqual(result[1]!.statusCode, 504)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('extractDelays: computes total delay', () => {
|
||||||
|
const events = [
|
||||||
|
{ type: 'inbound-delay', target: 'inbound', delayMs: 100 },
|
||||||
|
{ type: 'outbound-delay', target: 'outbound', delayMs: 200 },
|
||||||
|
] as ChaosEvent[]
|
||||||
|
const result = extractDelays(events)
|
||||||
|
assert.strictEqual(result.totalMs, 300)
|
||||||
|
assert.strictEqual(result.events.length, 2)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('sleep: resolves after specified ms', async () => {
|
||||||
|
const start = Date.now()
|
||||||
|
await sleep(10)
|
||||||
|
const elapsed = Date.now() - start
|
||||||
|
assert.ok(elapsed >= 8 && elapsed < 100)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('hasAppliedChaos: detects applied chaos', () => {
|
||||||
|
assert.strictEqual(hasAppliedChaos([{ type: 'none', target: 'inbound' }]), false)
|
||||||
|
assert.strictEqual(hasAppliedChaos([{ type: 'inbound-error', target: 'inbound' }]), true)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('formatChaosEvents: formats events for diagnostics', () => {
|
||||||
|
const events = [
|
||||||
|
{ type: 'inbound-error', target: 'inbound', statusCode: 503 },
|
||||||
|
] as ChaosEvent[]
|
||||||
|
const formatted = formatChaosEvents(events)
|
||||||
|
assert.ok(formatted.includes('inbound-error'))
|
||||||
|
assert.ok(formatted.includes('503'))
|
||||||
|
})
|
||||||
|
|
||||||
|
test('formatChaosEvents: returns "No chaos applied" for empty events', () => {
|
||||||
|
assert.strictEqual(formatChaosEvents([]), 'No chaos applied')
|
||||||
|
assert.strictEqual(formatChaosEvents([{ type: 'none', target: 'inbound' }]), 'No chaos applied')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('createChaosEventArbitrary: generates deterministic events with seed', () => {
|
||||||
|
const arb = createChaosEventArbitrary(
|
||||||
|
{ probability: 1, delay: { probability: 1, minMs: 100, maxMs: 200 } },
|
||||||
|
[]
|
||||||
|
)
|
||||||
|
const sample1 = fc.sample(arb, { seed: 42, numRuns: 5 })
|
||||||
|
const sample2 = fc.sample(arb, { seed: 42, numRuns: 5 })
|
||||||
|
assert.deepStrictEqual(sample1, sample2)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('createChaosEventArbitrary: returns empty array when no config', () => {
|
||||||
|
const arb = createChaosEventArbitrary(undefined, [])
|
||||||
|
const sample = fc.sample(arb, { numRuns: 10 })
|
||||||
|
assert.ok(sample.every((events) => events.length === 0 || events.every((e) => e.type === 'none')))
|
||||||
|
})
|
||||||
|
|
||||||
|
test('createChaosEventArbitrary: generates outbound events for contracts', () => {
|
||||||
|
const arb = createChaosEventArbitrary(
|
||||||
|
{
|
||||||
|
probability: 1,
|
||||||
|
error: { probability: 1, statusCode: 503 },
|
||||||
|
outbound: [
|
||||||
|
{
|
||||||
|
target: 'api',
|
||||||
|
error: {
|
||||||
|
probability: 1,
|
||||||
|
responses: [{ statusCode: 500 }],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
['api']
|
||||||
|
)
|
||||||
|
const sample = fc.sample(arb, { numRuns: 20 })
|
||||||
|
const hasOutbound = sample.some((events) =>
|
||||||
|
events.some((e) => e.target === 'outbound')
|
||||||
|
)
|
||||||
|
assert.ok(hasOutbound, 'expected some outbound events')
|
||||||
|
})
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Flake Detection Tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test('FlakeDetector: deterministic failure returns high confidence', async () => {
|
||||||
|
const detector = new FlakeDetector({ sameSeedReruns: 1, seedVariations: 2 })
|
||||||
|
const report = await detector.detectFlake(
|
||||||
|
makeTestResult(false),
|
||||||
|
async () => ({ passed: false }),
|
||||||
|
42
|
||||||
|
)
|
||||||
|
assert.strictEqual(report.isFlaky, false)
|
||||||
|
assert.strictEqual(report.confidence, 'high')
|
||||||
|
assert.strictEqual(report.reruns.length, 3)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('FlakeDetector: passing same-seed rerun flags flaky', async () => {
|
||||||
|
const detector = new FlakeDetector({ sameSeedReruns: 1, seedVariations: 0 })
|
||||||
|
let callCount = 0
|
||||||
|
const report = await detector.detectFlake(
|
||||||
|
makeTestResult(false),
|
||||||
|
async () => {
|
||||||
|
callCount++
|
||||||
|
return { passed: callCount === 1 }
|
||||||
|
},
|
||||||
|
42
|
||||||
|
)
|
||||||
|
assert.strictEqual(report.isFlaky, true)
|
||||||
|
assert.strictEqual(report.confidence, 'low')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('FlakeDetector: medium confidence when some pass', async () => {
|
||||||
|
const detector = new FlakeDetector({ sameSeedReruns: 1, seedVariations: 2 })
|
||||||
|
let callCount = 0
|
||||||
|
const report = await detector.detectFlake(
|
||||||
|
makeTestResult(false),
|
||||||
|
async () => {
|
||||||
|
callCount++
|
||||||
|
return { passed: callCount % 2 === 0 }
|
||||||
|
},
|
||||||
|
42
|
||||||
|
)
|
||||||
|
assert.strictEqual(report.isFlaky, true)
|
||||||
|
assert.strictEqual(report.confidence, 'medium')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('FlakeDetector: uses Date.now() when no seed provided', async () => {
|
||||||
|
const detector = new FlakeDetector({ sameSeedReruns: 0, seedVariations: 1 })
|
||||||
|
const report = await detector.detectFlake(
|
||||||
|
makeTestResult(false),
|
||||||
|
async () => ({ passed: false })
|
||||||
|
)
|
||||||
|
assert.strictEqual(report.reruns.length, 1)
|
||||||
|
assert.ok(report.reruns[0]!.seed > 0)
|
||||||
|
})
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Mutation Testing Tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
function makeContract(ensures: string[], requires: string[] = []): RouteContract {
|
||||||
|
return {
|
||||||
|
path: '/items',
|
||||||
|
method: 'POST',
|
||||||
|
category: 'constructor',
|
||||||
|
requires,
|
||||||
|
ensures,
|
||||||
|
invariants: [],
|
||||||
|
regexPatterns: {},
|
||||||
|
validateRuntime: false,
|
||||||
|
schema: {
|
||||||
|
body: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { name: { type: 'string' } },
|
||||||
|
},
|
||||||
|
response: {
|
||||||
|
201: {
|
||||||
|
type: 'object',
|
||||||
|
properties: { id: { type: 'string' } },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test('testMutation: detects flipped operator mutation', async () => {
|
||||||
|
const fastify = Fastify()
|
||||||
|
await fastify.register(swagger)
|
||||||
|
await fastify.register(apophisPlugin)
|
||||||
|
fastify.post('/items', {
|
||||||
|
schema: {
|
||||||
|
body: { type: 'object', properties: { name: { type: 'string' } } },
|
||||||
|
response: { 201: { type: 'object', properties: { id: { type: 'string' } } } },
|
||||||
|
'x-ensures': ['response_code(this) == 201'],
|
||||||
|
},
|
||||||
|
}, async (request, reply) => {
|
||||||
|
reply.status(201)
|
||||||
|
return { id: '1' }
|
||||||
|
})
|
||||||
|
await fastify.ready()
|
||||||
|
|
||||||
|
const mutation: Mutation = {
|
||||||
|
id: 'm1',
|
||||||
|
route: 'POST /items',
|
||||||
|
original: 'response_code(this) == 201',
|
||||||
|
mutated: 'response_code(this) != 201',
|
||||||
|
type: 'flip-operator',
|
||||||
|
}
|
||||||
|
|
||||||
|
const killed = await testMutation(fastify, makeContract(['response_code(this) == 201']), mutation)
|
||||||
|
assert.strictEqual(killed, true)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('runMutationTesting: produces report with score', async () => {
|
||||||
|
const fastify = Fastify()
|
||||||
|
await fastify.register(swagger)
|
||||||
|
await fastify.register(apophisPlugin)
|
||||||
|
fastify.post('/items', {
|
||||||
|
schema: {
|
||||||
|
body: { type: 'object', properties: { name: { type: 'string' } } },
|
||||||
|
response: { 201: { type: 'object', properties: { id: { type: 'string' } } } },
|
||||||
|
'x-ensures': ['response_code(this) == 201'],
|
||||||
|
},
|
||||||
|
}, async (request, reply) => {
|
||||||
|
reply.status(201)
|
||||||
|
return { id: '1' }
|
||||||
|
})
|
||||||
|
await fastify.ready()
|
||||||
|
|
||||||
|
const report = await runMutationTesting(fastify, { runs: 5, seed: 42, maxMutationsPerContract: 3 })
|
||||||
|
assert.ok(typeof report.score === 'number')
|
||||||
|
assert.ok(report.score >= 0 && report.score <= 100)
|
||||||
|
assert.ok(report.mutations.length > 0)
|
||||||
|
assert.ok(report.durationMs > 0)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('runMutationTesting: skips routes without contracts', async () => {
|
||||||
|
const fastify = Fastify()
|
||||||
|
await fastify.register(swagger)
|
||||||
|
await fastify.register(apophisPlugin)
|
||||||
|
fastify.get('/health', {}, async () => 'ok')
|
||||||
|
await fastify.ready()
|
||||||
|
|
||||||
|
const report = await runMutationTesting(fastify, { runs: 5, seed: 42 })
|
||||||
|
assert.strictEqual(report.mutations.length, 0)
|
||||||
|
assert.strictEqual(report.score, 0)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('runMutationTesting: filters by route', async () => {
|
||||||
|
const fastify = Fastify()
|
||||||
|
await fastify.register(swagger)
|
||||||
|
await fastify.register(apophisPlugin)
|
||||||
|
fastify.post('/items', {
|
||||||
|
schema: {
|
||||||
|
body: { type: 'object', properties: { name: { type: 'string' } } },
|
||||||
|
response: { 201: { type: 'object', properties: { id: { type: 'string' } } } },
|
||||||
|
'x-ensures': ['response_code(this) == 201'],
|
||||||
|
},
|
||||||
|
}, async (request, reply) => {
|
||||||
|
reply.status(201)
|
||||||
|
return { id: '1' }
|
||||||
|
})
|
||||||
|
fastify.post('/other', {
|
||||||
|
schema: {
|
||||||
|
body: { type: 'object', properties: { name: { type: 'string' } } },
|
||||||
|
response: { 201: { type: 'object', properties: { id: { type: 'string' } } } },
|
||||||
|
'x-ensures': ['response_code(this) == 201'],
|
||||||
|
},
|
||||||
|
}, async (request, reply) => {
|
||||||
|
reply.status(201)
|
||||||
|
return { id: '2' }
|
||||||
|
})
|
||||||
|
await fastify.ready()
|
||||||
|
|
||||||
|
const report = await runMutationTesting(fastify, { runs: 5, seed: 42, routes: ['/items'] })
|
||||||
|
const itemsMutations = report.mutations.filter((m) => m.mutation.route.includes('/items'))
|
||||||
|
const otherMutations = report.mutations.filter((m) => m.mutation.route.includes('/other'))
|
||||||
|
assert.ok(itemsMutations.length > 0)
|
||||||
|
assert.strictEqual(otherMutations.length, 0)
|
||||||
|
})
|
||||||
|
|
||||||
|
test('runMutationTesting: identifies weak contracts', async () => {
|
||||||
|
const fastify = Fastify()
|
||||||
|
await fastify.register(swagger)
|
||||||
|
await fastify.register(apophisPlugin)
|
||||||
|
fastify.post('/items', {
|
||||||
|
schema: {
|
||||||
|
body: { type: 'object', properties: { name: { type: 'string' } } },
|
||||||
|
response: { 201: { type: 'object', properties: { id: { type: 'string' } } } },
|
||||||
|
'x-ensures': ['response_code(this) == 201', 'response_body(this).id != null'],
|
||||||
|
},
|
||||||
|
}, async (request, reply) => {
|
||||||
|
reply.status(201)
|
||||||
|
return { id: '1' }
|
||||||
|
})
|
||||||
|
await fastify.ready()
|
||||||
|
|
||||||
|
const report = await runMutationTesting(fastify, { runs: 5, seed: 42 })
|
||||||
|
// All mutations should be killed since the handler always returns 201 with id
|
||||||
|
assert.strictEqual(report.weakContracts.length, 0)
|
||||||
|
})
|
||||||
Reference in New Issue
Block a user