chore: crush git history - reborn from consolidation on 2026-03-10

2026-03-10 00:00:00 -07:00
commit d278c4b105
313 changed files with 87549 additions and 0 deletions
@@ -0,0 +1,396 @@
+# Arbiter → Apophis Feedback Report
+
+**Date:** 2026-04-27
+**Reporter:** Arbiter Engineering Team
+**Context:** Integration of Apophis v2.2 into Arbiter Platform for behavioral contract testing
+
+---
+
+## Executive Summary
+
+Apophis provides genuinely valuable capabilities for behavioral contract testing that go beyond traditional unit/integration tests. The schema-to-contract inference, cross-operation verification, and chaos testing infrastructure are compelling. However, we encountered 3 bugs in core infrastructure and several design friction points that should be addressed for wider adoption.
+
+**Overall Assessment:** Strong value proposition for teams willing to invest in schema-driven testing. Needs polish on edge cases and configurability.
+
+---
+
+## Part 1: How Chaos Injection Would Help Arbiter
+
+### Current State
+Arbiter is a multi-tenant SaaS platform with:
+- 500+ API endpoints across 15 route families
+- Billing, graph storage, auth, sessions, webhooks, etc.
+- Mock Stripe integration for payment processing
+- In-memory and persistent storage backends
+- Complex middleware chain: auth → tenant boundary → permissions → preflight → handler
+
+### Where Chaos Testing Adds Value
+
+**1. Middleware Resilience Verification**
+
+Our middleware chain has implicit dependencies:
+```
+Transport → AuthN → Scope → AuthZ → Challenge → Preflight → Handler
+```
+
+Chaos testing would verify:
+- What happens when `preflight()` times out? Does the handler still execute?
+- If auth middleware fails with 503, do we get proper retry headers?
+- Does a slow tenant boundary check cascade to response timeouts?
+
+**Concrete scenario:** If the billing preflight gate (budget check) is slow, does the subscription creation handler wait or fail? Our contracts say `response_time < 2000ms` — chaos would tell us if that's actually enforced.
+
+**2. Mock Service Degradation**
+
+We use `MockStripeService` for payment processing. In production, Stripe can:
+- Return 429 (rate limit)
+- Time out on `paymentIntents.create`
+- Return network errors
+
+Chaos testing would inject:
+```
+if chaos:stripe-timeout then response_code == 503
+if chaos:stripe-rate-limit then retry-after header != null
+```
+
+This validates our fallback logic — currently untested because mocks always succeed.
+
+**3. Resource Leak Detection**
+
+Our `BillingApplicationService` uses in-memory Maps. Chaos scenarios:
+- Create 1000 plans, delete 500, verify GET on deleted returns 404
+- Cancel subscriptions mid-renewal cycle
+- Concurrent PATCH operations on same plan
+
+Cross-operation contracts catch this for single requests, but chaos tests concurrent state corruption.
+
+**4. Entitlement Boundary Testing**
+
+We have credit-based preflight gates. Chaos could:
+- Exhaust credits mid-test
+- Verify 402 (Payment Required) is returned
+- Ensure no partial mutations occur when budget is depleted
+
+This is business-critical: we cannot bill customers for operations that fail.
+
+**5. Auth Token Expiry**
+
+JWT tokens expire. Chaos could:
+- Expire tokens between POST and follow-up GET
+- Verify 401 with proper `WWW-Authenticate` header
+- Test refresh token flow under load
+
+### Proposed Chaos Scenarios for Arbiter
+
+```yaml
+billing_chaos:
+  - name: stripe-timeout
+    target: POST /billing/invoices/:id/pay
+    inject: { stripe_delay_ms: 5000 }
+    expected: { status: 503, retry_after: "> 0" }
+  
+  - name: storage-corruption
+    target: DELETE /billing/plans/:id
+    inject: { skip_deletion: true }
+    expected: { status: 200, follow_up_get: 404 }
+  
+  - name: rate-limit
+    target: POST /billing/plans
+    inject: { rate_limit: 10 }
+    expected: { status: 429, x_retry_after: "> 0" }
+  
+  - name: auth-expiry
+    target: PATCH /billing/plans/:id
+    inject: { expire_token_after_ms: 100 }
+    expected: { status: 401, www_authenticate: "Bearer" }
+```
+
+---
+
+## Part 2: Bugs Found
+
+### Bug 1: Scope Registry Ignores Configured Default Scope
+
+**Severity:** High (breaks auth in cross-operation tests)
+**File:** `dist/infrastructure/scope-registry.js`
+**Line:** 60, 76-77
+
+**Problem:**
+```javascript
+const scope = scopeName !== null ? this.scopes.get(scopeName) : undefined;
+const base = scope ?? this.defaultScope;  // Always uses empty DEFAULT_SCOPE
+```
+
+When `getHeaders(null)` is called, it uses `this.defaultScope` which is initialized to `{ headers: {}, metadata: {} }` on line 60, ignoring any "default" scope passed in the constructor.
+
+**Impact:** Cross-operation requests (e.g., `response_code(GET /users/{id})`) don't inherit auth headers from the configured scope, causing 401 failures on protected routes.
+
+**Fix:**
+```javascript
+const base = scope ?? this.scopes.get('default') ?? this.defaultScope;
+```
+
+**Reproduction:**
+```javascript
+await app.register(apophis, {
+  scopes: {
+    default: { headers: { 'authorization': 'Bearer token' } }
+  }
+});
+// Cross-operation GET /users/123 gets 401 because auth header is not passed
+```
+
+### Bug 2: Contract Builder Drops Routes Option
+
+**Severity:** High (route filtering doesn't work)
+**File:** `dist/plugin/contract-builder.js`
+**Line:** 8-15
+
+**Problem:**
+```javascript
+const config = {
+    depth: opts.depth ?? 'standard',
+    scope: opts.scope,
+    seed: opts.seed,
+    timeout: opts.timeout,
+    chaos: opts.chaos,
+    // Missing: routes: opts.routes
+};
+```
+
+The `routes` option is documented but never passed to `runPetitTests`, causing all routes to be tested regardless of the `routes` filter.
+
+**Impact:** Tests run against all 500+ routes instead of the 4 specified, making debugging impossible and CI times explode.
+
+**Fix:**
+```javascript
+const config = {
+    depth: opts.depth ?? 'standard',
+    scope: opts.scope,
+    seed: opts.seed,
+    timeout: opts.timeout,
+    chaos: opts.chaos,
+    routes: opts.routes,  // Add this
+};
+```
+
+**Reproduction:**
+```javascript
+await app.apophis.contract({
+  routes: ['POST /billing/plans']  // Tests ALL routes instead
+});
+```
+
+### Bug 3: Invariant Checking Not Configurable
+
+**Severity:** Medium (false failures for non-hierarchical APIs)
+**File:** `dist/test/petit-runner.js`
+**Line:** 386-398
+
+**Problem:** Built-in invariants (`no-orphaned-resources`, `parent-reference-integrity`, `resource-integrity`) run unconditionally for all routes. These assume parent-child resource hierarchies (e.g., `/workspaces/:id/projects/:id`).
+
+**Impact:** For flat resource models (like our billing plans), routes with `x-category: 'constructor'` trigger invariant failures because resources don't have `parentType`/`parentId`.
+
+**Workaround:** We set `x-category: 'observer'` to avoid resource tracking, but this loses the semantic meaning of the route.
+
+**Suggested Fix:**
+```javascript
+// In config
+invariants: ['resource-integrity']  // Opt-in per test
+// Or
+invariants: false  // Disable all
+// Or per-route
+schema: {
+  'x-invariants': ['custom-only']
+}
+```
+
+---
+
+## Part 3: Design Feedback
+
+### 1. Schema Inference is Too Aggressive
+
+**Issue:** `const` values in JSON Schema generate unconditional contracts.
+
+Example:
+```json
+{
+  "response": {
+    "200": {
+      "properties": {
+        "fragment_type": { "const": "Action" }
+      }
+    }
+  }
+}
+```
+
+Generates: `response_body(this).fragment_type == "Action"` (checked for ALL responses)
+
+This fails when the route returns 404 with `fragment_type: "Error"`.
+
+**Suggestion:** Infer conditional contracts based on status code:
+```
+if status:200 then response_body(this).fragment_type == "Action" else true
+```
+
+Or add an option to disable inference: `inferContracts: false`.
+
+### 2. Cross-Operation Headers Not Documented
+
+The `scope.headers` behavior for cross-operation requests is not documented. We had to read source code to discover that:
+- `createOperationResolver(fastify, request.headers)` passes request headers
+- But `request.headers` comes from `scope.getHeaders(null)`
+- Which had bug #1 above
+
+**Suggestion:** Document that cross-operation requests inherit the scope headers of the original request.
+
+### 3. Missing 400 Response Handling
+
+When Fastify schema validation fails (e.g., enum mismatch), it returns 400 with a validation error object. Apophis treats this as a contract failure unless:
+- The schema has a 400 response documented
+- The contract explicitly accepts 400
+
+Most developers won't document 400 responses. Apophis should either:
+- Auto-generate 400 contracts from validation rules
+- Or provide a global 400 handler pattern
+
+### 4. HEAD Routes Cause Noise
+
+Fastify auto-generates HEAD routes for every GET. These have no response body, causing `response_body(this).id != null` failures.
+
+**Suggestion:** Auto-skip HEAD routes in contract tests, or provide `skipMethods: ['HEAD']` option.
+
+### 5. Error Suggestions Need Context
+
+When a contract fails, the error is:
+```
+Field 'fragment_type' does not match expected value 'Error'.
+```
+
+But it doesn't say:
+- What the actual status code was
+- What the actual response body was
+- Which route generated the request
+
+**Suggestion:** Include actual vs expected in violation objects.
+
+---
+
+## Part 4: What We Love
+
+### 1. Cross-Operation Contracts
+
+```
+if status:201 then response_code(GET /billing/plans/{response_body(this).data.plan_id}) == 200 else true
+```
+
+This is genuinely hard to test manually. Apophis makes it declarative and automatic.
+
+### 2. Property-Based Generation
+
+Fast-check found edge cases we missed:
+- Empty string `name` (schema allowed it, service rejected it)
+- Invalid `billing_interval` values
+- Missing required fields
+
+### 3. Schema as Single Source of Truth
+
+Once schemas are correct, contracts are free. The `x-ensures` array supplements rather than replaces schema validation.
+
+### 4. Fast Feedback Loop
+
+Contract tests run in ~1.5s for 4 routes. Much faster than spinning up a full test environment.
+
+---
+
+## Part 5: Feature Requests
+
+### 1. Hypermedia Contract Support
+
+Arbiter returns LDF (Linked Data Fragment) responses with `controls` and `actions`. We'd love to verify:
+
+```
+if status:200 then response_body(this).controls.self == request_url(this) else true
+if status:200 then response_body(this).actions.create.method == "POST" else true
+if status:200 then response_body(this).actions.update.target == "/billing/plans/{response_body(this).data.id}" else true
+```
+
+Currently we have to write these manually. Could Apophis infer hypermedia controls from route registration?
+
+### 2. Conditional Schema Contracts
+
+Instead of removing `const` from schemas, allow:
+
+```json
+{
+  "response": {
+    "200": {
+      "properties": {
+        "fragment_type": { "const": "Action", "x-apophis-conditional": "status:200" }
+      }
+    }
+  }
+}
+```
+
+This preserves schema expressiveness while generating correct contracts.
+
+### 3. Middleware Contract Verification
+
+Our middleware chain is critical. We'd like to verify:
+
+```
+if request_headers(this).authorization == null then status:401 else true
+if request_headers(this).x-tenant-id == null then status:400 else true
+```
+
+Apophis already supports `request_headers` — making this a first-class feature (e.g., `x-requires`) would be powerful.
+
+### 4. State Cleanup Hooks
+
+After destructive tests (DELETE), we need to clean up:
+
+```javascript
+await app.apophis.contract({
+  routes: ['DELETE /billing/plans/:id'],
+  cleanup: async (state) => {
+    // Remove created plans from database
+    await db.plans.deleteMany({ id: { $in: state.createdPlans } });
+  }
+});
+```
+
+This would enable stateful testing without polluting the test environment.
+
+### 5. Contract Coverage Report
+
+After running tests, we'd like:
+```
+Contract Coverage:
+  POST /billing/plans:
+    - 201 response: ✓ tested (42 cases)
+    - 400 response: ✓ tested (8 cases)
+    - 503 response: ✗ not tested
+    - Cross-op GET: ✓ tested (42 cases)
+```
+
+This helps identify gaps in contract coverage.
+
+---
+
+## Conclusion
+
+Apophis is a powerful tool that fills a gap in API testing — behavioral contracts and chaos testing. The core concepts are solid, but the implementation needs hardening for production use:
+
+**Must-fix:** Bugs #1 and #2 (scope registry, route filtering)
+**Should-fix:** Bug #3 (configurable invariants), inference aggressiveness
+**Nice-to-have:** Hypermedia support, middleware contracts, coverage reports
+
+We're committed to using Apophis for Arbiter's contract testing and will contribute fixes upstream. The value of cross-operation verification alone justifies the investment.
+
+---
+
+**Contact:** Arbiter Engineering Team
+**Repository:** https://github.com/anomalyco/apophis (we'll open issues for each bug)