Files

1271 lines
37 KiB
Markdown
Raw Permalink Normal View History

# NEXT_STEPS_423.md — Object Inference, Request Structure & Logic/Invariants
## Executive Summary
APOPHIS currently tests routes in isolation with naive resource tracking and no request structure awareness. For Arbiter (11K routes, multi-tenant auth server), we need:
1. **Object Inference**: Schema-driven resource extraction with parent-child relationships
2. **Request Structure**: Path/body/query/header discrimination based on route schemas
3. **Logic/Invariants**: Cross-route temporal assertions, authorization boundaries, state consistency
**Timeline**: 2-3 weeks for core implementation, 1 week for Arbiter-specific invariants
**Impact**: 10-100x more bugs caught, especially authorization leaks and invalid state transitions
---
## 1. OBJECT INFERENCE
### 1.1 Current State (Naive)
**File**: `src/test/petit-runner.ts:205-223`
```typescript
const updateState = (command: ApiCommand, ctx: EvalContext, state: ModelState): ModelState => {
if (command.route.category !== 'constructor') return state
const body = ctx.response.body as Record<string, unknown> | undefined
if (body === undefined) return state
const id = body.id ?? body.uuid ?? body._id
if (id === undefined) return state
const resourceType = command.route.path.split('/').filter(Boolean).pop() ?? 'resource'
// ... stores flat resource
}
```
**Problems**:
- Only looks for `id`/`uuid`/`_id` in response body
- Resource type = last path segment (e.g., `POST /tenant/applications` → type=`applications`)
- No parent tracking (application is scoped to tenant)
- No nested resource support (e.g., `/tenants/:id/applications/:appId/rules`)
- No schema-driven identity field detection
### 1.2 Required: Schema-Driven Resource Extraction
**New File**: `src/domain/resource-inference.ts`
```typescript
interface ResourceIdentity {
resourceType: string
id: string
parentType?: string
parentId?: string
tenantId?: string
applicationId?: string
scope: string | null
}
interface ResourceSchema {
identityField: string // e.g., 'id', 'appId', 'tenantId'
parentField?: string // e.g., 'tenantId' for applications
identityPattern?: string // e.g., '^[a-z]+-[0-9]+$'
scopeFields: string[] // e.g., ['tenantId', 'applicationId']
}
export const extractResourceIdentity = (
route: RouteContract,
responseBody: unknown,
responseSchema?: Record<string, unknown>
): ResourceIdentity | null => {
// 1. Determine identity field from schema
// - Look for field named 'id', 'uuid', '_id', or ending in 'Id'
// - Prefer schema.required fields
// - Use schema.pattern if available for validation
// 2. Determine resource type from route path
// - /tenants/:id/applications → type='application', parent='tenant'
// - /oauth/token → type='token' (special case)
// - /graph/nodes/:nodeId/relations → type='relation', parent='node'
// 3. Extract parent from response body (e.g., body.tenantId)
// or from route path params (e.g., :id in /tenants/:id/applications)
// 4. Extract scope (tenantId, applicationId) from response or request headers
}
export const inferResourceHierarchy = (path: string): {
resourceType: string
parentType?: string
isNested: boolean
} => {
const segments = path.split('/').filter(Boolean)
// e.g., ['tenants', ':id', 'applications', ':appId']
// resourceType = 'application', parentType = 'tenant'
// Special cases:
// /oauth/token → resourceType='token', no parent
// /graph/nodes/:nodeId/relations → resourceType='relation', parentType='node'
// /authz/evaluate → not a resource (utility)
}
```
**File to Modify**: `src/types.ts`
Add to `ModelState`:
```typescript
export interface ResourceHierarchy {
readonly id: string
readonly type: string
readonly parentId?: string
readonly parentType?: string
readonly scope: {
readonly tenantId?: string
readonly applicationId?: string
}
readonly data: unknown
readonly createdAt: number
}
export interface ModelState {
readonly resources: ReadonlyMap<string, ReadonlyMap<string, ResourceHierarchy>>
readonly counters: ReadonlyMap<string, number>
readonly relationships: ReadonlyMap<string, ReadonlyArray<{ from: string; to: string; type: string }>>
}
```
### 1.3 Arbiter-Specific Resource Types
**File**: `src/domain/arbiter-resources.ts` (new)
```typescript
// Arbiter has specific resource hierarchies we need to understand
export const ARBITER_RESOURCE_PATTERNS = {
// Tenant hierarchy
'POST /tenants': { type: 'tenant', identityField: 'id' },
'POST /tenants/:id/applications': {
type: 'application',
identityField: 'id',
parentType: 'tenant',
parentField: 'tenantId'
},
'POST /tenants/:id/users': {
type: 'user',
identityField: 'id',
parentType: 'tenant',
parentField: 'tenantId'
},
// Graph resources
'POST /graph/nodes': { type: 'node', identityField: 'id' },
'POST /graph/nodes/:id/relations': {
type: 'relation',
identityField: 'id',
parentType: 'node',
parentField: 'sourceId'
},
// OAuth tokens
'POST /oauth/token': { type: 'token', identityField: 'access_token' },
'POST /oauth/refresh': { type: 'token', identityField: 'access_token' },
// Sessions
'POST /sessions': { type: 'session', identityField: 'id' },
// Permissions
'POST /authz/grants': { type: 'grant', identityField: 'id' },
// Rules
'POST /tenants/:id/applications/:appId/rules': {
type: 'rule',
identityField: 'id',
parentType: 'application',
parentField: 'applicationId'
}
} as const
export const isArbiterResource = (method: string, path: string): boolean => {
const key = `${method} ${path}`
return key in ARBITER_RESOURCE_PATTERNS
}
```
### 1.4 Enhanced State Updates
**File**: `src/test/petit-runner.ts` — Replace `updateState` and `makeResource`
```typescript
const updateState = (command: ApiCommand, ctx: EvalContext, state: ModelState): ModelState => {
if (command.route.category !== 'constructor') return state
const body = ctx.response.body as Record<string, unknown> | undefined
if (body === undefined) return state
const identity = extractResourceIdentity(command.route, body, command.route.schema?.response as Record<string, unknown>)
if (!identity) return state
const hierarchy: ResourceHierarchy = {
id: identity.id,
type: identity.resourceType,
parentId: identity.parentId,
parentType: identity.parentType,
scope: {
tenantId: identity.tenantId,
applicationId: identity.applicationId
},
data: body,
createdAt: Date.now()
}
// Store in typed resource map
const existing = state.resources.get(identity.resourceType) ?? new Map<string, ResourceHierarchy>()
const updated = new Map(existing)
updated.set(identity.id, hierarchy)
const newResources = new Map(state.resources)
newResources.set(identity.resourceType, updated)
// Track relationships if present
let newRelationships = state.relationships
if (identity.parentId && identity.parentType) {
const rels = state.relationships.get(identity.resourceType) ?? []
newRelationships = new Map(state.relationships)
newRelationships.set(identity.resourceType, [
...rels,
{ from: identity.id, to: identity.parentId, type: 'childOf' }
])
}
return { ...state, resources: newResources, relationships: newRelationships }
}
```
### 1.5 Test Cases
**New File**: `src/test/resource-inference.test.ts`
```typescript
test('extracts tenant resource from POST /tenants', () => {
const route = makeRoute('POST', '/tenants')
const body = { id: 'tenant-123', name: 'Acme' }
const identity = extractResourceIdentity(route, body)
assert.strictEqual(identity?.resourceType, 'tenant')
assert.strictEqual(identity?.id, 'tenant-123')
assert.strictEqual(identity?.parentType, undefined)
})
test('extracts nested application with parent tenant', () => {
const route = makeRoute('POST', '/tenants/:id/applications')
const body = { id: 'app-456', tenantId: 'tenant-123', name: 'My App' }
const identity = extractResourceIdentity(route, body)
assert.strictEqual(identity?.resourceType, 'application')
assert.strictEqual(identity?.id, 'app-456')
assert.strictEqual(identity?.parentType, 'tenant')
assert.strictEqual(identity?.parentId, 'tenant-123')
})
test('extracts OAuth token with access_token identity', () => {
const route = makeRoute('POST', '/oauth/token')
const body = { access_token: 'tok-789', token_type: 'Bearer' }
const identity = extractResourceIdentity(route, body)
assert.strictEqual(identity?.resourceType, 'token')
assert.strictEqual(identity?.id, 'tok-789')
})
test('returns null for non-resource routes', () => {
const route = makeRoute('GET', '/health')
const body = { status: 'ok' }
const identity = extractResourceIdentity(route, body)
assert.strictEqual(identity, null)
})
test('uses schema to find identity field', () => {
const route = makeRoute('POST', '/custom/resources', {
response: {
type: 'object',
properties: {
resourceId: { type: 'string' },
name: { type: 'string' }
},
required: ['resourceId']
}
})
const body = { resourceId: 'res-999', name: 'Test' }
const identity = extractResourceIdentity(route, body, route.schema?.response as Record<string, unknown>)
assert.strictEqual(identity?.id, 'res-999')
})
```
---
## 2. REQUEST STRUCTURE INFERENCE
### 2.1 Current State (Blind Parameter Passing)
**File**: `src/test/petit-runner.ts:111-168`
```typescript
const executeCommand = async (fastify: FastifyInstance, command: ApiCommand): Promise<EvalContext> => {
const method = command.route.method
let url = command.route.path
// Replace path params with generated values
const params = command.params
for (const [key, value] of Object.entries(params)) {
if (url.includes(`:${key}`)) {
url = url.replace(`:${key}`, String(value))
}
}
// Everything else goes to query (GET/DELETE) or body (others)
const queryParams: Record<string, string> = {}
const bodyParams: Record<string, unknown> = {}
for (const [key, value] of Object.entries(params)) {
if (!command.route.path.includes(`:${key}`)) {
if (method === 'GET' || method === 'DELETE') {
queryParams[key] = String(value)
} else {
bodyParams[key] = value
}
}
}
// ...
}
```
**Problems**:
- Assumes ALL non-path params are body params for POST/PUT/PATCH
- No understanding of `body.properties` from schema
- No handling of nested body structures (`body.nested.field`)
- No automatic header injection (x-tenant-id, authorization)
- No content-type negotiation
- Query params for GET/DELETE are just dumped as-is
### 2.2 Required: Schema-Aware Request Building
**New File**: `src/domain/request-builder.ts`
```typescript
interface RequestStructure {
method: string
url: string // With path params replaced
headers: Record<string, string>
query?: Record<string, string>
body?: unknown
contentType?: string
}
interface RouteParamSchema {
pathParams: string[] // e.g., ['tenantId', 'appId']
bodySchema?: Record<string, unknown>
querySchema?: Record<string, unknown>
headerRequirements: string[] // e.g., ['x-tenant-id', 'authorization']
}
export const parseRouteParams = (path: string): string[] => {
const params: string[] = []
const segments = path.split('/')
for (const segment of segments) {
if (segment.startsWith(':')) {
params.push(segment.slice(1))
} else if (segment.startsWith('{') && segment.endsWith('}')) {
params.push(segment.slice(1, -1))
}
}
return params
}
export const buildRequest = (
route: RouteContract,
generatedData: Record<string, unknown>,
scopeHeaders: Record<string, string>,
state: ModelState
): RequestStructure => {
const pathParams = parseRouteParams(route.path)
const url = substitutePathParams(route.path, generatedData, state)
// Extract body params from schema
const bodySchema = route.schema?.body as Record<string, unknown> | undefined
const body = bodySchema
? extractBodyParams(generatedData, bodySchema)
: undefined
// Extract query params from schema
const querySchema = route.schema?.querystring as Record<string, unknown> | undefined
const query = querySchema
? extractQueryParams(generatedData, querySchema)
: extractRemainingParams(generatedData, pathParams, body)
// Build headers
const headers = buildHeaders(route, scopeHeaders, generatedData, state)
// Determine content type
const contentType = body ? 'application/json' : undefined
return { method: route.method, url, headers, query, body, contentType }
}
const substitutePathParams = (
path: string,
data: Record<string, unknown>,
state: ModelState
): string => {
let url = path
const pathParams = parseRouteParams(path)
for (const param of pathParams) {
let value = data[param]
// If param is an ID reference, try to find it in state
if (value === undefined && param.endsWith('Id')) {
const resourceType = param.replace(/Id$/, '')
const resources = state.resources.get(resourceType)
if (resources && resources.size > 0) {
// Pick a random existing resource
const ids = Array.from(resources.keys())
value = ids[Math.floor(Math.random() * ids.length)]
}
}
if (value !== undefined) {
url = url.replace(`:${param}`, String(value))
}
}
return url
}
const extractBodyParams = (
data: Record<string, unknown>,
bodySchema: Record<string, unknown>
): Record<string, unknown> => {
const properties = bodySchema.properties as Record<string, Record<string, unknown>> | undefined
if (!properties) return data
const body: Record<string, unknown> = {}
for (const key of Object.keys(properties)) {
if (key in data) {
body[key] = data[key]
}
}
// Handle nested objects
for (const [key, propSchema] of Object.entries(properties)) {
if (propSchema.type === 'object' && propSchema.properties) {
body[key] = extractBodyParams(data, propSchema)
}
}
return body
}
const buildHeaders = (
route: RouteContract,
scopeHeaders: Record<string, string>,
data: Record<string, unknown>,
state: ModelState
): Record<string, string> => {
const headers: Record<string, string> = { ...scopeHeaders }
// Auto-inject tenant ID if route requires it
if (route.requires.some(r => r.includes('x-tenant-id'))) {
const tenantId = data['tenantId'] || scopeHeaders['x-tenant-id']
if (tenantId) {
headers['x-tenant-id'] = String(tenantId)
}
}
// Auto-inject authorization if required
if (route.requires.some(r => r.includes('authorization'))) {
// Could look up session/token from state
const tokens = state.resources.get('token')
if (tokens && tokens.size > 0) {
const token = Array.from(tokens.values())[0]
headers['authorization'] = `Bearer ${token.id}`
}
}
// Content-Type for body requests
if (route.schema?.body) {
headers['content-type'] = 'application/json'
}
return headers
}
```
### 2.3 Enhanced Execution
**File**: `src/test/petit-runner.ts` — Replace `executeCommand`
```typescript
const executeCommand = async (
fastify: FastifyInstance,
command: ApiCommand,
state: ModelState
): Promise<EvalContext> => {
const request = buildRequest(command.route, command.params, command.headers, state)
// Build query string
const queryString = request.query
? Object.entries(request.query)
.map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`)
.join('&')
: ''
const fullUrl = queryString ? `${request.url}?${queryString}` : request.url
const response = await fastify.inject({
method: request.method,
url: fullUrl,
payload: request.body,
headers: request.headers
})
return {
request: {
body: request.body,
headers: request.headers,
query: request.query || {},
params: extractPathParams(command.route.path, request.url)
},
response: {
body: response.json(),
headers: Object.fromEntries(
Object.entries(response.headers).map(([k, v]) => [k, String(v)])
),
statusCode: response.statusCode
}
}
}
```
### 2.4 Arbiter-Specific Request Patterns
**File**: `src/domain/arbiter-requests.ts` (new)
```typescript
// Arbiter has specific request structure patterns
export const ARBITER_REQUEST_PATTERNS = {
'POST /oauth/token': {
bodyFields: ['grant_type', 'code', 'refresh_token', 'client_id', 'client_secret'],
headerFields: ['authorization'],
contentType: 'application/x-www-form-urlencoded'
},
'POST /oauth/authorize': {
queryFields: ['client_id', 'response_type', 'redirect_uri', 'scope', 'state'],
headerFields: ['cookie']
},
'POST /tenant/applications': {
bodyFields: ['name', 'slug', 'description', 'callbackURLs', 'allowedOrigins'],
headerFields: ['x-tenant-id', 'authorization'],
pathParams: [] // No path params, tenant from header
},
'GET /tenant/applications/:appId': {
pathParams: ['appId'],
headerFields: ['x-tenant-id', 'authorization']
},
'POST /tenants/:tenantId/applications/:appId/rules': {
pathParams: ['tenantId', 'appId'],
bodyFields: ['dsl', 'priority', 'enabled'],
headerFields: ['x-tenant-id', 'authorization']
},
'POST /graph/nodes/:nodeId/relations': {
pathParams: ['nodeId'],
bodyFields: ['targetId', 'relationType', 'metadata'],
headerFields: ['x-tenant-id', 'authorization']
},
'POST /authz/evaluate': {
bodyFields: ['userId', 'resourceId', 'permission', 'context'],
headerFields: ['x-tenant-id', 'authorization']
}
} as const
export const getArbiterRequestPattern = (method: string, path: string) => {
const key = `${method} ${path}`
return ARBITER_REQUEST_PATTERNS[key as keyof typeof ARBITER_REQUEST_PATTERNS]
}
```
### 2.5 Test Cases
**New File**: `src/test/request-builder.test.ts`
```typescript
test('builds request with path params substituted', () => {
const route = makeRoute('GET', '/users/:id')
const data = { id: 'user-123' }
const request = buildRequest(route, data, {}, emptyState())
assert.strictEqual(request.url, '/users/user-123')
})
test('builds request with body from schema', () => {
const route = makeRoute('POST', '/users', {
body: {
type: 'object',
properties: {
name: { type: 'string' },
email: { type: 'string' }
}
}
})
const data = { name: 'John', email: 'john@example.com', extra: 'ignored' }
const request = buildRequest(route, data, {}, emptyState())
assert.deepStrictEqual(request.body, { name: 'John', email: 'john@example.com' })
assert.strictEqual(request.body?.extra, undefined)
})
test('injects tenant ID from scope headers', () => {
const route = makeRoute('GET', '/tenant/applications', {
'x-requires': ['request_headers(this).x-tenant-id != null']
})
const scopeHeaders = { 'x-tenant-id': 'tenant-123' }
const request = buildRequest(route, {}, scopeHeaders, emptyState())
assert.strictEqual(request.headers['x-tenant-id'], 'tenant-123')
})
test('looks up path param from state if not in data', () => {
const route = makeRoute('GET', '/users/:userId')
const state = stateWithResource('user', 'user-456', {})
const request = buildRequest(route, {}, {}, state)
assert.strictEqual(request.url, '/users/user-456')
})
test('handles OAuth token request with form encoding', () => {
const route = makeRoute('POST', '/oauth/token')
const data = {
grant_type: 'authorization_code',
code: 'auth-code-123',
client_id: 'client-456'
}
const request = buildRequest(route, data, {}, emptyState())
assert.strictEqual(request.headers['content-type'], 'application/x-www-form-urlencoded')
assert.strictEqual(request.body, undefined) // Form data handled differently
})
```
---
## 3. LOGIC & INVARIANTS
### 3.1 Current State (Status Code Only)
**File**: `src/test/petit-runner.ts:186-199`
```typescript
const checkPostconditions = (command: ApiCommand, ctx: EvalContext): EvalResult => {
for (const ensure of command.route.ensures) {
if (ensure.startsWith('status:')) {
const expected = parseInt(ensure.replace('status:', ''), 10)
if (ctx.response.statusCode !== expected) {
return { success: false, error: `Expected status ${expected}, got ${ctx.response.statusCode}` }
}
}
}
return { success: true, value: ctx.response.statusCode }
}
```
**Problems**:
- Only checks `status:###` patterns
- Ignores actual APOSTL formulas in `x-ensures`
- No cross-route invariant checking
- No temporal logic (what happens after state changes)
- No authorization boundary verification
### 3.2 Required: Full Formula Evaluation
**File**: `src/test/petit-runner.ts` — Replace `checkPostconditions`
```typescript
import { parse } from '../formula/parser.js'
import { evaluateBooleanResult } from '../formula/evaluator.js'
const checkPostconditions = (command: ApiCommand, ctx: EvalContext): EvalResult => {
for (const ensure of command.route.ensures) {
// Legacy status check
if (ensure.startsWith('status:')) {
const expected = parseInt(ensure.replace('status:', ''), 10)
if (ctx.response.statusCode !== expected) {
return {
success: false,
error: `Expected status ${expected}, got ${ctx.response.statusCode}`
}
}
continue
}
// Full APOSTL formula evaluation
try {
const ast = parse(ensure)
const result = evaluateBooleanResult(ast.ast, ctx)
if (!result) {
return {
success: false,
error: `Contract violation: ${ensure}`
}
}
} catch (err) {
return {
success: false,
error: `Formula error in "${ensure}": ${err instanceof Error ? err.message : String(err)}`
}
}
}
return { success: true, value: ctx.response.statusCode }
}
```
### 3.3 Required: Cross-Route Invariant Registry
**New File**: `src/domain/invariant-registry.ts`
```typescript
interface Invariant {
readonly name: string
readonly description: string
readonly check: (state: ModelState, history: ReadonlyArray<EvalContext>) => InvariantResult
}
interface InvariantResult {
readonly success: boolean
readonly error?: string
}
// Built-in invariants
export const BUILTIN_INVARIANTS: Invariant[] = [
{
name: 'resource-consistency',
description: 'Created resources must be retrievable',
check: (state, history) => {
// For each constructor in history, check that GET returns same data
const constructors = history.filter((ctx, i) => {
// Would need to track which history entries are constructors
// This requires enhancing the history tracking
return false
})
return { success: true }
}
},
{
name: 'tenant-isolation',
description: 'Resources from tenant A must not be accessible in tenant B',
check: (state) => {
for (const [type, resources] of state.resources) {
for (const [id, resource] of resources) {
if (resource.scope.tenantId) {
// Check no other tenant has access
// Would need to simulate cross-tenant requests
}
}
}
return { success: true }
}
},
{
name: 'authorization-transitivity',
description: 'If parent has permission, child must inherit it',
check: (state) => {
// For graph relations: if node A -> node B, then B inherits A's permissions
const relations = state.relationships.get('relation') || []
// Would need to check authorization evaluations
return { success: true }
}
}
]
// Arbiter-specific invariants
export const ARBITER_INVARIANTS: Invariant[] = [
{
name: 'oauth-token-validity',
description: 'Issued tokens must be valid until revoked',
check: (state, history) => {
const tokens = state.resources.get('token') || new Map()
// Check each token was issued properly and hasn't expired
return { success: true }
}
},
{
name: 'session-consistency',
description: 'Active sessions must have valid users',
check: (state) => {
const sessions = state.resources.get('session') || new Map()
const users = state.resources.get('user') || new Map()
for (const [sessionId, session] of sessions) {
const userId = (session.data as Record<string, unknown>)?.userId
if (userId && !users.has(String(userId))) {
return {
success: false,
error: `Session ${sessionId} references non-existent user ${userId}`
}
}
}
return { success: true }
}
},
{
name: 'permission-graph-acyclic',
description: 'Permission inheritance graph must not have cycles',
check: (state) => {
// Detect cycles in relation graph
const relations = state.relationships.get('relation') || []
// Run cycle detection algorithm
return { success: true }
}
},
{
name: 'tenant-application-consistency',
description: 'Applications must belong to existing tenants',
check: (state) => {
const tenants = state.resources.get('tenant') || new Map()
const applications = state.resources.get('application') || new Map()
for (const [appId, app] of applications) {
const tenantId = app.parentId
if (tenantId && !tenants.has(tenantId)) {
return {
success: false,
error: `Application ${appId} references non-existent tenant ${tenantId}`
}
}
}
return { success: true }
}
}
]
export const checkInvariants = (
invariants: ReadonlyArray<Invariant>,
state: ModelState,
history: ReadonlyArray<EvalContext>
): Array<{ name: string; result: InvariantResult }> => {
return invariants.map(inv => ({
name: inv.name,
result: inv.check(state, history)
}))
}
```
### 3.4 Required: Temporal Logic in APOSTL
**File**: `src/formula/parser.ts` — Extend grammar
Add temporal operators:
```typescript
// New node types for temporal logic
export type FormulaNode =
| ...existing nodes...
| { type: 'temporal'; operator: 'eventually' | 'always' | 'until'; body: FormulaNode }
| { type: 'previous'; inner: FormulaNode }
```
**File**: `src/formula/evaluator.ts` — Extend evaluation
```typescript
// Temporal evaluation requires history context
function evaluateTemporal(
operator: string,
body: FormulaNode,
ctx: EvalContext,
history: ReadonlyArray<EvalContext>
): boolean {
switch (operator) {
case 'eventually':
// True if body is true at some point in future
// For testing: check if body will be true after some operation
return true // Placeholder
case 'always':
// True if body is true at all points
return history.every(h => evaluateNode(body, h))
case 'until':
// True if left is true until right becomes true
// Requires binary temporal operator
return true // Placeholder
default:
throw new Error(`Unknown temporal operator: ${operator}`)
}
}
```
### 3.5 Required: Stateful Test Runner Enhancement
**File**: `src/test/petit-runner.ts` — Add invariant checking to main loop
```typescript
export const runPetitTests = async (
fastify: FastifyInstance,
config: TestConfig
): Promise<TestSuite> => {
const startTime = Date.now()
const depth = DEPTH_CONFIGS[config.depth]
const allRoutes = discoverRoutes(fastify)
const filtered = filterByMode(allRoutes, config.mode)
const routes = sortByCategory(filtered)
const { commands: commandGroups, cacheHits, cacheMisses } = generateCommands(routes, depth, config.seed)
const allCommands = commandGroups.flat()
let state: ModelState = {
resources: new Map(),
counters: new Map(),
relationships: new Map()
}
const resources: TrackedResource[] = []
const results: TestResult[] = []
const history: EvalContext[] = [] // Track all request/response contexts
let testId = 0
for (const command of allCommands) {
testId++
const name = `${command.route.method} ${command.route.path} (#${testId})`
// Check preconditions
const preOk = checkPreconditions(command, state)
if (!preOk) {
results.push({
ok: true,
name,
id: testId,
directive: 'SKIP preconditions not met'
})
continue
}
// Execute command
let ctx: EvalContext
try {
ctx = await executeCommand(fastify, command, state)
history.push(ctx)
} catch (err) {
results.push({
ok: false,
name,
id: testId,
diagnostics: { error: err instanceof Error ? err.message : String(err) }
})
continue
}
// Check postconditions
const post = checkPostconditions(command, ctx)
if (!post.success) {
results.push({
ok: false,
name,
id: testId,
diagnostics: { error: post.error, statusCode: ctx.response.statusCode }
})
} else {
results.push({ ok: true, name, id: testId })
}
// Update state
state = updateState(command, ctx, state)
// Check invariants after state change
const invariantResults = checkInvariants(ARBITER_INVARIANTS, state, history)
for (const inv of invariantResults) {
if (!inv.result.success) {
results.push({
ok: false,
name: `INVARIANT: ${inv.name} (#${testId})`,
id: testId,
diagnostics: { error: inv.result.error }
})
}
}
const resource = makeResource(command, ctx)
if (resource !== null) {
resources.push(resource)
}
}
// Final invariant check
const finalInvariantResults = checkInvariants(BUILTIN_INVARIANTS, state, history)
for (const inv of finalInvariantResults) {
if (!inv.result.success) {
results.push({
ok: false,
name: `FINAL INVARIANT: ${inv.name}`,
id: testId + 1,
diagnostics: { error: inv.result.error }
})
}
}
const passed = results.filter((r) => r.ok && r.directive === undefined).length
const failed = results.filter((r) => !r.ok).length
const skipped = results.filter((r) => r.directive !== undefined).length
return {
version: 13,
plan: { start: 1, end: results.length },
tests: results,
summary: { passed, failed, skipped, timeMs: Date.now() - startTime, cacheHits, cacheMisses }
}
}
```
### 3.6 Arbiter-Specific Contract Examples
**Example contracts for Arbiter routes**:
```javascript
// Tenant isolation
{
'x-requires': [
'request_headers(this).x-tenant-id != null',
'request_headers(this).authorization matches "^Bearer .+"'
],
'x-ensures': [
'response_code(this) == 200',
'response_body(this).tenantId == request_headers(this).x-tenant-id',
'if request_headers(this).x-tenant-id != response_body(this).tenantId then response_code(this) == 403 else T'
],
'x-invariants': [
'response_body(this).resources.all(r => r.tenantId == request_headers(this).x-tenant-id)'
]
}
// OAuth authorization
{
'x-requires': [
'query_params(this).client_id != null',
'query_params(this).response_type == "code"',
'request_headers(this).cookie != null'
],
'x-ensures': [
'response_code(this) == 302',
'response_headers(this).location matches "\\?code="'
]
}
// Token issuance
{
'x-requires': [
'request_body(this).grant_type in ["authorization_code", "refresh_token"]',
'request_body(this).code != null || request_body(this).refresh_token != null'
],
'x-ensures': [
'response_code(this) == 200',
'response_body(this).access_token != null',
'response_body(this).token_type == "Bearer"',
'response_body(this).expires_in > 0'
]
}
// Graph authorization evaluation
{
'x-requires': [
'request_body(this).userId != null',
'request_body(this).resourceId != null',
'request_body(this).permission != null'
],
'x-ensures': [
'response_code(this) == 200',
'response_body(this).allowed == true || response_body(this).allowed == false',
'if response_body(this).allowed == true then response_body(this).reason != null else T'
]
}
// Rule creation
{
'x-requires': [
'request_headers(this).x-tenant-id != null',
'request_body(this).dsl != null',
'request_body(this).priority >= 0'
],
'x-ensures': [
'response_code(this) == 201',
'response_body(this).id != null',
'response_body(this).dsl == request_body(this).dsl',
'response_body(this).tenantId == request_headers(this).x-tenant-id'
]
}
```
---
## 4. IMPLEMENTATION ROADMAP
### Week 1: Object Inference & Request Structure
**Day 1-2**: Resource Inference
- [ ] Create `src/domain/resource-inference.ts`
- [ ] Implement `extractResourceIdentity()`
- [ ] Implement `inferResourceHierarchy()`
- [ ] Add `ResourceHierarchy` to types
- [ ] Write tests (10+ cases)
**Day 3-4**: Request Builder
- [ ] Create `src/domain/request-builder.ts`
- [ ] Implement `buildRequest()`
- [ ] Implement `substitutePathParams()`
- [ ] Implement `extractBodyParams()`
- [ ] Implement `buildHeaders()`
- [ ] Write tests (10+ cases)
**Day 5**: Integration
- [ ] Update `petit-runner.ts` to use new modules
- [ ] Update `executeCommand()` signature
- [ ] Update `updateState()` for hierarchies
- [ ] Run full test suite
- [ ] Fix regressions
### Week 2: Logic & Invariants
**Day 1-2**: Formula Evaluation in Tests
- [ ] Update `checkPostconditions()` to use formula evaluator
- [ ] Handle parse errors gracefully
- [ ] Add formula error diagnostics
- [ ] Write tests for formula-based postconditions
**Day 3-4**: Invariant Registry
- [ ] Create `src/domain/invariant-registry.ts`
- [ ] Implement builtin invariants
- [ ] Implement Arbiter-specific invariants
- [ ] Add invariant checking to test runner loop
- [ ] Write tests for invariant detection
**Day 5**: Temporal Logic (Basic)
- [ ] Add `previous()` operator support
- [ ] Implement basic history tracking
- [ ] Add `always` temporal operator
- [ ] Write tests for temporal assertions
### Week 3: Stateful Testing & Polish
**Day 1-2**: Command Sequences
- [ ] Implement fast-check `commands()` arbitrary
- [ ] Generate valid command sequences respecting preconditions
- [ ] Add sequence shrinkers
- [ ] Write property tests for sequences
**Day 3-4**: Arbiter Integration
- [ ] Create Arbiter-specific resource patterns
- [ ] Create Arbiter request patterns
- [ ] Add Arbiter-specific invariants
- [ ] Test against Arbiter route samples
**Day 5**: Documentation & Performance
- [ ] Update README with advanced examples
- [ ] Update SKILL.md with new patterns
- [ ] Profile performance with 1000+ routes
- [ ] Optimize hot paths
- [ ] Final test suite: 250+ tests
---
## 5. TEST COVERAGE TARGETS
### Current: 198 tests
### Target after implementation:
- **Resource inference**: +30 tests
- **Request builder**: +25 tests
- **Formula evaluation in tests**: +20 tests
- **Invariant registry**: +25 tests
- **Temporal logic**: +15 tests
- **Stateful sequences**: +20 tests
- **Arbiter-specific**: +20 tests
- **Total**: ~353 tests
---
## 6. FILES TO CREATE/MODIFY
### New Files
```
src/domain/resource-inference.ts
src/domain/request-builder.ts
src/domain/invariant-registry.ts
src/domain/arbiter-resources.ts
src/domain/arbiter-requests.ts
src/test/resource-inference.test.ts
src/test/request-builder.test.ts
src/test/invariant-registry.test.ts
src/test/temporal-logic.test.ts
```
### Modified Files
```
src/types.ts — Add ResourceHierarchy, update ModelState
src/test/petit-runner.ts — Use resource inference, request builder, invariants
src/formula/parser.ts — Add temporal operators (if needed)
src/formula/evaluator.ts — Add history context support
src/domain/category.ts — Add Arbiter-specific category rules
```
---
## 7. ACCEPTANCE CRITERIA
### Object Inference
- [ ] Can extract resource identity from response body using schema
- [ ] Can determine parent-child relationships from route paths
- [ ] Can handle OAuth tokens, sessions, graph nodes/relations
- [ ] Returns null for non-resource routes (health, utility)
- [ ] All Arbiter resource types are correctly identified
### Request Structure
- [ ] Path params are substituted from generated data or state
- [ ] Body params match schema properties (no extra fields)
- [ ] Query params are extracted for GET/DELETE
- [ ] Headers include x-tenant-id from scope
- [ ] Authorization header is injected when required
- [ ] Content-Type is set correctly (JSON vs form-encoded)
### Logic & Invariants
- [ ] APOSTL formulas in x-ensures are evaluated (not just status:###)
- [ ] Cross-route invariants are checked after each state change
- [ ] Tenant isolation is verified
- [ ] Resource consistency is verified (created resources are retrievable)
- [ ] Authorization transitivity is checked
- [ ] Session consistency is verified
- [ ] Temporal operators work (previous, always)
### Performance
- [ ] < 2s overhead for 1000 routes
- [ ] < 5s overhead for 10,000 routes
- [ ] Incremental cache still provides 10x+ speedup
- [ ] Memory usage < 500MB for full Arbiter test run
---
## 8. RISKS & MITIGATIONS
| Risk | Impact | Mitigation |
|------|--------|------------|
| Formula evaluation too slow | High | Cache parsed ASTs, use WeakMap |
| Invariant checking O(n²) | Medium | Batch checks, use Set lookups |
| Memory leak in history | Medium | Limit history size, use ring buffer |
| Arbiter routes lack schemas | High | Fallback to path-based inference |
| Fastify v5 compatibility | Low | Already using inject() API |
| Breaking changes to existing tests | Medium | Maintain backward compatibility for status:### |
---
**Next Action**: Start with `src/domain/resource-inference.ts` — this is the foundation everything else builds on.