John Dvorak
1652 lines
58 KiB
Markdown
1652 lines
58 KiB
Markdown
# NEXT_STEPS_425.md — Post-v1.1 Integration Feedback & Priorities
|
|
|
|
## Status: v1.3 Complete (2026-04-25)
|
|
|
|
**Test count**: 551 passing, 0 failures
|
|
**New in v1.3**: All 8 protocol extensions (JWT, Time, Stateful, X.509, SPIFFE, Token Hash, HTTP Signature, Request Context), Plugin Contract System with lazy extension resolution, built-in plugin contracts for `@fastify/auth`, `@fastify/cors`, `@fastify/compress`, `@fastify/rate-limit`
|
|
|
|
## Completed
|
|
|
|
### v1.2 (2026-04-25)
|
|
|
|
- [x] **F1**: APOSTL `else` is optional — defaults to `else T`
|
|
- [x] **F2**: Value proposition comparison table in README and skills.md
|
|
- [x] **F3**: Auth extension factory — `createAuthExtension()`
|
|
- [x] **F4**: ContractViolation includes full request/response context
|
|
- [x] **F5**: Fastify App Structure Guide
|
|
- [x] **Chaos Mode**: Content-type aware corruption with extension strategies (21 tests)
|
|
|
|
### v1.3 (2026-04-25)
|
|
|
|
- [x] **P0.1**: JWT Extension — claims, headers, format detection, Base64URL decode, `seen_jtis` tracking
|
|
- [x] **P0.2**: Time Control — `now()` predicate + `createTimeControl().advance()` API
|
|
- [x] **P1.1**: Stateful predicates — `already_seen()`, `is_consumed()`, `previous(category)`
|
|
- [x] **P1.2**: X.509 Extension — URI SANs, CA check, expiration, self-signed, issuer, subject
|
|
- [x] **P2.1**: SPIFFE Extension — trust domain, path, validation
|
|
- [x] **P2.2**: Token Hash Extension — `ath_valid()`, `tth_valid()`, `token_hash()`
|
|
- [x] **P2.3**: HTTP Signature Extension — `signature_input()`, `signature_covers()`
|
|
- [x] **P2.4**: Request Context — `request_url()`, `request_tls()`, `request_body_hash()`
|
|
- [x] **Plugin Contract System** — Registry, pattern matching, composition, lazy extension resolution
|
|
- [x] **Built-in Plugin Contracts** — `@fastify/auth`, `@fastify/cors`, `@fastify/compress`, `@fastify/rate-limit`
|
|
- [x] **Extension Registry Link** — `setPluginContractRegistry()` notifies on extension registration
|
|
- [x] **Runner Integration** — Plugin contracts composed with route contracts in `petit-runner.ts`
|
|
|
|
## Expert Assessment Remediation Plan
|
|
|
|
### Chaos Engineering (Critical — Grade: F)
|
|
|
|
#### Issue C1: Two-Level Probability Bug
|
|
**Location**: `src/quality/chaos.ts:55, 82`
|
|
**Problem**: Global gate at line 55 applies `config.probability`, then `pickEventType()` at line 82 applies per-event probability. Actual injection rate = `config.probability * eventProbability`, not `eventProbability`.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
FUNCTION executeWithChaos(executeHttp, route, request, extensionRegistry):
|
|
assertTestEnv('Chaos mode')
|
|
this.events = []
|
|
|
|
// Pick event type using weighted probabilities (no global gate)
|
|
eventType = this.pickEventType()
|
|
|
|
IF eventType IS NULL:
|
|
ctx = await executeHttp()
|
|
RETURN { ctx, events: [] }
|
|
|
|
// Apply the selected event directly
|
|
SWITCH eventType:
|
|
CASE 'delay': RETURN this.injectDelay(executeHttp)
|
|
CASE 'dropout': RETURN this.injectDropout(route, request)
|
|
CASE 'error': RETURN this.injectError(executeHttp, route, request)
|
|
CASE 'corruption': RETURN this.injectCorruption(executeHttp, route, request, extensionRegistry)
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: The probability of event type X being injected MUST equal `config[X].probability / sum(all configured probabilities)`
|
|
- MUST: The `pickEventType()` function MUST be the sole probability gate; no secondary filtering
|
|
- MAY NEVER: A global probability gate AND per-event probability multiply
|
|
|
|
#### Issue C2: `Math.random()` in Corruption Breaks Determinism
|
|
**Location**: `src/quality/corruption.ts:165`
|
|
**Problem**: `rng ?? new SeededRng(Date.now())` uses `Date.now()` when no RNG provided, making corruption non-deterministic.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
FUNCTION corruptResponse(ctx, contentType, extensionRegistry, rng):
|
|
// MUST receive RNG from caller; no fallback to Math.random() or Date.now()
|
|
ASSERT rng IS NOT NULL, "corruptResponse requires injected SeededRng"
|
|
|
|
// Check extension-provided strategies first
|
|
IF extensionRegistry HAS strategy FOR contentType:
|
|
RETURN applyExtensionStrategy(ctx, strategy, rng)
|
|
|
|
// Fall back to built-in strategies using injected RNG
|
|
baseType = contentType.split(';')[0].trim()
|
|
builtin = BUILTIN_STRATEGIES[baseType]
|
|
IF builtin EXISTS:
|
|
RETURN {
|
|
ctx: applyCorruption(ctx, (data) => builtin.strategy(data, rng), contentType),
|
|
strategy: builtin.name,
|
|
description: builtin.description
|
|
}
|
|
|
|
// Generic fallback with injected RNG
|
|
RETURN {
|
|
ctx: applyCorruption(ctx, (data) => truncateText(data, rng), contentType),
|
|
strategy: 'generic-truncate',
|
|
description: 'Generic truncation'
|
|
}
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: `corruptResponse` MUST require `rng` parameter (non-optional)
|
|
- MUST: All corruption strategies MUST use the injected `rng`, never `Math.random()` or `Date.now()`
|
|
- MAY NEVER: Corruption produce different results for the same seed across runs
|
|
|
|
#### Issue C3: Seed Collision Risk in ChaosEngine
|
|
**Location**: `src/quality/chaos.ts:39`
|
|
**Problem**: `seed + 0xCA05` can collide for nearby seeds.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
CONSTRUCTOR(config, seed):
|
|
this.config = config
|
|
// Use hash-based seed derivation to avoid collisions
|
|
IF seed IS DEFINED:
|
|
chaosSeed = hashCombine(seed, 0xCA05)
|
|
ELSE:
|
|
chaosSeed = Date.now() // Only for undefined seed
|
|
this.rng = new SeededRng(chaosSeed)
|
|
|
|
FUNCTION hashCombine(a, b):
|
|
// FNV-1a inspired combination
|
|
hash = 0x811c9dc5
|
|
hash = ((hash ^ (a & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ ((a >>> 8) & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ ((a >>> 16) & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ ((a >>> 24) & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ (b & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ ((b >>> 8) & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ ((b >>> 16) & 0xFF)) * 0x01000193) >>> 0
|
|
hash = ((hash ^ ((b >>> 24) & 0xFF)) * 0x01000193) >>> 0
|
|
RETURN hash
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Chaos seed derivation MUST use a hash function, not simple addition
|
|
- MUST: Different test seeds MUST produce different chaos sequences with high probability
|
|
- MAY NEVER: Two test seeds within 100,000 of each other produce identical chaos behavior
|
|
|
|
### Runtime Hook Safety (Critical)
|
|
|
|
#### Issue H1: Hook Validator Throws 500s for Formula Parse Errors
|
|
**Location**: `src/infrastructure/hook-validator.ts:89-93, 101`
|
|
**Problem**: `preParseFormulas` throws on parse error, which becomes a 500 in Fastify hooks instead of failing at plugin registration time.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Split into two phases:
|
|
|
|
// Phase 1: Registration-time validation (in plugin/index.ts)
|
|
FUNCTION validateAllContracts(routes):
|
|
FOR EACH route IN routes:
|
|
FOR EACH formula IN route.requires + route.ensures:
|
|
TRY:
|
|
parse(formula)
|
|
CATCH err:
|
|
THROW Error(`Invalid formula in ${route.method} ${route.path}: ${formula}\n${err.message}`)
|
|
|
|
// Pre-parse and cache ASTs at registration time
|
|
routeAsts = new Map()
|
|
FOR EACH route IN routes:
|
|
routeAsts.set(route, {
|
|
requires: route.requires.map(f => parse(f).ast),
|
|
ensures: route.ensures.map(f => parse(f).ast)
|
|
})
|
|
RETURN routeAsts
|
|
|
|
// Phase 2: Hook uses pre-parsed ASTs (O(1), never throws)
|
|
FUNCTION createPreHandler(opts, routeAsts):
|
|
RETURN (request, reply, done) =>
|
|
contract = getRouteContract(request)
|
|
IF shouldSkipRoute(contract, opts):
|
|
done()
|
|
RETURN
|
|
|
|
asts = routeAsts.get(contract)?.requires
|
|
IF NOT asts:
|
|
done() // No requires to check
|
|
RETURN
|
|
|
|
context = buildPreContext(request)
|
|
evaluateFormulas(context, asts, contract.requires)
|
|
done()
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: All formulas MUST be parsed at plugin registration time, not at request time
|
|
- MUST: Parse errors MUST fail plugin registration with a clear error message
|
|
- MAY NEVER: A request-time hook throw a 500 due to formula syntax error
|
|
- MAY NEVER: Formula parsing happen on the request hot path
|
|
|
|
#### Issue H2: env-guard Throws at Runtime Instead of Registration
|
|
**Location**: `src/quality/env-guard.ts:8-14`
|
|
**Problem**: `assertTestEnv` throws when chaos/flake/mutation are first used, not when configured.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// At plugin registration time (src/plugin/index.ts):
|
|
FUNCTION registerApophis(fastify, opts):
|
|
IF opts.chaos IS DEFINED AND process.env.NODE_ENV !== 'test':
|
|
THROW Error('Chaos mode requires NODE_ENV=test')
|
|
|
|
IF opts.flake IS DEFINED AND process.env.NODE_ENV !== 'test':
|
|
THROW Error('Flake detection requires NODE_ENV=test')
|
|
|
|
IF opts.mutation IS DEFINED AND process.env.NODE_ENV !== 'test':
|
|
THROW Error('Mutation testing requires NODE_ENV=test')
|
|
|
|
// ... rest of registration
|
|
|
|
// Remove assertTestEnv from runtime paths entirely
|
|
// Quality features are only constructed in test environment
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Environment validation MUST happen at plugin registration time
|
|
- MUST: Quality feature configuration in non-test env MUST prevent plugin startup
|
|
- MAY NEVER: A runtime quality feature throw an environment error during test execution
|
|
|
|
### Architecture & Design (Martin Fowler / Uncle Bob)
|
|
|
|
#### Issue A1: petit-runner.ts Violates SRP (583 lines)
|
|
**Location**: `src/test/petit-runner.ts`
|
|
**Problem**: Single file handles command generation, precondition checking, HTTP execution, chaos injection, flake detection, postcondition validation, deduplication, and result formatting.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Extract into focused modules:
|
|
|
|
// src/test/command-generator.ts (lines 92-149)
|
|
FUNCTION generateCommands(routes, depth, seed):
|
|
// Pure: cache lookup, schema conversion, fast-check sampling
|
|
RETURN { commands, cacheHits, cacheMisses }
|
|
|
|
// src/test/precondition-checker.ts (lines 155-165)
|
|
FUNCTION checkPreconditions(command, state):
|
|
// Pure: check resource existence
|
|
RETURN boolean
|
|
|
|
// src/test/chaos-wrapper.ts (lines 288-298)
|
|
FUNCTION executeWithChaos(chaosEngine, executeFn, route, request, extensionRegistry):
|
|
// Effect: inject chaos if enabled
|
|
RETURN { ctx, chaosEvents }
|
|
|
|
// src/test/flake-detector.ts (lines 341-412)
|
|
FUNCTION detectFlake(failingResult, rerunFn, config, extensionRegistry, pluginContractRegistry):
|
|
// Effect: rerun with varied seeds
|
|
RETURN flakeReport
|
|
|
|
// src/test/postcondition-validator.ts (lines 324-339, 400-408)
|
|
FUNCTION validatePostconditionsWithPlugins(route, ctx, pluginContractRegistry, extensionRegistry):
|
|
// Pure: compose plugin + route contracts, validate
|
|
RETURN validationResult
|
|
|
|
// src/test/result-deduplicator.ts (lines 489-528)
|
|
FUNCTION deduplicateFailures(results):
|
|
// Pure: group by route+formula, keep first
|
|
RETURN dedupedResults
|
|
|
|
// src/test/petit-runner.ts (reduced to ~150 lines)
|
|
FUNCTION runPetitTests(fastify, config, scopeRegistry, extensionRegistry, pluginContractRegistry):
|
|
// Orchestrator: delegates to extracted modules
|
|
routes = discoverAndFilterRoutes(fastify, config)
|
|
{ commands, cacheHits, cacheMisses } = generateCommands(routes, depth, config.seed)
|
|
|
|
FOR EACH command IN commands:
|
|
IF NOT checkPreconditions(command, state):
|
|
results.push(SKIP)
|
|
CONTINUE
|
|
|
|
{ ctx, chaosEvents } = await executeWithChaos(...)
|
|
validation = validatePostconditionsWithPlugins(...)
|
|
|
|
IF NOT validation.success:
|
|
flakeReport = await detectFlake(...)
|
|
results.push(FAILURE with diagnostics)
|
|
ELSE:
|
|
results.push(SUCCESS)
|
|
|
|
state = updateModelState(command.route, ctx, state)
|
|
|
|
RETURN formatSuite(results, deduplicateFailures)
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: No test runner module exceed 200 lines of code
|
|
- MUST: Each module have a single, clearly stated responsibility
|
|
- MUST: Orchestrator modules contain only delegation logic, no implementation
|
|
- MAY NEVER: A module mix pure computation with side effects
|
|
|
|
#### Issue A2: stateful-runner.ts Duplicates petit-runner Logic
|
|
**Location**: `src/test/stateful-runner.ts:54-64, 66-72`
|
|
**Problem**: Precondition checking (`ApiOperation.check`) and HTTP execution (`ApiOperation.run`) duplicate petit-runner logic.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Extract shared operations:
|
|
|
|
// src/test/operations.ts
|
|
CLASS ApiOperation:
|
|
CONSTRUCTOR(route, params):
|
|
this.route = route
|
|
this.params = params
|
|
|
|
check(model):
|
|
RETURN checkPreconditions(this.route, model) // Shared with petit-runner
|
|
|
|
async run(real):
|
|
request = buildRequest(this.route, this.params, real.scopeHeaders, real.state, real.rng)
|
|
ctx = await executeHttp(real.fastify, this.route, request, real.previousCtx)
|
|
real.history.push(ctx)
|
|
real.previousCtx = ctx
|
|
|
|
// src/test/stateful-runner.ts
|
|
IMPORT { ApiOperation } from './operations.js'
|
|
IMPORT { checkPreconditions } from './precondition-checker.js'
|
|
|
|
// Stateful runner focuses on fast-check command() integration only
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Precondition checking logic exist in exactly one module
|
|
- MUST: HTTP execution logic exist in exactly one module
|
|
- MAY NEVER: Two runners implement the same logic differently
|
|
|
|
#### Issue A3: Plugin Entry Point is God Object Factory
|
|
**Location**: `src/plugin/index.ts`
|
|
**Problem**: Lines 24-48 do 7 things: swagger registration, spec building, contract testing, stateful testing, health checking, route capture, and cleanup.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/plugin/swagger.ts
|
|
FUNCTION registerSwagger(fastify, opts):
|
|
IF fastify HAS swagger: RETURN
|
|
swagger = await import('@fastify/swagger')
|
|
await fastify.register(swagger.default, opts.swagger ?? {})
|
|
|
|
// src/plugin/spec-builder.ts
|
|
FUNCTION buildSpec(fastify):
|
|
routes = discoverRoutes(fastify)
|
|
spec = fastify.swagger()
|
|
RETURN { ...spec, 'x-apophis-contracts': routes.map(...) }
|
|
|
|
// src/plugin/contract-runner.ts
|
|
FUNCTION buildContract(fastify, scope, extensionRegistry, pluginContractRegistry):
|
|
RETURN async (opts) =>
|
|
config = normalizeConfig(opts)
|
|
suite = await runPetitTests(fastify, config, scope, extensionRegistry, pluginContractRegistry)
|
|
validateNonEmptyDiscovery(suite, fastify)
|
|
RETURN suite
|
|
|
|
// src/plugin/index.ts (reduced to ~80 lines)
|
|
FUNCTION apophisPlugin(fastify, opts):
|
|
await registerSwagger(fastify, opts)
|
|
|
|
scopeRegistry = new ScopeRegistry()
|
|
cleanupManager = new CleanupManager(fastify, scopeRegistry)
|
|
extensionRegistry = createExtensionRegistry()
|
|
pluginContractRegistry = createPluginContractRegistry()
|
|
|
|
fastify.decorate('apophis', {
|
|
scope: scopeRegistry,
|
|
contract: buildContract(fastify, scopeRegistry, extensionRegistry, pluginContractRegistry),
|
|
stateful: buildStateful(fastify, scopeRegistry, cleanupManager, extensionRegistry, pluginContractRegistry),
|
|
check: buildCheck(fastify, scopeRegistry, extensionRegistry, pluginContractRegistry),
|
|
spec: buildSpec(fastify),
|
|
capture: captureRoute(fastify),
|
|
cleanup: cleanupManager,
|
|
extend: extensionRegistry.register.bind(extensionRegistry),
|
|
use: pluginContractRegistry.use.bind(pluginContractRegistry),
|
|
})
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Plugin entry point only orchestrate, never implement
|
|
- MUST: Each decoration factory live in its own module
|
|
- MAY NEVER: A single function perform more than 3 distinct responsibilities
|
|
|
|
### Type Safety (Uncle Bob)
|
|
|
|
#### Issue T1: `OperationHeader` Union with `string` Defeats Exhaustiveness
|
|
**Location**: `src/types.ts:79-83`
|
|
**Problem**: `| string` makes the union non-exhaustive; TypeScript can't verify all cases handled.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/types.ts
|
|
// Remove the | string catch-all; use branded type for extensions
|
|
|
|
export type CoreOperationHeader =
|
|
| 'request_body' | 'response_body' | 'response_code'
|
|
| 'request_headers' | 'response_headers' | 'query_params'
|
|
| 'cookies' | 'response_time' | 'redirect_count'
|
|
| 'redirect_url' | 'redirect_status'
|
|
| 'timeout_occurred' | 'timeout_value'
|
|
|
|
// Extension headers use branded type
|
|
export type ExtensionHeader = string & { readonly __brand: 'ExtensionHeader' }
|
|
|
|
export type OperationHeader = CoreOperationHeader | ExtensionHeader
|
|
|
|
// Extension registration validates and brands headers:
|
|
FUNCTION registerExtensionHeader(name: string): ExtensionHeader {
|
|
IF NOT /^[a-z_][a-z0-9_]*$/.test(name):
|
|
THROW Error(`Invalid extension header name: ${name}`)
|
|
RETURN name as ExtensionHeader
|
|
}
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Core operation headers be exhaustively listable
|
|
- MUST: Extension headers require explicit registration and validation
|
|
- MAY NEVER: An unvalidated string be accepted as an operation header
|
|
|
|
#### Issue T2: `RequestStructure.body?: unknown` is Lazy Typing
|
|
**Location**: `src/domain/request-builder.ts:14`
|
|
**Problem**: `unknown` provides no type safety for body construction.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/domain/request-builder.ts
|
|
export type RequestBody =
|
|
| { type: 'json'; data: Record<string, unknown> }
|
|
| { type: 'multipart'; fields: Record<string, unknown>; files: MultipartFiles }
|
|
| { type: 'text'; data: string }
|
|
| undefined
|
|
|
|
export interface RequestStructure {
|
|
method: string
|
|
url: string
|
|
headers: Record<string, string>
|
|
query?: Record<string, string>
|
|
body?: RequestBody
|
|
contentType?: string
|
|
}
|
|
|
|
// Build functions return discriminated union
|
|
FUNCTION buildJsonRequest(route, data, scopeHeaders, state):
|
|
RETURN {
|
|
method: route.method,
|
|
url: substitutePathParams(route.path, data, state),
|
|
headers: { ...scopeHeaders, 'content-type': 'application/json' },
|
|
body: { type: 'json', data: extractBodyParams(data, route.schema.body) }
|
|
}
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Request body type be a discriminated union, not `unknown`
|
|
- MUST: Body type determine serialization strategy
|
|
- MAY NEVER: A body be accepted without knowing its content type
|
|
|
|
### Performance & Implementation (John Carmack)
|
|
|
|
#### Issue P1: Hand-Rolled charCodeAt Parser (915 lines)
|
|
**Location**: `src/formula/parser.ts`
|
|
**Problem**: 915 lines of manual charCodeAt parsing is unmaintainable. Should use a parser generator or at least regex-based tokenizer.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Option A: Use a parser generator (PEG.js / nearley)
|
|
// grammar.ne:
|
|
// main -> expression
|
|
// expression -> comparison | boolean | conditional | quantified
|
|
// comparison -> operation _ comparator _ operation
|
|
// boolean -> expression _ ("&&" | "||" | "=>") _ expression
|
|
// ...
|
|
|
|
// Option B: Regex tokenizer + recursive descent (maintainable)
|
|
// src/formula/tokenizer.ts (~100 lines)
|
|
TOKEN_PATTERNS = [
|
|
{ type: 'IF', pattern: /^if\b/ },
|
|
{ type: 'THEN', pattern: /^then\b/ },
|
|
{ type: 'ELSE', pattern: /^else\b/ },
|
|
{ type: 'FOR', pattern: /^for\b/ },
|
|
{ type: 'EXISTS', pattern: /^exists\b/ },
|
|
{ type: 'IN', pattern: /^in\b/ },
|
|
{ type: 'IDENTIFIER', pattern: /^[a-zA-Z_][a-zA-Z0-9_-]*/ },
|
|
{ type: 'NUMBER', pattern: /^-?\d+(\.\d+)?/ },
|
|
{ type: 'STRING', pattern: /^"([^"\\]|\\.)*"/ },
|
|
{ type: 'COMPARATOR', pattern: /^(==|!=|<=|>=|<|>|matches)/ },
|
|
{ type: 'BOOLEAN_OP', pattern: /^(&&|\|\||=>)/ },
|
|
{ type: 'LPAREN', pattern: /^\(/ },
|
|
{ type: 'RPAREN', pattern: /^\)/ },
|
|
{ type: 'LBRACKET', pattern: /^\[/ },
|
|
{ type: 'RBRACKET', pattern: /^\]/ },
|
|
{ type: 'DOT', pattern: /^\./ },
|
|
{ type: 'COMMA', pattern: /^,/ },
|
|
{ type: 'WS', pattern: /^\s+/, skip: true }
|
|
]
|
|
|
|
FUNCTION tokenize(input):
|
|
tokens = []
|
|
pos = 0
|
|
WHILE pos < input.length:
|
|
matched = FALSE
|
|
FOR EACH pattern IN TOKEN_PATTERNS:
|
|
match = input.slice(pos).match(pattern.pattern)
|
|
IF match:
|
|
IF NOT pattern.skip:
|
|
tokens.push({ type: pattern.type, value: match[0], pos })
|
|
pos += match[0].length
|
|
matched = TRUE
|
|
BREAK
|
|
IF NOT matched:
|
|
THROW parseError(input, pos, `Unexpected character: ${input[pos]}`)
|
|
RETURN tokens
|
|
|
|
// src/formula/parser.ts (~200 lines, recursive descent on tokens)
|
|
FUNCTION parse(tokens):
|
|
pos = 0
|
|
|
|
FUNCTION peek():
|
|
RETURN tokens[pos]
|
|
|
|
FUNCTION consume(expectedType):
|
|
IF peek().type !== expectedType:
|
|
THROW parseError(...)
|
|
RETURN tokens[pos++]
|
|
|
|
FUNCTION parseExpression():
|
|
RETURN parseConditional()
|
|
|
|
FUNCTION parseConditional():
|
|
IF peek().type === 'IF':
|
|
consume('IF')
|
|
condition = parseBoolean()
|
|
consume('THEN')
|
|
thenBranch = parseExpression()
|
|
consume('ELSE')
|
|
elseBranch = parseExpression()
|
|
RETURN { type: 'conditional', condition, then: thenBranch, else: elseBranch }
|
|
RETURN parseBoolean()
|
|
|
|
// ... etc
|
|
|
|
RETURN parseExpression()
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Parser be maintainable by developers without parser expertise
|
|
- MUST: Tokenizer use regex patterns, not manual charCodeAt
|
|
- MUST: Parser structure follow standard recursive descent pattern
|
|
- MAY NEVER: A single parser file exceed 300 lines
|
|
|
|
#### Issue P2: `hashSchema` Only Keeps 16 Chars of SHA-256
|
|
**Location**: `src/incremental/hash.ts:88`
|
|
**Problem**: Truncating SHA-256 to 16 hex chars (64 bits) creates collision risk with large schemas.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Use full hash or at least 32 chars (128 bits)
|
|
FUNCTION hashSchema(schema):
|
|
IF schema IS undefined:
|
|
RETURN 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' // Empty SHA-256
|
|
|
|
cached = hashMemo.get(schema)
|
|
IF cached IS NOT undefined:
|
|
RETURN cached
|
|
|
|
hash = createHash('sha256')
|
|
FOR EACH key IN Object.keys(schema):
|
|
IF NOT RELEVANT_KEYS.has(key): CONTINUE
|
|
hash.update(key)
|
|
hash.update('=')
|
|
hashValue(hash, schema[key], new WeakSet())
|
|
hash.update(';')
|
|
|
|
result = hash.digest('hex') // Full 64 chars
|
|
hashMemo.set(schema, result)
|
|
RETURN result
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Schema hash use full SHA-256 output (64 hex chars)
|
|
- MUST: Hash memoization use WeakMap to avoid memory leaks
|
|
- MAY NEVER: Hash output be truncated below 256 bits
|
|
|
|
#### Issue P3: `PARSE_CACHE` Map Has No TTL
|
|
**Location**: `src/formula/parser.ts` (implied by cache pattern)
|
|
**Problem**: Parsed formula ASTs accumulate indefinitely in memory.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/formula/parser.ts
|
|
class LruCache<K, V>:
|
|
private cache = new Map<K, V>()
|
|
private maxSize: number
|
|
|
|
CONSTRUCTOR(maxSize = 1000):
|
|
this.maxSize = maxSize
|
|
|
|
get(key):
|
|
IF NOT this.cache.has(key): RETURN undefined
|
|
value = this.cache.get(key)
|
|
// Move to end (most recently used)
|
|
this.cache.delete(key)
|
|
this.cache.set(key, value)
|
|
RETURN value
|
|
|
|
set(key, value):
|
|
IF this.cache.has(key):
|
|
this.cache.delete(key)
|
|
ELSE IF this.cache.size >= this.maxSize:
|
|
// Evict least recently used (first item)
|
|
firstKey = this.cache.keys().next().value
|
|
this.cache.delete(firstKey)
|
|
this.cache.set(key, value)
|
|
|
|
const PARSE_CACHE = new LruCache<string, ParseResult>(1000)
|
|
|
|
FUNCTION parse(formula):
|
|
cached = PARSE_CACHE.get(formula)
|
|
IF cached IS NOT undefined:
|
|
RETURN cached
|
|
|
|
result = parseInternal(formula)
|
|
PARSE_CACHE.set(formula, result)
|
|
RETURN result
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Parse cache use LRU eviction with configurable max size
|
|
- MUST: Max cache size default to 1000 entries
|
|
- MAY NEVER: Cache grow unbounded in long-running processes
|
|
|
|
#### Issue P4: `Promise.race` in `executeHttp` Doesn't Cancel Inject
|
|
**Location**: `src/infrastructure/http-executor.ts:104-113`
|
|
**Problem**: When timeout wins the race, the `fastify.inject()` promise continues running, potentially causing memory leaks or side effects.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Use AbortController for cancellation
|
|
FUNCTION executeHttp(fastify, route, request, previous, timeoutMs):
|
|
// ... setup ...
|
|
|
|
const controller = new AbortController()
|
|
let timeoutId: NodeJS.Timeout | undefined
|
|
|
|
try:
|
|
const injectPromise = fastify.inject({
|
|
method: request.method,
|
|
url: fullUrl,
|
|
payload: request.multipart ? buildMultipartPayload(request.multipart) : request.body,
|
|
headers: request.headers,
|
|
})
|
|
|
|
IF timeoutMs AND timeoutMs > 0:
|
|
timeoutId = setTimeout(() => {
|
|
timedOut = true
|
|
controller.abort()
|
|
}, timeoutMs)
|
|
|
|
response = await injectPromise
|
|
|
|
IF timeoutId:
|
|
clearTimeout(timeoutId)
|
|
|
|
CATCH err:
|
|
IF timeoutId:
|
|
clearTimeout(timeoutId)
|
|
|
|
IF timedOut:
|
|
RETURN buildTimeoutContext(...)
|
|
|
|
THROW err
|
|
|
|
// Note: Fastify inject() may not support AbortController.
|
|
// Alternative: track active requests and clean up on suite end.
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Timeout mechanism prevent resource leaks from abandoned requests
|
|
- MUST: Active request tracking enable cleanup on test suite completion
|
|
- MAY NEVER: A timed-out request continue consuming resources indefinitely
|
|
|
|
#### Issue P5: Streaming NDJSON Loads Entire Response Into Memory
|
|
**Location**: `src/infrastructure/http-executor.ts:170-186`
|
|
**Problem**: `responseBody` is fully loaded as string, then split and parsed. No backpressure for large streams.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// For NDJSON streaming, use chunked processing
|
|
IF isStreaming AND streamFormat === 'ndjson':
|
|
const chunks = []
|
|
const maxChunks = streamConfig.maxChunks ?? 100
|
|
const maxChunkSize = streamConfig.maxChunkSize ?? 65536 // 64KB
|
|
const maxTotalSize = streamConfig.maxTotalSize ?? 1048576 // 1MB
|
|
|
|
let totalSize = 0
|
|
const lines = responseBody.split('\n')
|
|
|
|
FOR EACH line IN lines:
|
|
IF line.trim().length === 0: CONTINUE
|
|
|
|
const lineSize = line.length
|
|
IF lineSize > maxChunkSize:
|
|
log.warn(`NDJSON chunk exceeds max size: ${lineSize} > ${maxChunkSize}`)
|
|
CONTINUE
|
|
|
|
totalSize += lineSize
|
|
IF totalSize > maxTotalSize:
|
|
log.warn(`NDJSON total size exceeds max: ${totalSize} > ${maxTotalSize}`)
|
|
BREAK
|
|
|
|
IF chunks.length >= maxChunks:
|
|
log.warn(`NDJSON chunk count exceeds max: ${chunks.length} >= ${maxChunks}`)
|
|
BREAK
|
|
|
|
TRY:
|
|
chunks.push(JSON.parse(line))
|
|
CATCH:
|
|
chunks.push(line) // Keep raw line if not valid JSON
|
|
|
|
RETURN {
|
|
...ctx,
|
|
response: {
|
|
...ctx.response,
|
|
chunks: chunks as readonly unknown[],
|
|
streamDurationMs,
|
|
truncated: lines.length > maxChunks || totalSize > maxTotalSize
|
|
}
|
|
}
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: NDJSON processing enforce max chunk count, chunk size, and total size limits
|
|
- MUST: Exceeded limits truncate gracefully with warning, not error
|
|
- MAY NEVER: Streaming response processing consume unbounded memory
|
|
|
|
#### Issue P6: `request-builder.ts` Uses `Math.random()` as Fallback
|
|
**Location**: `src/domain/request-builder.ts:112`
|
|
**Problem**: When no RNG provided, falls back to `Math.random()` for path param selection.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/domain/request-builder.ts
|
|
FUNCTION substitutePathParams(path, data, state, rng):
|
|
url = path
|
|
pathParams = parseRouteParams(path)
|
|
|
|
FOR EACH param IN pathParams:
|
|
value = data[param]
|
|
|
|
IF value IS undefined AND param.endsWith('Id'):
|
|
resourceType = param.replace(/Id$/, '').toLowerCase()
|
|
resources = state.resources.get(resourceType)
|
|
IF resources AND resources.size > 0:
|
|
ids = Array.from(resources.keys())
|
|
IF rng IS DEFINED:
|
|
value = rng.pick(ids)
|
|
ELSE:
|
|
// Deterministic fallback: use first ID (consistent across runs)
|
|
value = ids[0]
|
|
log.warn(`No RNG provided for path param selection; using deterministic fallback`)
|
|
|
|
IF value IS NOT undefined:
|
|
url = url.replace(`:${param}`, String(value))
|
|
|
|
RETURN url
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Path param selection use injected RNG when available
|
|
- MUST: Missing RNG produce deterministic, logged fallback
|
|
- MAY NEVER: `Math.random()` be used in test generation paths
|
|
|
|
#### Issue P7: Duplicate Sync/Async Evaluation Paths in `evaluator.ts`
|
|
**Location**: `src/formula/evaluator.ts`
|
|
**Problem**: Two parallel code paths for sync and async evaluation; easy to drift.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// Unify on async; sync is just async with no await
|
|
FUNCTION evaluate(node, ctx, extensionRegistry):
|
|
SWITCH node.type:
|
|
CASE 'literal':
|
|
RETURN node.value
|
|
|
|
CASE 'variable':
|
|
RETURN resolveVariable(node.name, ctx)
|
|
|
|
CASE 'operation':
|
|
resolver = extensionRegistry?.resolvePredicate(node.header)
|
|
IF resolver:
|
|
// Always async; await if needed
|
|
result = resolver(ctx, node.parameter, node.accessor)
|
|
IF result IS Promise:
|
|
RETURN await result
|
|
RETURN result
|
|
RETURN resolveBuiltinOperation(node.header, ctx, node.accessor)
|
|
|
|
CASE 'comparison':
|
|
left = await evaluate(node.left, ctx, extensionRegistry)
|
|
right = await evaluate(node.right, ctx, extensionRegistry)
|
|
RETURN applyComparator(node.op, left, right)
|
|
|
|
CASE 'boolean':
|
|
left = await evaluate(node.left, ctx, extensionRegistry)
|
|
|
|
// Short-circuit
|
|
IF node.op === '&&' AND NOT left:
|
|
RETURN false
|
|
IF node.op === '||' AND left:
|
|
RETURN true
|
|
IF node.op === '=>'':
|
|
IF NOT left:
|
|
RETURN true // False antecedent = true
|
|
RETURN await evaluate(node.right, ctx, extensionRegistry)
|
|
|
|
right = await evaluate(node.right, ctx, extensionRegistry)
|
|
RETURN applyBooleanOp(node.op, left, right)
|
|
|
|
CASE 'conditional':
|
|
condition = await evaluate(node.condition, ctx, extensionRegistry)
|
|
IF condition:
|
|
RETURN await evaluate(node.then, ctx, extensionRegistry)
|
|
RETURN await evaluate(node.else, ctx, extensionRegistry)
|
|
|
|
CASE 'quantified':
|
|
collection = resolveCollection(node.collection, ctx)
|
|
IF node.quantifier === 'for':
|
|
FOR EACH item IN collection:
|
|
result = await evaluate(node.body, ctx.withBinding(node.variable, item), extensionRegistry)
|
|
IF NOT result:
|
|
RETURN false
|
|
RETURN true
|
|
ELSE: // exists
|
|
FOR EACH item IN collection:
|
|
result = await evaluate(node.body, ctx.withBinding(node.variable, item), extensionRegistry)
|
|
IF result:
|
|
RETURN true
|
|
RETURN false
|
|
|
|
CASE 'previous':
|
|
RETURN evaluate(node.inner, ctx.previous, extensionRegistry)
|
|
|
|
CASE 'status':
|
|
RETURN ctx.response.statusCode === node.code
|
|
|
|
// Public API: always async
|
|
export async function evaluateFormula(node, ctx, extensionRegistry):
|
|
RETURN evaluate(node, ctx, extensionRegistry)
|
|
|
|
// Backward compat: sync wrapper for simple cases
|
|
export function evaluateFormulaSync(node, ctx, extensionRegistry):
|
|
result = evaluate(node, ctx, extensionRegistry)
|
|
IF result IS Promise:
|
|
THROW Error('Sync evaluation encountered async predicate; use evaluateFormula()')
|
|
RETURN result
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Single evaluation implementation, not parallel sync/async paths
|
|
- MUST: All evaluation be async by default; sync wrapper fail on async encounter
|
|
- MAY NEVER: Sync and async paths diverge in behavior
|
|
|
|
#### Issue P8: `topologicalSort` Re-sorts Entire Array on Every `register()`
|
|
**Location**: `src/extension/registry.ts:159`
|
|
**Problem**: O(n²) complexity when registering extensions one at a time.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/extension/registry.ts
|
|
class ExtensionRegistryImpl:
|
|
private _extensions: ApophisExtension[] = []
|
|
private _sorted = false
|
|
|
|
register(extension):
|
|
// Validate uniqueness
|
|
IF this._extensions.some(e => e.name === extension.name):
|
|
THROW Error(`Extension '${extension.name}' already registered`)
|
|
|
|
// Add without sorting
|
|
this._extensions.push(extension)
|
|
this._sorted = false
|
|
|
|
// Notify plugin contract registry
|
|
IF this._pluginContractRegistry:
|
|
this._pluginContractRegistry.registerAvailableExtension(extension.name)
|
|
|
|
// Cache predicates and corruption strategies immediately
|
|
this._cacheExtensionData(extension)
|
|
|
|
// Lazy sort: only when hook arrays are needed
|
|
private ensureSorted():
|
|
IF this._sorted: RETURN
|
|
|
|
this._extensions = topologicalSort(this._extensions)
|
|
this._rebuildHookArrays()
|
|
this._sorted = true
|
|
|
|
get extensions():
|
|
this.ensureSorted()
|
|
RETURN this._extensions
|
|
|
|
runBuildRequestHooks(ctx):
|
|
this.ensureSorted()
|
|
FOR EACH ext IN this._buildRequestExts:
|
|
// ... run hook
|
|
|
|
// ... other hook runners call ensureSorted() first
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Extension registration be O(1) amortized
|
|
- MUST: Sorting happen lazily, only when hooks are first accessed
|
|
- MAY NEVER: Registration trigger full re-sort of all extensions
|
|
|
|
#### Issue P9: `safe-regex` Has False Positives/Negatives, No Timeout Enforcement
|
|
**Location**: `src/infrastructure/regex-guard.ts`
|
|
**Problem**: `safe-regex` is a heuristic with known false positives and negatives. No actual ReDoS protection via timeout.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/infrastructure/regex-guard.ts
|
|
import { Worker } from 'node:worker_threads'
|
|
|
|
const SAFE_REGEX_TIMEOUT_MS = 1000
|
|
|
|
FUNCTION validateRegexPattern(pattern):
|
|
TRY:
|
|
// Fast heuristic check
|
|
IF NOT safeRegex(pattern):
|
|
RETURN { safe: false, reason: 'Pattern flagged by safe-regex heuristic', severity: 'exponential' }
|
|
|
|
// Compile and test with timeout
|
|
regex = new RegExp(pattern)
|
|
|
|
// Test with a pathological input in a worker with timeout
|
|
testResult = await testRegexWithTimeout(regex, SAFE_REGEX_TIMEOUT_MS)
|
|
|
|
IF testResult.timedOut:
|
|
RETURN { safe: false, reason: 'Pattern timed out during test (potential ReDoS)', severity: 'exponential' }
|
|
|
|
RETURN { safe: true, severity: 'safe' }
|
|
|
|
CATCH err:
|
|
RETURN { safe: false, reason: `Validation error: ${err.message}`, severity: 'exponential' }
|
|
|
|
FUNCTION testRegexWithTimeout(regex, timeoutMs):
|
|
RETURN new Promise((resolve) => {
|
|
const worker = new Worker(`
|
|
const { parentPort } = require('worker_threads');
|
|
parentPort.once('message', ({ pattern, input }) => {
|
|
const regex = new RegExp(pattern);
|
|
const start = Date.now();
|
|
try {
|
|
regex.test(input);
|
|
parentPort.postMessage({ elapsed: Date.now() - start });
|
|
} catch (err) {
|
|
parentPort.postMessage({ error: err.message });
|
|
}
|
|
});
|
|
`, { eval: true });
|
|
|
|
const timer = setTimeout(() => {
|
|
worker.terminate();
|
|
resolve({ timedOut: true });
|
|
}, timeoutMs);
|
|
|
|
worker.once('message', (result) => {
|
|
clearTimeout(timer);
|
|
worker.terminate();
|
|
resolve(result);
|
|
});
|
|
|
|
// Pathological input: repeated 'a' followed by 'b'
|
|
worker.postMessage({ pattern: regex.source, input: 'a'.repeat(100) + 'b' });
|
|
});
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Regex validation include actual execution timeout test
|
|
- MUST: Timeout test run in isolated worker thread
|
|
- MUST: Patterns timing out be rejected regardless of heuristic result
|
|
- MAY NEVER: A regex be accepted solely based on heuristic analysis
|
|
|
|
#### Issue P10: Redaction Logic is Overly Broad
|
|
**Location**: `src/extension/redaction.ts:48, 77`
|
|
**Problem**: `lowerKey.includes(sensitive)` matches partial substrings (e.g., "authorization" matches "auth" but also false-positives on "author_name").
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/extension/redaction.ts
|
|
const SENSITIVE_FIELDS = new Set([
|
|
'authorization',
|
|
'x-api-key',
|
|
'x-auth-token',
|
|
'api-key',
|
|
'token',
|
|
'access_token',
|
|
'refresh_token',
|
|
'id_token',
|
|
'client_secret',
|
|
'cookie',
|
|
'session',
|
|
'sessionid',
|
|
'phpsessid',
|
|
'password',
|
|
'secret',
|
|
'private_key',
|
|
'api_secret',
|
|
'ssn',
|
|
'social_security',
|
|
'credit_card',
|
|
'creditcard',
|
|
'cvv',
|
|
])
|
|
|
|
FUNCTION isSensitiveField(key):
|
|
lowerKey = key.toLowerCase()
|
|
|
|
// Exact match
|
|
IF SENSITIVE_FIELDS.has(lowerKey):
|
|
RETURN true
|
|
|
|
// Prefix match for known patterns (e.g., x-auth-token-xxx)
|
|
FOR EACH sensitive IN SENSITIVE_FIELDS:
|
|
IF lowerKey === sensitive OR lowerKey.startsWith(sensitive + '-') OR lowerKey.startsWith(sensitive + '_'):
|
|
RETURN true
|
|
|
|
RETURN false
|
|
|
|
FUNCTION redactHeaders(headers):
|
|
result = {}
|
|
FOR EACH [key, value] IN Object.entries(headers):
|
|
IF isSensitiveField(key):
|
|
result[key] = '[REDACTED]'
|
|
ELSE:
|
|
result[key] = value
|
|
RETURN result
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Redaction use exact or prefix matching, not substring matching
|
|
- MUST: Redaction set be explicitly listed, not dynamically generated
|
|
- MAY NEVER: A non-sensitive field be redacted due to partial string match
|
|
|
|
#### Issue P11: `substitutor.ts` PARAM_PATTERN Could Inject Arbitrary APOSTL
|
|
**Location**: Implied by parameter substitution pattern
|
|
**Problem**: If user input reaches parameter substitution without validation, arbitrary APOSTL formulas could be injected.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/domain/substitutor.ts (or request-builder.ts)
|
|
const PARAM_PATTERN = /:([a-zA-Z_][a-zA-Z0-9_]*)/g
|
|
|
|
FUNCTION substitutePathParams(path, data, state, rng):
|
|
url = path
|
|
|
|
FOR EACH match OF path.matchAll(PARAM_PATTERN):
|
|
paramName = match[1]
|
|
|
|
// Validate param name is alphanumeric + underscore only
|
|
IF NOT /^[a-zA-Z_][a-zA-Z0-9_]*$/.test(paramName):
|
|
THROW Error(`Invalid path parameter name: ${paramName}`)
|
|
|
|
value = data[paramName]
|
|
|
|
// Sanitize value before substitution
|
|
IF value IS NOT undefined:
|
|
sanitized = String(value).replace(/[^a-zA-Z0-9_.~-]/g, (c) => `%${c.charCodeAt(0).toString(16).toUpperCase().padStart(2, '0')}`)
|
|
url = url.replace(match[0], sanitized)
|
|
|
|
RETURN url
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Path parameter names be validated against whitelist
|
|
- MUST: Parameter values be URL-encoded before substitution
|
|
- MAY NEVER: Unsanitized user input be substituted into URLs
|
|
|
|
### Observability (Charity Majors)
|
|
|
|
#### Issue O1: Zero OpenTelemetry Integration
|
|
**Location**: Entire codebase
|
|
**Problem**: No distributed tracing, no metrics, no correlation between CI test failures and production incidents.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/infrastructure/telemetry.ts
|
|
import { trace, metrics, context } from '@opentelemetry/api'
|
|
|
|
class ApophisTelemetry:
|
|
private tracer = trace.getTracer('apophis', '1.3.0')
|
|
private meter = metrics.getMeter('apophis', '1.3.0')
|
|
|
|
// Counters
|
|
private testsRun = this.meter.createCounter('apophis.tests.run')
|
|
private testsPassed = this.meter.createCounter('apophis.tests.passed')
|
|
private testsFailed = this.meter.createCounter('apophis.tests.failed')
|
|
private testsFlaky = this.meter.createCounter('apophis.tests.flaky')
|
|
private chaosEvents = this.meter.createCounter('apophis.chaos.events')
|
|
private contractViolations = this.meter.createCounter('apophis.contract.violations')
|
|
|
|
// Histograms
|
|
private testDuration = this.meter.createHistogram('apophis.test.duration_ms')
|
|
private requestDuration = this.meter.createHistogram('apophis.request.duration_ms')
|
|
|
|
startTestSpan(testName, attributes):
|
|
RETURN this.tracer.startSpan('apophis.test', {
|
|
attributes: {
|
|
'apophis.test.name': testName,
|
|
'apophis.test.runner': 'petit',
|
|
...attributes
|
|
}
|
|
})
|
|
|
|
recordTestResult(span, result):
|
|
this.testsRun.add(1)
|
|
IF result.ok:
|
|
this.testsPassed.add(1)
|
|
ELSE:
|
|
this.testsFailed.add(1)
|
|
IF result.diagnostics?.flake?.isFlaky:
|
|
this.testsFlaky.add(1)
|
|
|
|
span.setAttribute('apophis.test.result', result.ok ? 'pass' : 'fail')
|
|
span.end()
|
|
|
|
recordChaosEvent(eventType):
|
|
this.chaosEvents.add(1, { 'apophis.chaos.type': eventType })
|
|
|
|
recordContractViolation(formula, route):
|
|
this.contractViolations.add(1, {
|
|
'apophis.contract.formula': formula,
|
|
'apophis.contract.route': route
|
|
})
|
|
|
|
// Integration in petit-runner.ts
|
|
FUNCTION runPetitTests(fastify, config, ...):
|
|
telemetry = new ApophisTelemetry()
|
|
|
|
FOR EACH command IN allCommands:
|
|
span = telemetry.startTestSpan(command.route.path, {
|
|
'http.method': command.route.method,
|
|
'http.route': command.route.path
|
|
})
|
|
|
|
TRY:
|
|
ctx = await executeHttp(...)
|
|
validation = validatePostconditions(...)
|
|
|
|
IF NOT validation.success:
|
|
telemetry.recordContractViolation(validation.formula, command.route.path)
|
|
telemetry.recordTestResult(span, { ok: false })
|
|
ELSE:
|
|
telemetry.recordTestResult(span, { ok: true })
|
|
|
|
CATCH err:
|
|
span.recordException(err)
|
|
telemetry.recordTestResult(span, { ok: false })
|
|
|
|
FINALLY:
|
|
span.end()
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Every test execution produce a trace span
|
|
- MUST: Every contract violation produce a metric
|
|
- MUST: Chaos events be counted and tagged by type
|
|
- MUST: Flaky tests be distinguishable from consistent failures in metrics
|
|
- MAY NEVER: A test run produce zero telemetry output
|
|
|
|
#### Issue O2: No Per-Route Chaos Granularity
|
|
**Location**: `src/quality/chaos.ts`
|
|
**Problem**: Chaos config is global; cannot disable chaos for specific routes or apply different strategies per route.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/types.ts
|
|
export interface ChaosConfig:
|
|
probability: number
|
|
delay?: { probability: number; minMs: number; maxMs: number }
|
|
error?: { probability: number; statusCode: number; body?: unknown }
|
|
dropout?: { probability: number }
|
|
corruption?: { probability: number }
|
|
|
|
// Per-route overrides
|
|
routeOverrides?: Map<string, Partial<ChaosConfig>> // key: "METHOD path"
|
|
|
|
// Route matchers (regex patterns)
|
|
excludeRoutes?: string[] // Routes to never chaos
|
|
includeOnlyRoutes?: string[] // If set, only chaos these routes
|
|
|
|
// src/quality/chaos.ts
|
|
FUNCTION shouldApplyChaos(route, config):
|
|
routeKey = `${route.method} ${route.path}`
|
|
|
|
// Check exclusions
|
|
IF config.excludeRoutes:
|
|
FOR EACH pattern IN config.excludeRoutes:
|
|
IF new RegExp(pattern).test(routeKey):
|
|
RETURN false
|
|
|
|
// Check inclusions
|
|
IF config.includeOnlyRoutes:
|
|
matched = false
|
|
FOR EACH pattern IN config.includeOnlyRoutes:
|
|
IF new RegExp(pattern).test(routeKey):
|
|
matched = true
|
|
BREAK
|
|
IF NOT matched:
|
|
RETURN false
|
|
|
|
RETURN true
|
|
|
|
FUNCTION executeWithChaos(executeHttp, route, request, extensionRegistry):
|
|
IF NOT shouldApplyChaos(route, this.config):
|
|
ctx = await executeHttp()
|
|
RETURN { ctx, events: [] }
|
|
|
|
// Merge route-specific config
|
|
routeKey = `${route.method} ${route.path}`
|
|
routeConfig = this.config.routeOverrides?.get(routeKey) ?? {}
|
|
effectiveConfig = { ...this.config, ...routeConfig }
|
|
|
|
// ... rest of chaos logic using effectiveConfig
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Chaos support per-route enable/disable
|
|
- MUST: Route matching use regex patterns for flexibility
|
|
- MUST: Per-route config override global config for that route
|
|
- MAY NEVER: Chaos be applied to excluded routes
|
|
|
|
#### Issue O3: No Resilience Verification After Chaos
|
|
**Location**: `src/quality/chaos.ts`
|
|
**Problem**: Chaos injects failures but doesn't verify the system recovers (resilience testing).
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/quality/chaos.ts
|
|
FUNCTION executeWithChaos(executeHttp, route, request, extensionRegistry):
|
|
// ... inject chaos ...
|
|
|
|
// After chaos injection, verify system health
|
|
IF this.config.resilienceCheck:
|
|
healthResult = await this.checkResilience(route, request, extensionRegistry)
|
|
|
|
IF NOT healthResult.healthy:
|
|
this.events.push({
|
|
type: 'resilience_failure',
|
|
injected: false,
|
|
details: {
|
|
reason: `System did not recover after ${eventType}: ${healthResult.reason}`,
|
|
recoveryTimeMs: healthResult.recoveryTimeMs
|
|
}
|
|
})
|
|
|
|
RETURN { ctx, events: this.events }
|
|
|
|
FUNCTION checkResilience(route, request, extensionRegistry):
|
|
startTime = Date.now()
|
|
|
|
// Retry the same request without chaos
|
|
retryCtx = await executeHttp()
|
|
|
|
// Check if response is valid
|
|
validation = validatePostconditions(route.ensures, retryCtx, route, extensionRegistry)
|
|
|
|
recoveryTimeMs = Date.now() - startTime
|
|
|
|
IF validation.success AND retryCtx.response.statusCode < 500:
|
|
RETURN { healthy: true, recoveryTimeMs }
|
|
ELSE:
|
|
RETURN {
|
|
healthy: false,
|
|
recoveryTimeMs,
|
|
reason: validation.error ?? `HTTP ${retryCtx.response.statusCode}`
|
|
}
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Chaos mode optionally verify system recovery after injection
|
|
- MUST: Resilience check measure recovery time
|
|
- MUST: Resilience failures be reported as distinct event type
|
|
- MAY NEVER: Chaos injection silently leave system in degraded state
|
|
|
|
#### Issue O4: Runtime Hooks Evaluate on EVERY Request
|
|
**Location**: `src/infrastructure/hook-validator.ts:110-128, 135-153`
|
|
**Problem**: Hooks run on every request in production, adding overhead even for routes with no contracts.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/infrastructure/hook-validator.ts
|
|
FUNCTION registerValidationHooks(fastify, opts, routes):
|
|
// Pre-filter: only routes with contracts need hooks
|
|
contractRoutes = routes.filter(r => hasContractAnnotations(r) AND r.validateRuntime)
|
|
|
|
IF contractRoutes.length === 0:
|
|
log.info('No runtime validation hooks registered (no contracts with validateRuntime)')
|
|
RETURN
|
|
|
|
// Pre-parse all formulas at registration time
|
|
routeAsts = new Map()
|
|
FOR EACH route IN contractRoutes:
|
|
routeAsts.set(`${route.method} ${route.path}`, {
|
|
requires: route.requires.map(f => parse(f).ast),
|
|
ensures: route.ensures.map(f => parse(f).ast)
|
|
})
|
|
|
|
// Register hooks only for routes with contracts
|
|
fastify.addHook('preHandler', createPreHandler(opts, routeAsts))
|
|
fastify.addHook('preSerialization', createPreSerializer())
|
|
fastify.addHook('onSend', createOnSend(opts, routeAsts))
|
|
|
|
log.info(`Registered runtime validation for ${contractRoutes.length} routes`)
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Hooks only register for routes with runtime validation enabled
|
|
- MUST: Routes without contracts incur zero hook overhead
|
|
- MUST: Registration log count of hooked routes
|
|
- MAY NEVER: A route without contracts trigger hook execution
|
|
|
|
### Category Inference (Martin Fowler)
|
|
|
|
#### Issue Cat1: Hardcoded Exact Paths Miss Prefixed Variants
|
|
**Location**: `src/domain/category.ts:12-47`
|
|
**Problem**: `/api/health`, `/v1/health`, `/internal/health` are not recognized as utility paths.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/domain/category.ts
|
|
const UTILITY_PATTERNS = [
|
|
/^\/?(api\/)?(v\d+\/)?(reset|health|ping|login|logout|auth|callback|purge|clear|initialize|setup|webhook)\/?$/i,
|
|
]
|
|
|
|
const isUtilityPath = (path):
|
|
FOR EACH pattern IN UTILITY_PATTERNS:
|
|
IF pattern.test(path):
|
|
RETURN true
|
|
RETURN false
|
|
|
|
// Or use suffix matching
|
|
const UTILITY_SUFFIXES = new Set([
|
|
'reset', 'health', 'ping', 'login', 'logout', 'auth',
|
|
'callback', 'purge', 'clear', 'initialize', 'setup', 'webhook'
|
|
])
|
|
|
|
const isUtilityPath = (path):
|
|
// Remove leading/trailing slashes and version prefixes
|
|
normalized = path.replace(/^\//, '').replace(/\/$/, '')
|
|
segments = normalized.split('/')
|
|
lastSegment = segments[segments.length - 1]
|
|
|
|
// Check if last segment is a known utility suffix
|
|
IF UTILITY_SUFFIXES.has(lastSegment.toLowerCase()):
|
|
RETURN true
|
|
|
|
// Check exact matches for root-level paths
|
|
RETURN UTILITY_SUFFIXES.has(normalized.toLowerCase())
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Category inference recognize utility paths regardless of prefix
|
|
- MUST: Path normalization handle leading/trailing slashes and version segments
|
|
- MAY NEVER: A `/api/health` route be categorized as non-utility
|
|
|
|
### APOSTL Formula Language (Martin Fowler)
|
|
|
|
#### Issue F1: No Arithmetic Operators
|
|
**Location**: `src/formula/parser.ts`, `src/formula/evaluator.ts`
|
|
**Problem**: APOSTL lacks `+`, `-`, `*`, `/` operators, limiting expressiveness.
|
|
|
|
**Pseudocode Fix**:
|
|
```
|
|
// src/types.ts
|
|
export type FormulaNode =
|
|
| ...existing nodes...
|
|
| { type: 'arithmetic'; op: '+' | '-' | '*' | '/'; left: FormulaNode; right: FormulaNode }
|
|
|
|
// src/formula/parser.ts
|
|
// Add to grammar:
|
|
// expression -> additive
|
|
// additive -> multiplicative (('+' | '-') multiplicative)*
|
|
// multiplicative -> unary (('*' | '/') unary)*
|
|
// unary -> ('-' unary) | primary
|
|
|
|
FUNCTION parseExpression():
|
|
RETURN parseAdditive()
|
|
|
|
FUNCTION parseAdditive():
|
|
left = parseMultiplicative()
|
|
|
|
WHILE peek().type IN ['+', '-']:
|
|
op = consume(peek().type).type
|
|
right = parseMultiplicative()
|
|
left = { type: 'arithmetic', op, left, right }
|
|
|
|
RETURN left
|
|
|
|
FUNCTION parseMultiplicative():
|
|
left = parseUnary()
|
|
|
|
WHILE peek().type IN ['*', '/']:
|
|
op = consume(peek().type).type
|
|
right = parseUnary()
|
|
left = { type: 'arithmetic', op, left, right }
|
|
|
|
RETURN left
|
|
|
|
FUNCTION parseUnary():
|
|
IF peek().type === '-':
|
|
consume('-')
|
|
operand = parseUnary()
|
|
RETURN { type: 'arithmetic', op: '*', left: { type: 'literal', value: -1 }, right: operand }
|
|
|
|
RETURN parsePrimary()
|
|
|
|
// src/formula/evaluator.ts
|
|
CASE 'arithmetic':
|
|
left = await evaluate(node.left, ctx, extensionRegistry)
|
|
right = await evaluate(node.right, ctx, extensionRegistry)
|
|
|
|
IF typeof left !== 'number' OR typeof right !== 'number':
|
|
THROW Error(`Arithmetic operator ${node.op} requires numeric operands`)
|
|
|
|
SWITCH node.op:
|
|
CASE '+': RETURN left + right
|
|
CASE '-': RETURN left - right
|
|
CASE '*': RETURN left * right
|
|
CASE '/':
|
|
IF right === 0:
|
|
THROW Error('Division by zero')
|
|
RETURN left / right
|
|
```
|
|
|
|
**Invariants**:
|
|
- MUST: Arithmetic operators support `+`, `-`, `*`, `/`
|
|
- MUST: Arithmetic require numeric operands
|
|
- MUST: Division by zero produce clear error
|
|
- MAY NEVER: Arithmetic operators accept non-numeric operands silently
|
|
|
|
## Remaining
|
|
|
|
### Medium Priority
|
|
|
|
- [ ] **F6**: CI/CD examples (`docs/ci-cd.md`) — GitHub Actions, GitLab CI, CircleCI workflows
|
|
|
|
### Quality Features (Phase 2-3)
|
|
|
|
- [x] **Flake Detection** (`src/quality/flake.ts`) — Auto-rerun failing tests with varied seeds
|
|
- [ ] **Mutation Testing** (`src/quality/mutation.ts`) — Synthetic bug injection, contract strength scoring
|
|
|
|
## Metrics
|
|
|
|
| Metric | v1.1 | v1.2 | v1.3 | Target |
|
|
|--------|------|------|------|--------|
|
|
| Tests passing | 482 | 502 | 551 | 551+ |
|
|
| Protocol extensions | 0 | 0 | 8 | 8 |
|
|
| Plugin contracts | 0 | 0 | 4 built-in | 4+ |
|
|
| Chaos mode | 0 | 1 engine | 1 engine | 1 engine |
|
|
| Flake detection | 0 | 0 | 0 | Auto-rerun |
|
|
| Mutation testing | 0 | 0 | 0 | Score reporting |
|
|
| CI/CD examples | 0 | 0 | 0 | 3 workflows |
|
|
|
|
## v2.0: APOSTL → Justin (Subscript) Migration
|
|
|
|
### Decision: Migrate to Justin Expression Language
|
|
|
|
**Rationale**: Pre-release, minimal internal adoption. Perfect time for clean break. Justin provides:
|
|
- Arithmetic, null coalescing, optional chaining (free)
|
|
- "Just JS" syntax — no new DSL to learn
|
|
- ~3KB bundle (smaller than custom parser)
|
|
- Sandboxed execution (no `__proto__`, `constructor`, `eval`)
|
|
- IDE support out of the box (ESLint, Prettier, syntax highlighting)
|
|
|
|
### Design Decisions
|
|
|
|
1. **Property naming**: HTTP standard names (`statusCode`, `request.body`, `response.headers`)
|
|
2. **Previous context**: First-class via `previous` object (`previous.response.statusCode`)
|
|
3. **Extensions**: Register as context methods and variables (not operation headers)
|
|
4. **Quantifiers**: Built-in array methods (`every`, `some`, `find`, `filter`)
|
|
5. **Bundle size**: Acceptable (net reduction after deleting custom parser)
|
|
6. **Rigor**: Despite JS syntax, still extract invariants and implications for model testing
|
|
|
|
### Schema Annotation Changes
|
|
|
|
```javascript
|
|
// BEFORE: APOSTL
|
|
x-ensures: 'response_body(this).name == "John"'
|
|
x-requires: 'response_code(this) == 200'
|
|
x-requires: 'for item in response_body: item.price > 0'
|
|
|
|
// AFTER: Justin
|
|
x-ensures: 'response.body.name == "John"'
|
|
x-requires: 'statusCode == 200'
|
|
x-requires: 'response.body.every(item => item.price > 0)'
|
|
```
|
|
|
|
### Context Mapping
|
|
|
|
Justin receives a flat context built from `EvalContext`:
|
|
|
|
```javascript
|
|
{
|
|
// Request properties
|
|
request: {
|
|
body: ctx.request.body,
|
|
headers: ctx.request.headers,
|
|
query: ctx.request.query,
|
|
params: ctx.request.params,
|
|
cookies: ctx.request.cookies,
|
|
multipart: ctx.request.multipart
|
|
},
|
|
|
|
// Response properties
|
|
response: {
|
|
body: ctx.response.body,
|
|
headers: ctx.response.headers,
|
|
statusCode: ctx.response.statusCode,
|
|
status: ctx.response.statusCode, // alias
|
|
responseTime: ctx.response.responseTime,
|
|
chunks: ctx.response.chunks,
|
|
streamDurationMs: ctx.response.streamDurationMs
|
|
},
|
|
|
|
// Redirects
|
|
redirects: ctx.redirects,
|
|
redirectCount: ctx.redirects?.length,
|
|
|
|
// Timeout
|
|
timedOut: ctx.timedOut,
|
|
timeoutMs: ctx.timeoutMs,
|
|
|
|
// Previous context (for temporal assertions)
|
|
previous: ctx.previous ? buildContext(ctx.previous) : null
|
|
}
|
|
```
|
|
|
|
### Extension Integration
|
|
|
|
Extensions register context variables and methods:
|
|
|
|
```javascript
|
|
// Extension registers itself
|
|
extensionRegistry.register({
|
|
name: 'jwt',
|
|
context: {
|
|
// Variables
|
|
jwtClaims: (ctx) => parseJwt(ctx.request.headers.authorization),
|
|
jwtExpired: (ctx) => isJwtExpired(ctx.request.headers.authorization),
|
|
|
|
// Methods
|
|
jwtHasClaim: (ctx, claim) => {
|
|
const claims = parseJwt(ctx.request.headers.authorization)
|
|
return claims?.[claim] !== undefined
|
|
}
|
|
}
|
|
})
|
|
|
|
// Used in formulas:
|
|
// x-ensures: 'jwtClaims.role == "admin"'
|
|
// x-requires: 'jwtHasClaim("sub")'
|
|
```
|
|
|
|
### Manual Migration Approach
|
|
|
|
No automated migration scripts. All formula conversions done by hand to ensure correctness and take advantage of Justin's richer syntax.
|
|
|
|
**Conversion examples:**
|
|
```javascript
|
|
// APOSTL: Justin:
|
|
response_body(this).name response.body.name
|
|
response_code(this) == 200 statusCode == 200
|
|
status:200 statusCode == 200
|
|
T true
|
|
F false
|
|
a => b !a || b
|
|
x matches /regex/ /regex/.test(x)
|
|
for item in arr: item.price > 0 arr.every(item => item.price > 0)
|
|
exists item in arr: item.ok arr.some(item => item.ok)
|
|
previous(response_body(this)) previous.response.body
|
|
```
|
|
|
|
### Files to Delete
|
|
|
|
- `src/formula/parser.ts` (316 lines)
|
|
- `src/formula/tokenizer.ts` (170 lines)
|
|
- `src/formula/evaluator.ts` (421 lines)
|
|
- `src/formula/substitutor.ts` (75 lines)
|
|
- `src/types.ts`: `FormulaNode`, `Comparator`, `BooleanOperator`, `OperationHeader`, `OperationParameter`, `OperationCall`, `ParseResult` types
|
|
|
|
### Files to Create
|
|
|
|
- `src/formula/justin.ts` (~80 lines)
|
|
- Wraps `subscript/justin.js`
|
|
- Builds context from `EvalContext`
|
|
- Adds `matches` operator via subscript extension API
|
|
- `src/formula/context-builder.ts` (~50 lines)
|
|
- Maps `EvalContext` → flat context object
|
|
- Handles `previous` nesting
|
|
- `src/formula/justin-context.ts` (~30 lines)
|
|
- Type definitions for Justin evaluation context
|
|
|
|
### Files to Modify
|
|
|
|
- `package.json`: Add `subscript` dependency
|
|
- `src/types.ts`:
|
|
- Replace `ValidatedFormula` with `string`
|
|
- Remove APOSTL-specific types
|
|
- Add `JustinContext` interface
|
|
- `src/infrastructure/hook-validator.ts`:
|
|
- Replace `evaluateBooleanResult` with Justin evaluation
|
|
- Pre-compile formulas at registration time using `subscript`
|
|
- `src/domain/contract-validation.ts`:
|
|
- Replace APOSTL evaluation with Justin
|
|
- `src/extension/types.ts`:
|
|
- Change extension predicate API to context variables/methods
|
|
- `src/extension/registry.ts`:
|
|
- Build combined context from all extensions
|
|
- All test files with APOSTL formulas (~40 test files)
|
|
|
|
### Invariant Extraction
|
|
|
|
Despite JS syntax, we still extract logical invariants for property-based testing:
|
|
|
|
```javascript
|
|
// Formula: 'statusCode == 200 && response.body.id != null'
|
|
// Extracted invariants:
|
|
// - statusCode ∈ {200}
|
|
// - response.body.id ≠ null
|
|
// - response.body has property 'id'
|
|
|
|
// Formula: 'request.body.price * request.body.quantity <= 10000'
|
|
// Extracted invariants:
|
|
// - request.body.price is number
|
|
// - request.body.quantity is number
|
|
// - request.body.price * request.body.quantity ≤ 10000
|
|
```
|
|
|
|
### Implementation Order
|
|
|
|
1. Add `subscript` dependency
|
|
2. Create `justin.ts` and `context-builder.ts`
|
|
3. Modify `types.ts` (remove APOSTL types, add Justin types)
|
|
4. Modify `hook-validator.ts` and `contract-validation.ts`
|
|
5. Update extension API (context variables/methods)
|
|
6. Hand-convert all schema annotations in test files
|
|
7. Update all test assertions
|
|
8. Delete old parser/evaluator/tokenizer/substitutor files
|
|
9. Verify all tests pass
|
|
|
|
## Reference
|
|
|
|
- **Protocol Extensions Spec**: `docs/protocol-extensions-spec.md`
|
|
- **Plugin Contracts Spec**: `docs/PLUGIN_CONTRACTS_SPEC.md`
|
|
- **Quality Features Plan**: `docs/QUALITY_FEATURES_PLAN.md`
|
|
- **CHANGELOG**: `CHANGELOG.md`
|
|
- **Subscript/Justin**: https://github.com/dy/subscript
|