chore: crush git history - reborn from consolidation on 2026-03-10
This commit is contained in:
John Dvorak
@@ -0,0 +1,549 @@
|
||||
# APOPHIS v1.1 Architecture — Hybrid Core + Extensions
|
||||
|
||||
## Status: Architecture Specification
|
||||
## Date: 2026-04-24
|
||||
## Scope: v1.1 First-Class Features & Extension Ecosystem
|
||||
|
||||
---
|
||||
|
||||
## 1. Philosophy: Core HTTP vs Extensions
|
||||
|
||||
**First-class**: Standard HTTP features that require deep integration with APOPHIS core:
|
||||
- Schema-to-arbitrary integration (teaching fast-check to generate custom data)
|
||||
- Request builder integration (constructing specialized payloads)
|
||||
- HTTP executor integration (handling specialized responses)
|
||||
- APOSTL parser/evaluator integration (new operations)
|
||||
|
||||
**Extensions**: Specialized protocols or features with heavy dependencies that should be opt-in:
|
||||
- Different protocols (WebSockets, not HTTP)
|
||||
- Heavy dependencies (Protobuf, MessagePack)
|
||||
- Protocol-specific features such as SSE
|
||||
|
||||
**This split keeps common HTTP testing in core while moving specialized protocols out of the default path.**
|
||||
|
||||
---
|
||||
|
||||
## 2. First-Class Features (v1.1 Core)
|
||||
|
||||
### 2.1 Multipart File Uploads
|
||||
|
||||
**Module**: Core — `src/infrastructure/multipart.ts`, `src/domain/multipart-generator.ts`
|
||||
|
||||
**Schema Annotations**:
|
||||
```typescript
|
||||
schema: {
|
||||
body: {
|
||||
type: 'object',
|
||||
'x-content-type': 'multipart/form-data',
|
||||
'x-multipart-fields': {
|
||||
description: { type: 'string', maxLength: 500 }
|
||||
},
|
||||
'x-multipart-files': {
|
||||
avatar: {
|
||||
maxSize: 5 * 1024 * 1024,
|
||||
mimeTypes: ['image/jpeg', 'image/png'],
|
||||
maxCount: 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**APOSTL Operations**:
|
||||
```typescript
|
||||
// request_files(this).avatar.count == 1
|
||||
// request_files(this).avatar.size <= 5242880
|
||||
// request_files(this).avatar.mimetype matches "image/(jpeg|png)"
|
||||
// request_fields(this).description != null
|
||||
```
|
||||
|
||||
**Core Integration Points**:
|
||||
1. **Schema-to-arbitrary**: Detect `x-content-type: multipart/form-data`, generate `{ fields: {...}, files: [...] }`
|
||||
2. **Request builder**: Convert generated data to `multipart` payload on `RequestStructure`
|
||||
3. **HTTP executor**: Build `FormData` from `request.multipart`, inject via Fastify
|
||||
4. **Parser**: Add `request_files`, `request_fields` to `VALID_HEADERS`
|
||||
5. **Evaluator**: Add multipart operations to `resolveOperation`
|
||||
|
||||
### 2.2 Streaming / NDJSON
|
||||
|
||||
**Module**: Core — `src/infrastructure/stream-collector.ts`
|
||||
|
||||
**Schema Annotations**:
|
||||
```typescript
|
||||
schema: {
|
||||
response: {
|
||||
200: {
|
||||
type: 'object',
|
||||
'x-streaming': true,
|
||||
'x-stream-format': 'ndjson',
|
||||
'x-stream-max-chunks': 100,
|
||||
'x-stream-timeout': 5000
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**APOSTL Operations**:
|
||||
```typescript
|
||||
// response_body(this) — array of parsed chunks
|
||||
// stream_chunks(this) — alias for response_body(this)
|
||||
// stream_duration(this) — total stream time in ms
|
||||
```
|
||||
|
||||
**Core Integration Points**:
|
||||
1. **Contract extraction**: Extract `x-streaming`, `x-stream-format`, `x-stream-max-chunks`, `x-stream-timeout`
|
||||
2. **HTTP executor**: After inject, check if route has streaming config. If so:
|
||||
- Read response payload as string
|
||||
- Split by `\n`
|
||||
- `JSON.parse` each line (for NDJSON)
|
||||
- Respect `maxChunks` and `timeoutMs`
|
||||
- Store result in `EvalContext.response.body` and `EvalContext.response.chunks`
|
||||
3. **Parser**: Add `stream_chunks`, `stream_duration` to `VALID_HEADERS`
|
||||
4. **Evaluator**: Add streaming operations to `resolveOperation`
|
||||
|
||||
---
|
||||
|
||||
## 3. Extension System (v1.1+ Ecosystem)
|
||||
|
||||
The extension system handles features that don't require core HTTP integration.
|
||||
|
||||
### 3.1 Extension Interface
|
||||
|
||||
```typescript
|
||||
export interface ApophisExtension {
|
||||
/** Unique name. Used for state isolation and error attribution. */
|
||||
name: string
|
||||
|
||||
/** APOSTL headers this extension adds. Used for parser validation. */
|
||||
headers?: string[]
|
||||
|
||||
/** APOSTL predicates exposed by this extension. */
|
||||
predicates?: Record<string, PredicateResolver>
|
||||
|
||||
/** Lifecycle hooks. */
|
||||
hooks?: {
|
||||
onBuildRequest?: Hook<RequestBuildContext, void>
|
||||
onBeforeRequest?: Hook<ExecutionContext, void>
|
||||
onAfterRequest?: Hook<ExecutionContext, void>
|
||||
onSuiteStart?: Hook<{ routes: RouteContract[] }, void>
|
||||
onSuiteEnd?: Hook<{ summary: TestSummary }, void>
|
||||
onViolation?: Hook<{ violation: ContractViolation }, void>
|
||||
}
|
||||
|
||||
/** Severity: 'fatal' (block test), 'warn' (log, don't block). Default: 'fatal'. */
|
||||
severity?: 'fatal' | 'warn'
|
||||
|
||||
/** Redaction: fields to mask in violation output. */
|
||||
redactFields?: string[]
|
||||
|
||||
/** Initial state for this extension. Passed to hooks/predicates. */
|
||||
state?: Record<string, unknown>
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Extension Registration
|
||||
|
||||
```typescript
|
||||
await fastify.register(apophis, {
|
||||
extensions: [
|
||||
sseExtension,
|
||||
createSerializerExtension(mySerializerRegistry),
|
||||
websocketExtension,
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
### 3.3 Extensions Available
|
||||
|
||||
#### SSE Extension
|
||||
**Module**: `src/extensions/sse/`
|
||||
|
||||
```typescript
|
||||
export const sseExtension: ApophisExtension = {
|
||||
name: 'sse',
|
||||
headers: ['sse_events'],
|
||||
predicates: {
|
||||
sse_events: (ctx) => {
|
||||
const events = ctx.evalContext.response.sseEvents ?? []
|
||||
if (ctx.accessor.length === 0) return { value: events, success: true }
|
||||
|
||||
const idx = parseInt(ctx.accessor[0], 10)
|
||||
const event = events[idx]
|
||||
if (!event) return { value: null, success: true }
|
||||
|
||||
if (ctx.accessor[1] === 'event') return { value: event.event, success: true }
|
||||
if (ctx.accessor[1] === 'data') return { value: event.data, success: true }
|
||||
if (ctx.accessor[1] === 'id') return { value: event.id, success: true }
|
||||
if (ctx.accessor[1] === 'retry') return { value: event.retry, success: true }
|
||||
|
||||
return { value: event, success: true }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Serializers Extension
|
||||
**Module**: `src/extensions/serializers/`
|
||||
|
||||
```typescript
|
||||
export interface Serializer {
|
||||
readonly name: string
|
||||
encode(data: unknown): Buffer
|
||||
decode(buffer: Buffer): unknown
|
||||
}
|
||||
|
||||
export interface SerializerRegistry {
|
||||
get(name: string): Serializer | undefined
|
||||
register(name: string, serializer: Serializer): void
|
||||
}
|
||||
|
||||
export const createSerializerExtension = (registry: SerializerRegistry): ApophisExtension => ({
|
||||
name: 'serializers',
|
||||
hooks: {
|
||||
onBuildRequest: async (ctx) => {
|
||||
const serializerName = ctx.route.serializer?.name
|
||||
if (!serializerName) return
|
||||
|
||||
const serializer = registry.get(serializerName)
|
||||
if (!serializer) return
|
||||
|
||||
// Modify request: encode body, set content-type
|
||||
ctx.request.body = serializer.encode(ctx.request.body)
|
||||
ctx.request.headers = {
|
||||
...ctx.request.headers,
|
||||
'content-type': `application/x-${serializerName}`,
|
||||
}
|
||||
},
|
||||
onAfterRequest: async (ctx) => {
|
||||
const serializerName = ctx.route.serializer?.name
|
||||
if (!serializerName) return
|
||||
|
||||
const serializer = registry.get(serializerName)
|
||||
if (!serializer) return
|
||||
|
||||
// Modify response: decode body
|
||||
const rawBody = Buffer.from(JSON.stringify(ctx.evalContext.response.body))
|
||||
ctx.evalContext.response.body = serializer.decode(rawBody)
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### WebSockets Extension
|
||||
**Module**: `src/extensions/websocket/`
|
||||
|
||||
**Note**: WebSockets are fundamentally different from HTTP. They require a dedicated runner, not just hooks.
|
||||
|
||||
```typescript
|
||||
export const websocketExtension: ApophisExtension = {
|
||||
name: 'websocket',
|
||||
headers: ['ws_message', 'ws_state'],
|
||||
predicates: {
|
||||
ws_message: (ctx) => {
|
||||
const msg = ctx.evalContext.ws?.message ?? null
|
||||
if (ctx.accessor.length === 0) return { value: msg, success: true }
|
||||
if (!msg) return { value: null, success: true }
|
||||
|
||||
if (ctx.accessor[0] === 'type') return { value: msg.type, success: true }
|
||||
if (ctx.accessor[0] === 'payload') return { value: msg.payload, success: true }
|
||||
if (ctx.accessor[0] === 'direction') return { value: msg.direction, success: true }
|
||||
|
||||
return { value: msg, success: true }
|
||||
},
|
||||
ws_state: (ctx) => {
|
||||
return { value: ctx.evalContext.ws?.state ?? null, success: true }
|
||||
}
|
||||
},
|
||||
hooks: {
|
||||
onSuiteStart: async ({ routes }) => {
|
||||
// Pre-validate all WS contracts
|
||||
const wsRoutes = routes.filter(r => r.ws !== undefined)
|
||||
for (const route of wsRoutes) {
|
||||
validateWebSocketContract(route.ws!)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**WebSocket runner**: Invoked by plugin separately from HTTP runners:
|
||||
```typescript
|
||||
// In plugin/index.ts
|
||||
const buildContract = (fastify, scope) => async (opts) => {
|
||||
const httpSuite = await runPetitTests(fastify, opts, scope)
|
||||
const wsSuite = await runWebSocketTests(fastify, opts, scope) // From extension
|
||||
return mergeSuites(httpSuite, wsSuite)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Changes (Phase 1)
|
||||
|
||||
### 4.1 Parser Extensibility
|
||||
|
||||
**Current**: `VALID_HEADERS` is hardcoded. Extensions can't add headers.
|
||||
|
||||
**Solution**: Extensions register headers. Parser validates against registered + core headers.
|
||||
|
||||
```typescript
|
||||
// src/formula/parser.ts
|
||||
const CORE_HEADERS: OperationHeader[] = [
|
||||
'request_body', 'response_body', 'response_code',
|
||||
'request_headers', 'response_headers', 'query_params', 'cookies', 'response_time',
|
||||
'redirect_count', 'redirect_url', 'redirect_status',
|
||||
'timeout_occurred', 'timeout_value',
|
||||
// v1.1 first-class
|
||||
'request_files', 'request_fields', 'stream_chunks', 'stream_duration',
|
||||
]
|
||||
|
||||
// ExtensionRegistry provides additional headers
|
||||
function getValidHeaders(registry?: ExtensionRegistry): string[] {
|
||||
const extensionHeaders = registry
|
||||
? registry.extensions.flatMap(e => e.headers ?? [])
|
||||
: []
|
||||
return [...CORE_HEADERS, ...extensionHeaders]
|
||||
}
|
||||
|
||||
// In parseOperation, validate against getValidHeaders()
|
||||
```
|
||||
|
||||
### 4.2 Evaluator Extensibility
|
||||
|
||||
**Current**: `resolveOperation` checks core operations only.
|
||||
|
||||
**Solution**: Check extension predicates BEFORE core operations.
|
||||
|
||||
```typescript
|
||||
function resolveOperation(node, ctx, extensionRegistry, route) {
|
||||
const { header, accessor } = node
|
||||
|
||||
// 1. Check extension predicates FIRST
|
||||
if (extensionRegistry) {
|
||||
const resolver = extensionRegistry.resolvePredicate(header)
|
||||
if (resolver) {
|
||||
const ownerName = extensionRegistry.getPredicateOwner(header)
|
||||
const extState = ownerName ? (extensionRegistry.getState(ownerName) ?? {}) : {}
|
||||
const result = resolver({ route, evalContext: ctx, accessor: accessor ?? [], extensionState: extState })
|
||||
if (result && typeof result.then !== 'function') {
|
||||
return (result as PredicateResult).value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Fall back to core operations
|
||||
switch (header) {
|
||||
// ... core cases ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 HTTP Executor Hooks
|
||||
|
||||
**Current**: `executeHttp` is a monolithic function.
|
||||
|
||||
**Solution**: Add `onTransformResponse` hook point for extensions that need to modify responses.
|
||||
|
||||
```typescript
|
||||
export interface ResponseTransformContext {
|
||||
responseBody: unknown
|
||||
evalContext: EvalContext
|
||||
route: RouteContract
|
||||
}
|
||||
|
||||
export type ResponseTransformHook = (ctx: ResponseTransformContext) => EvalContext | Promise<EvalContext>
|
||||
|
||||
// In executeHttp:
|
||||
let ctx = buildEvalContext(request, response, route)
|
||||
|
||||
// Apply extension response transforms
|
||||
for (const ext of (extensionRegistry?.extensions ?? [])) {
|
||||
if (ext.hooks?.onAfterRequest) {
|
||||
await ext.hooks.onAfterRequest({
|
||||
route,
|
||||
request,
|
||||
evalContext: ctx,
|
||||
extensionState: extensionRegistry?.getState(ext.name) ?? {},
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Implementation Order
|
||||
|
||||
### Phase 1: Core Extension Points (1-2 days)
|
||||
1. Make parser accept registered headers (CORE_HEADERS + extension headers)
|
||||
2. Make evaluator check extension predicates before core operations
|
||||
3. Add response transform hook point to HTTP executor
|
||||
4. **Test**: Core operations still work; extension predicates resolve
|
||||
|
||||
### Phase 2A: Multipart (First-Class, 2-3 days)
|
||||
1. Add `MultipartFile`, `MultipartPayload` types
|
||||
2. Add multipart schema-to-arbitrary handler
|
||||
3. Add multipart request builder support
|
||||
4. Add multipart HTTP executor support (FormData construction)
|
||||
5. Add `request_files`, `request_fields` to parser/evaluator
|
||||
6. Extract multipart config from schema in contract.ts
|
||||
7. **Test**: `src/test/multipart.test.ts` (10+ tests)
|
||||
|
||||
### Phase 2B: Streaming (First-Class, 2-3 days)
|
||||
1. Add `chunks`, `streamDurationMs` to `EvalContext.response`
|
||||
2. Add streaming config extraction from schema
|
||||
3. Add stream collection to HTTP executor (NDJSON parsing)
|
||||
4. Add `stream_chunks`, `stream_duration` to parser/evaluator
|
||||
5. **Test**: `src/test/streaming.test.ts` (8+ tests)
|
||||
|
||||
### Phase 2C: Extension System Polish (1 day)
|
||||
1. Document extension registration API
|
||||
2. Add `extensions: ApophisExtension[]` to `ApophisOptions`
|
||||
3. Wire extension headers into parser
|
||||
4. Wire extension predicates into evaluator
|
||||
|
||||
### Phase 3: Extensions (Parallel, after Phase 2C)
|
||||
- **SSE Extension** (2-3 days)
|
||||
- **Serializers Extension** (2-3 days)
|
||||
- **WebSockets Extension** (1-2 weeks)
|
||||
|
||||
### Phase 4: Integration (2-3 days)
|
||||
1. Run full test suite
|
||||
2. Update README
|
||||
3. Verify benchmarks
|
||||
|
||||
---
|
||||
|
||||
## 6. File Layout
|
||||
|
||||
```
|
||||
src/
|
||||
# Core v1.1 First-Class Features
|
||||
infrastructure/
|
||||
http-executor.ts # ADD: multipart FormData, stream collection
|
||||
multipart.ts # NEW: FormData construction
|
||||
stream-collector.ts # NEW: NDJSON chunk parsing
|
||||
domain/
|
||||
schema-to-arbitrary.ts # ADD: multipart schema handler
|
||||
request-builder.ts # ADD: multipart payload construction
|
||||
contract.ts # ADD: multipart/streaming config extraction
|
||||
formula/
|
||||
parser.ts # MODIFY: extensible VALID_HEADERS
|
||||
evaluator.ts # MODIFY: extension predicate check
|
||||
types.ts # ADD: MultipartFile, MultipartPayload, stream fields
|
||||
|
||||
# Extension System
|
||||
extension/
|
||||
types.ts # ADD: headers, onTransformResponse to interface
|
||||
registry.ts # ADD: collect extension headers
|
||||
|
||||
# Extensions (opt-in)
|
||||
extensions/
|
||||
sse/ # SSE extension module
|
||||
serializers/ # Serializer extension module
|
||||
websocket/ # WebSocket extension module
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Test Strategy
|
||||
|
||||
### First-Class Features: Red-Green-Refactor
|
||||
|
||||
```typescript
|
||||
// Example: Multipart
|
||||
// 1. Test: Parser accepts request_files(this).avatar.size
|
||||
// 2. Implement: Add request_files to VALID_HEADERS
|
||||
// 3. Test: Evaluator resolves request_files
|
||||
// 4. Implement: Add multipart operations to resolveOperation
|
||||
// 5. Test: Schema-to-arbitrary generates fake files
|
||||
// 6. Implement: Add multipart handler to convertSchemaInternal
|
||||
// 7. Test: Request builder constructs multipart payload
|
||||
// 8. Implement: Add multipart support to buildRequest
|
||||
// 9. Test: HTTP executor sends multipart request
|
||||
// 10. Implement: Build FormData in executeHttp
|
||||
// 11. Test: Integration — upload route works end-to-end
|
||||
// 12. Implement: Full flow
|
||||
```
|
||||
|
||||
### Extensions: Self-Contained Tests
|
||||
|
||||
Each extension module has its own `test.ts`:
|
||||
|
||||
```typescript
|
||||
// src/extensions/sse/test.ts
|
||||
import { test } from 'node:test'
|
||||
import assert from 'node:assert'
|
||||
import { sseExtension } from './extension.js'
|
||||
|
||||
test('sse: predicate returns events', () => {
|
||||
const resolver = sseExtension.predicates!.sse_events
|
||||
const result = resolver({
|
||||
route: mockRoute,
|
||||
evalContext: { response: { sseEvents: [{ event: 'update', data: {} }] } },
|
||||
accessor: [],
|
||||
extensionState: {},
|
||||
})
|
||||
assert.strictEqual((result.value as any[]).length, 1)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Backward Compatibility
|
||||
|
||||
All v1.1 changes are additive:
|
||||
- Routes without multipart/streaming annotations work unchanged
|
||||
- Extensions are opt-in via `extensions: [...]` option
|
||||
- Existing APOSTL formulas work unchanged
|
||||
- No breaking changes to public API
|
||||
|
||||
**Migration path**:
|
||||
```typescript
|
||||
// v1.0
|
||||
await fastify.register(apophis)
|
||||
|
||||
// v1.1 (no changes required for existing code)
|
||||
await fastify.register(apophis)
|
||||
|
||||
// v1.1 with extensions
|
||||
await fastify.register(apophis, {
|
||||
extensions: [sseExtension, serializerExtension, websocketExtension]
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Parser changes break existing formulas | Comprehensive regression tests before parser modification |
|
||||
| Multipart adds heavy deps | Only use native FormData/Blob (no external deps) |
|
||||
| Streaming tests are flaky | Mock streams for unit tests; integration tests with deterministic timeouts |
|
||||
| Extension conflicts | Namespacing by extension name; `ExtensionRegistry.getState(name)` isolates state |
|
||||
| WebSocket extension too large | Split into sub-workstreams: client, runner, stateful, validation |
|
||||
|
||||
---
|
||||
|
||||
## 10. Success Criteria
|
||||
|
||||
| Criterion | Verification |
|
||||
|-----------|-------------|
|
||||
| Multipart upload routes tested | `multipart.test.ts` passes |
|
||||
| Streaming routes tested | `streaming.test.ts` passes |
|
||||
| Extension predicates work | Extension `test.ts` files pass |
|
||||
| No regression | Full source and CLI test suites pass |
|
||||
| Benchmark targets met | `benchmark.test.ts` passes |
|
||||
| Documentation updated | README covers multipart and streaming |
|
||||
|
||||
---
|
||||
|
||||
## 11. Quick Reference: First-Class vs Extension
|
||||
|
||||
| Feature | Type | Core Files | Tests | Effort |
|
||||
|---------|------|-----------|-------|--------|
|
||||
| **Multipart** | First-class | `multipart.ts`, `schema-to-arbitrary.ts`, `request-builder.ts`, `http-executor.ts`, `parser.ts`, `evaluator.ts` | `multipart.test.ts` | 2-3 days |
|
||||
| **Streaming** | First-class | `stream-collector.ts`, `http-executor.ts`, `parser.ts`, `evaluator.ts`, `contract.ts` | `streaming.test.ts` | 2-3 days |
|
||||
| **SSE** | Extension | `src/extensions/sse/*` | `src/extensions/sse/test.ts` | 2-3 days |
|
||||
| **Serializers** | Extension | `src/extensions/serializers/*` | `src/extensions/serializers/test.ts` | 2-3 days |
|
||||
| **WebSockets** | Extension | `src/extensions/websocket/*` | `src/extensions/websocket/test.ts` | 1-2 weeks |
|
||||
Reference in New Issue
Block a user