apophis-fastify/docs/attic/root-history/NEXT_STEPS_428

# NEXT_STEPS_428

Date: 2026-04-28
Scope: Protocol conformance uplift based on `docs/attic/root-history/FEEDBACK_PROTOCOL_CONFORMANCE_FROM_ARBITER.md`
Owner: APOPHIS core
Status: In progress (core protocol foundations shipped; docs and parser hardening follow-up)

## 0) Status Update (2026-04-28)

Completed in code and tests:
1. Parser/contract reliability uplift (nested conditionals, extension predicate diagnostics, parse error context).
2. `response_payload(this)` implemented in parser + evaluator + tests.
3. `contract({ variants })` implemented with deterministic variant ordering and reporting.
4. `apophis.scenario(...)` shipped with capture/rebind, cookie jar, and form-urlencoded support.
5. Scenario execution blocked in production.
6. Chaos testing remains active and integrated in contract/stateful execution.

Documentation sweep (current pass):
1. Canonical docs updated for variants/scenario/response_payload guidance.
2. Legacy/obsolete docs moved under `docs/attic/`.
3. Skill docs (`SKILL.md`, `.github/copilot/skills.md`, `skills.md`) reconciled to current API surface.
4. Subworker smoke audit executed from `/tmp/apophis-doc-audit` validating documented features against real plugin behavior.
5. Historical root markdown (feedback/plans/assessments) moved to `docs/attic/root-history/` for strict non-attic hygiene.
6. Remaining follow-up: optional deeper reconciliation for long-form extension specs.

Open protocol follow-ups:
1. ~~Route-level `x-variants` extraction~~ — **DONE**: stateful runner now collects route-level variants and runs per-variant with merged headers, deterministic seed derivation, and `[variant:name]` prefix in results. Config-level variants also supported.
2. ~~Protocol pack presets~~ — **DONE**: `packs: ['oauth21']` in config resolves built-in packs via config loader. Registry in `src/protocol-packs/index.ts`.

## 0.1) Parser Implementation Audit (Current Reality)

Current parser architecture (`src/formula/parser.ts`):
1. Hand-rolled recursive-descent parser with precedence layers (`quantified -> conditional -> boolean -> clause -> term`).
2. No Pratt parser implementation today.
3. No arena/bump allocator or typed-array-backed AST storage; AST nodes are plain JS objects.
4. Tagged unions are used at the type level (`FormulaNode.type`) rather than class polymorphism.
5. Fast-path character scanning is used heavily via `charCodeAt` and manual keyword/header matching.
6. Parse cache exists and behaves as an in-memory LRU (`Map` insertion-order eviction).
7. Operation execution cache exists for cross-operation calls in evaluator/runtime.

Not currently present:
1. Zero-copy fat pointers.
2. Ring-buffer token lookahead cache.
3. Branch prediction hints beyond current manual fast-path ordering.
4. Dedicated token stream object model.

Parser hardening/perf next ideas (post-428, measured before adoption):
1. Keep recursive-descent but isolate a tokenizer with bounded lookahead for cleaner diagnostics.
2. Replace `extensionHeaders.includes(...)` with `Set` membership in hot paths.
3. Add recursion-depth guardrails and fail-fast diagnostics for pathological nesting.
4. Add parse microbench suite (short/common, long/nested, extension-heavy) with perf budget checks.
5. Evaluate Pratt refactor only if grammar growth causes maintainability issues; performance alone is unlikely to justify a rewrite yet.

## 1) Objective for Tomorrow

Deliver a pragmatic Phase 1 protocol-testing uplift without destabilizing existing contract/stateful runners.

Critical update from `docs/attic/root-history/FEEDBACK_APOSTL_PARSER_LIMITATIONS.md`:

Before protocol expansion, parser reliability and documentation correctness must be stabilized. Current parser limitations are blocking Silver/Gold contract adoption.

Primary outcomes:
0. Fix parser and contract-validation blockers that force users back to Bronze contracts.
1. Introduce semantic payload normalization in formulas (`response_payload(this)`).
2. Add variant execution at `contract()` call-site (`contract({ variants: [...] })`) with clean reporting.
3. Land a thin scenario runner (`apophis.scenario`) with capture/rebind support.
4. Add cookie jar and first-class form-urlencoded support in scenarios.
5. Keep all current tests green and avoid breaking existing API behavior.

Non-goal for tomorrow:
- Do NOT implement full route-level `x-variants` contract extraction yet.
- Do NOT redesign core route schema model in one pass.

## 2) Constraints and Design Guardrails

1. Preserve production conformance behavior; protocol features must be additive.
2. Avoid introducing a second test engine; scenario runner should reuse existing evaluator/executor primitives.
3. Maintain deterministic behavior when seed is provided.
4. Keep modules small and focused (continue refactor direction).
5. Keep runtime safety semantics intact (no production-only behavior regressions).

## 3) Current Baseline (Confirmed)

1. Operation header parsing/evaluation is centralized and extensible:
   - `src/formula/parser.ts`
   - `src/formula/evaluator.ts`
   - `src/formula/types.ts`
2. HTTP execution is centralized and reusable:
   - `src/infrastructure/http-executor.ts`
3. Request builder currently supports JSON/multipart but not first-class form-urlencoded:
   - `src/domain/request-builder.ts`
4. Plugin decorations currently expose:
   - `contract`, `stateful`, `check`, `cleanup`, `spec`, `test`
   - `src/types.ts`, `src/plugin/index.ts`

## 4) Execution Plan (Tomorrow)

## Phase 0 — Parser and Contract Authoring Reliability (P0, blocker)

### Why
Arbiter feedback shows parser behavior currently blocks key behavioral features:

1. Extension predicates (for example `route_exists(...)`) fail when parsed in contexts lacking extension headers.
2. Nested/conditional expressions are not consistently handled for protocol-grade contracts.
3. Error messages lack route/contract clause context at parse failure time.
4. Documentation currently advertises unsupported/legacy patterns that send users in the wrong direction.

Without this phase, protocol improvements will not be adoptable at scale.

### Implementation
1. **Extension predicate parse context correctness**
   - Ensure every contract-parse call site includes extension headers where available.
   - Add explicit fallback behavior and diagnostics when a non-core header is used but not registered.
2. **Nested expression parsing + evaluation correctness**
   - Verify `if ... then route_exists(...) ... else ...` parses when extension is registered.
   - Verify cross-operation calls inside conditionals parse and evaluate.
3. **Parse error context enrichment**
   - Include route method/path and clause location (`x-requires[i]` / `x-ensures[i]`) in thrown errors.
   - Provide expression echo in diagnostics.
4. **Remove backward-compat syntax expectations**
   - Stop treating non-APOSTL legacy precondition patterns as supported contract syntax.
   - Emit actionable parser errors that point users to valid APOSTL alternatives.
5. **Documentation correction (full sweep)**
   - Remove/replace any legacy or unsupported syntax examples.
   - Clarify supported conditional/nesting patterns and extension-header requirements.
   - Document extension registration requirement for extension headers.

### Files likely touched
1. `src/infrastructure/hook-validator.ts`
2. `src/plugin/index.ts` (error context plumbing, if needed)
3. `src/domain/contract-validation.ts` (error context improvements)
4. `src/formula/parser.ts` and/or parse call sites
5. `src/test/formula.test.ts`
6. `src/test/integration.test.ts`
7. `src/test/cross-operation-support.test.ts`

### Acceptance criteria
1. `if status:200 then route_exists(this).controls.self.href == true else true` parses and evaluates when relationships extension is registered.
2. `if status:201 then response_code(GET /users/{response_body(this).id}) == 200 else true` parses and evaluates.
3. Parse failures include route + clause index + expression.
4. Legacy/non-APOSTL precondition syntax fails with explicit migration guidance (no silent compatibility mode).
5. Existing parser tests continue passing.

## Phase A — `response_payload(this)` (P0)

### Why
Enables one semantic formula across JSON and LDF responses.

### Implementation
1. Extend operation header union/types to include `response_payload`.
2. Parser: accept `response_payload` as a core header.
3. Evaluator: resolve `response_payload` as:
   - if `response.body` is object and looks like LDF wrapper with `data`, return `body.data`
   - else return `response.body`
4. Keep `response_body(this)` unchanged.

### Files likely touched
1. `src/formula/parser.ts`
2. `src/formula/evaluator.ts`
3. `src/formula/types.ts`
4. `src/domain/formula.ts` (if mirrored operation header union)
5. `src/test/formula.test.ts` (new tests)

### Acceptance criteria
1. Formula parser accepts `response_payload(this).field`.
2. Existing formulas remain unchanged.
3. New tests cover JSON, LDF-wrapper, and null/primitive body edge cases.

---

## Phase B — `contract({ variants })` execution (P0)

### Why
Runs same route under negotiated header sets without duplicating routes.

### Implementation
1. Extend `TestConfig` with:
   - `variants?: Array<{ name: string; headers?: Record<string, string> }>`
2. In contract builder/runner path:
   - If no variants, current behavior unchanged.
   - If variants provided, run contract suite once per variant.
   - Merge variant headers with scope headers for all generated requests.
3. Result naming/reporting:
   - Prefix or suffix each test name with variant marker, e.g. `[variant:json] POST /oauth/token (#1)`.
4. Ensure deterministic run ordering by variant list order.

### Files likely touched
1. `src/types.ts`
2. `src/plugin/contract-builder.ts`
3. `src/test/petit-runner.ts`
4. Possibly `src/test/petit-command-step.ts` (if header merge occurs there)
5. tests in `src/test/*` for variant reporting and header behavior

### Acceptance criteria
1. Variant runs are visible and attributable in `TestResult.name`.
2. Existing contract runs unchanged when `variants` omitted.
3. Variant headers correctly applied and override defaults when needed.

---

## Phase C — `apophis.scenario(...)` thin runner (P0)

### Why
Needed for multi-step protocol flows (OAuth authorize/token/refresh/revoke).

### Proposed API (initial)
1. Add decoration:
   - `fastify.apophis.scenario(opts)`
2. Minimal shape:
   - `name: string`
   - `steps: Array<{ name, request, expect, capture? }>`
3. Request shape:
   - `method`, `url`, `headers?`, `query?`, `body?`, `form?`
4. Expect shape:
   - array of APOSTL formulas against step context
5. Capture shape:
   - map name -> expression string (evaluated over step context)

### Execution model
1. Build step request with variable interpolation (`$stepName.captureKey`).
2. Execute via existing `executeHttp`.
3. Evaluate `expect` formulas via existing evaluator.
4. Compute captures and write into scenario store.
5. Return structured suite-like result with per-step pass/fail diagnostics.

### Files likely touched
1. `src/types.ts` (new scenario types + decoration type)
2. `src/plugin/index.ts` (decorate scenario)
3. `src/plugin/scenario-builder.ts` (new)
4. `src/test/scenario-runner.ts` (new)
5. formula/eval helpers if capture expression execution helper is needed
6. `src/test/*scenario*.test.ts` (new)

### Acceptance criteria
1. At least one OAuth-like 3-step scenario passes with capture/rebind.
2. Formula failures produce diagnostics similar quality to existing runners.
3. Scenario runner is additive; no regressions to `contract/stateful`.

---

## Phase D — Cookie jar + form-urlencoded support in scenarios (P0)

### Why
Essential for login/authorize/token flows.

### Implementation
1. Cookie jar:
   - Parse `Set-Cookie` from step responses.
   - Auto-apply matching `Cookie` header on next requests.
   - Explicit `headers.cookie` on a step overrides jar default.
2. Form-urlencoded:
   - If step has `form`, encode as URLSearchParams payload.
   - Set `content-type: application/x-www-form-urlencoded` if absent.
   - Keep `body` and `form` mutually exclusive in validation.

### Files likely touched
1. `src/test/scenario-runner.ts`
2. potentially `src/infrastructure/http-executor.ts` payload handling if required
3. `src/infrastructure/security.ts` (content-type constants, if needed)
4. new scenario tests for cookie persistence + form submission

### Acceptance criteria
1. Cookies persist across steps automatically.
2. Form token request works without custom string building.
3. Explicit cookie header override is respected.

## 5) Stretch (Only if Time Remains)

1. Add redirect helpers:
   - `redirect_query(this).0.code`
   - `redirect_fragment(this).0.access_token`
2. Add media/representation helpers:
   - `request_media_type(this)`
   - `response_media_type(this)`
   - `representation(this)`

If stretch work begins, keep it behind tests and avoid coupling to route extraction changes.

## 6) Test Plan for Tomorrow

Mandatory commands after each major phase:
1. `npm run build`
2. `npm run test:src`

Additional targeted tests to add:
0. Parser reliability tests:
   - extension predicate inside conditional (`route_exists`)
   - cross-operation call inside conditional
   - parse error context includes route and clause index
1. Parser/evaluator tests for `response_payload`.
2. Contract variant run test: verifies two variants produce variant-tagged results.
3. Scenario happy-path test with 2-3 step capture/rebind.
4. Scenario cookie jar persistence test.
5. Scenario form-urlencoded test.

## 7) Risk Register and Mitigations

1. Risk: Parser fixes regress existing formula behavior.
   - Mitigation: add explicit regression tests around extension + nested conditional parsing before feature phases.
2. Risk: Variant support causes duplicate/non-deterministic test IDs.
   - Mitigation: deterministic nested loops (variants first, then commands/routes), explicit name prefixes.
3. Risk: Scenario implementation drifts from existing diagnostics quality.
   - Mitigation: reuse existing violation/result formatting utilities where possible.
4. Risk: Cookie parsing edge cases.
   - Mitigation: minimal compliant parser for name/value + path/domain basics first, expand later.
5. Risk: Scope headers and variant headers conflict unpredictably.
   - Mitigation: define merge precedence explicitly: scope headers < variant headers < per-step headers.

## 8) Proposed Work Sequencing (Hour-by-hour)

1. Hour 1-3: Phase 0 parser reliability fixes + targeted regression tests.
2. Hour 3-4: Phase A (`response_payload`) + tests.
3. Hour 4-6: Phase B (`contract({ variants })`) + tests.
4. Hour 6-8: Phase C (scenario runner core + capture/rebind) + tests.
5. Hour 8-9: Phase D (cookie jar + form support) + tests.
6. Final hour: docs/update + full verification pass + cleanup refactor if needed.

## 9) Documentation Updates Required

1. **Full docs syntax sweep**:
   - Remove legacy/backward-compat examples that are not actually supported.
   - Replace with canonical APOSTL-only patterns.
2. `docs/getting-started.md`:
   - add brief `response_payload` example
   - add `contract({ variants })` example
   - ensure all `x-requires` examples are valid APOSTL and parser-compatible
3. `docs/protocol-extensions-spec.md`:
   - remove hard “state machine out of scope” phrasing for core scenario support
   - reference scenario API as preferred protocol composition layer
   - clarify extension predicate usage and registration prerequisites
4. `docs/chaos.md` (only if scenario/variants intersect reporting)
5. `skills.md` and `.github/copilot/skills.md`:
   - align examples with strict, current parser behavior
   - remove misleading legacy references

## 10) Definition of Done (Tomorrow)

Minimum Done:
1. Parser blockers from `docs/attic/root-history/FEEDBACK_APOSTL_PARSER_LIMITATIONS.md` addressed with tests.
2. `response_payload(this)` implemented and tested.
3. `contract({ variants })` implemented and tested.
4. `apophis.scenario(...)` implemented with capture/rebind and tested.
5. Cookie jar + form-urlencoded in scenario path implemented and tested.
6. Documentation sweep removes misleading legacy guidance.
7. `npm run build` and `npm run test:src` green.

Excellent Done:
1. Stretch redirect/media helpers included with tests.
2. docs updated for new protocol-first workflow.
3. no module exceeds intended maintainability bounds without clear follow-up notes.

## 11) Follow-up (Next After Tomorrow)

1. Route-level `x-variants` extraction + conditional variant selection (`when`).
2. Scenario runner integration with flake/chaos profile presets.
3. Protocol packs (`oauth21ProfilePack`, RFC-specific packs) built on scenario+variants+payload.

## 12) Everything Else for 428 (Full Impact Inventory)

This section captures cross-cutting tasks that are easy to miss but required for a complete 428 delivery.

### A) API Surface and Type System Touchpoints

1. Extend `TestConfig` to include `variants` without weakening existing typing contracts.
2. Add scenario request/result types to `src/types.ts` (step input, capture map, scenario summary).
3. Extend `ApophisDecorations` in `src/types.ts` to include `scenario`.
4. Ensure decoration typing stays aligned with `src/plugin/index.ts` and builder signatures.
5. Keep existing public method call sites (`contract`, `stateful`, `check`) stable while removing legacy contract syntax expectations.

### B) Formula Runtime and Developer Ergonomics

1. Add `response_payload` support in:
   - parser core headers list
   - evaluator operation resolution
   - compile-time formula header unions (`src/formula/types.ts`, `src/domain/formula.ts`).
2. Update formula diagnostics helpers to recognize new operation tokens:
   - `src/domain/contract-validation.ts` field extraction regexes
   - `src/domain/error-suggestions.ts` matchers/regexes.
3. Add parser error-suggestion coverage for any new operation headers.

### C) Contract Runner and Variant Execution Details

1. Variant header merge precedence must be explicit and tested.
2. Variant naming in output should remain TAP-friendly and dedup-safe.
3. Deduplication logic should include variant identity in route key to avoid false suppression.
4. Seed behavior should be deterministic per variant (stable ordering + seed derivation strategy).
5. Ensure `routes` filtering still works with variants.

### D) Scenario Engine Execution Details

1. Variable interpolation semantics:
   - `$step.capture` in URL
   - headers/query/body/form substitution
   - clear failure mode when reference is missing.
2. Capture expression evaluation should use same evaluator semantics as contracts.
3. Step failure output should include request/response context and formula info.
4. Scenario should stop-on-failure by default (or documented mode if configurable).
5. Add deterministic scenario test mode with `seed` where generation exists.

### E) Cookie Jar + Form Behavior Edge Cases

1. Preserve multiple cookies and cookie updates (same key replacement semantics).
2. Support explicit cookie override per step.
3. If both `body` and `form` provided, fail fast with clear error.
4. Ensure form encoding works with string/number/boolean values consistently.
5. Add `CONTENT_TYPE.FORM_URLENCODED` constant in `src/infrastructure/security.ts`.

### F) Infrastructure and Safety

1. Evaluate whether `scenario` is test-only or allowed in non-prod; enforce policy consistently.
2. If test-only, wire through `assertTestEnv` and document behavior.
3. If not test-only, still ensure no production safety violations are introduced.
4. Keep runtime hook production gating unchanged.

### G) Tests to Add Beyond Core Happy Paths

1. `response_payload` tests for:
   - plain JSON
   - LDF with `data`
   - non-object/null body fallback.
2. Variant tests for:
   - per-variant header injection
   - per-variant naming
   - dedup correctness with variants.
3. Scenario tests for:
   - capture from headers/body/redirects
   - missing capture reference error
   - cookie jar persistence and override
   - form-urlencoded token step.
4. Regression tests ensuring old APIs still behave unchanged.

### H) Documentation and Messaging Updates

1. Update protocol docs to replace strict “state machines out of scope” language.
2. Add canonical OAuth bilingual example using:
   - `contract({ variants })`
   - shared formulas with `response_payload(this)`.
3. Add scenario cookbook section:
   - login -> authorize -> token -> refresh minimal flow.
4. Keep README and getting-started aligned with new API surface.

### I) Acceptance and Exit Checklist for Issue 428

1. All 535 source tests remain green after each phase.
2. New tests cover all added API shapes and critical failure modes.
3. No existing public APIs are broken.
4. Docs and type definitions reflect final behavior.
5. Changelog/release notes prepared for protocol-conformance capabilities.