apophis-fastify/docs/qualify.md

Qualify Mode

Run scenario, stateful, and chaos checks against non-production Fastify services.

Qualify extends the invariant-driven approach from Invariant-Driven Automated Testing (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.

What Qualify Does

apophis qualify runs deeper testing than verify:

• Scenario execution: Multi-step protocol flows with capture/rebind
• Stateful testing: Constructor/mutator/observer/destructor sequences
• Chaos engineering: Controlled fault injection
• Adversity checks: Failure-path and edge-case validation

When to Use It

• Nightly CI: Scenario and stateful checks for critical flows
• Staging: Protocol flow validation before production
• Specialist teams: Auth, billing, workflow systems

Scenario Examples

OAuth Flow

profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Run with: apophis qualify --profile oauth-nightly --seed 42

Lifecycle Deep

profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Scenario Definitions

Scenarios are multi-step flows with capture and rebind:

await fastify.apophis.scenario({
  name: 'oauth-basic',
  steps: [
    {
      name: 'authorize',
      request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code' },
      expect: ['status:200', 'response_payload(this).code != null'],
      capture: { code: 'response_payload(this).code' }
    },
    {
      name: 'token',
      request: {
        method: 'POST',
        url: '/oauth/token',
        form: { grant_type: 'authorization_code', code: '$authorize.code' }
      },
      expect: ['status:200', 'response_payload(this).access_token != null']
    }
  ]
})

Scenario behavior:

1. Cookie jar persists Set-Cookie values across steps.
2. Step-level headers.cookie overrides jar values for that step.
3. form sends application/x-www-form-urlencoded payloads.

Stateful Testing

Stateful tests generate sequences of operations and track resources:

1. Constructor: Create resources (POST)
2. Mutator: Modify resources (PUT, PATCH)
3. Observer: Read resources (GET)
4. Destructor: Remove resources (DELETE)

APOPHIS tracks created resources and runs cleanup after test completion.

Run stateful tests via the API:

const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
console.log('Stateful tests:', stateful.summary)

Route Transparency

Artifacts include executedRoutes and skippedRoutes arrays. skippedRoutes contains reasons such as mode mismatch, environment policy, or route filter exclusion.

Chaos and Adversity

Chaos testing injects controlled failures:

• Delay: Slow responses
• Error: Return error status codes
• Dropout: Connection failures
• Truncate: Truncated response bodies
• Malformed: Invalid JSON or content-type
• Field-corrupt: Random field mutation in response objects

Configure chaos in your preset:

presets: {
  'protocol-lab': {
    name: 'protocol-lab',
    depth: 'deep',
    timeout: 15000,
    parallel: false,
    chaos: true,
    observe: false
  }
}

Non-Prod Boundaries

Qualify mode is gated away from production by default:

Environment	Scenario	Stateful	Chaos
local	enabled	enabled	enabled
test/CI	enabled	enabled	enabled
staging	enabled with allowlist	enabled	blocked on protected routes
production	disabled by default	disabled by default	disabled by default

Machine Output for CI

Qualify can produce large output. Use machine-readable formats and event filtering to keep CI logs manageable:

Concise formats

• --format json-summary — emits a single JSON document with summary, failures, and warnings. Omits per-step traces and cleanup outcomes.
• --format ndjson-summary — emits three NDJSON lines: run.started, run.summary, run.completed. No per-route events.

Filtering examples

# Extract only failed routes from full ndjson
apophis qualify --profile oauth-nightly --format ndjson | jq 'select(.type == "route.failed")'

# Write artifact to disk and parse the file instead of stdout
apophis qualify --profile oauth-nightly --format json --artifact-dir reports/apophis

Recommended CI retention strategy

• Keep artifacts for 30 days in CI storage (S3, GCS, Artifactory).
• Use --artifact-dir to write artifacts automatically.
• Parse json-summary output for dashboards; keep full json artifacts for debugging.

Exit Codes

Code	Meaning
0	All qualifications passed
1	One or more qualifications failed
2	Safety violation or invalid config
3	Internal APOPHIS error
130	Interrupted (SIGINT)

Config Example

// apophis.config.js
export default {
  mode: 'qualify',
  profile: 'oauth-nightly',
  profiles: {
    'oauth-nightly': {
      name: 'oauth-nightly',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    },
    'lifecycle-deep': {
      name: 'lifecycle-deep',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    }
  },
  presets: {
    'protocol-lab': {
      name: 'protocol-lab',
    runs: 200,
      timeout: 15000,
      parallel: false,
      chaos: true,
      observe: false
    }
  },
  environments: {
    local: {
      name: 'local',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    test: {
      name: 'test',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    staging: {
      name: 'staging',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: false,
      allowBlocking: false,
      requireSink: true
    }
  }
};

Gate Execution Counts

Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.

Zero-Execution Guardrail

Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.

--workspace Flag

Run qualify across all packages in a monorepo workspace:

apophis qualify --workspace --profile oauth-nightly

Test Budget

The runs field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:

presets: {
  'protocol-lab': {
    runs: 200,
    timeout: 15000
  }
}