# Webhook Recurrence Prevention Report

## Summary
Implemented recurrence-prevention controls for Notion webhook outages across:
- `/Users/openclaw/.openclaw/workspace`
- `/Users/openclaw/event-router`

All changes are advisory-safe and local-only (127.0.0.1).

## Files Changed

1. **Updated** `/Users/openclaw/event-router/router.js`
   - Removed hardcoded hook token usage.
   - Added config-backed token resolver (`hooks.token` from openclaw config).
   - Added local self-test route: `POST /selftest/forward`.
   - Added self-test logging marker: `[notion] selftest_forward ...`.

2. **Added** `/Users/openclaw/.openclaw/workspace/scripts/notion_webhook_watchdog.py`
   - Incremental log-window watchdog (offset-based state).
   - Detects repeated 401/5xx forward failures.
   - Detects no-success window breach.
   - Implements suspension sentinel (endpoint reachable + extended no-success).
   - Writes artifacts:
     - Health state: `tmp/notion-webhook-watchdog/health.json`
     - Alert artifact: `tmp/notion-webhook-watchdog/alert.md`
     - State: `tmp/notion-webhook-watchdog/state.json`

3. **Added** `/Users/openclaw/.openclaw/workspace/scripts/notion_webhook_rotate_and_verify.sh`
   - Atomic backup + rotate token workflow.
   - Updates `hooks.token`, `gateway.auth.token`, `gateway.remote.token`.
   - Restarts gateway + event-router.
   - Verifies: new token 200, old token 401, router forward self-test 200.
   - Produces rollback notes + rotation report in:
     - `tmp/notion-webhook-token-rotation/`

4. **Added** `/Users/openclaw/.openclaw/workspace/docs/runbooks/notion-webhook-recurrence-prevention.md`
   - Runbook with thresholds, alert interpretation, command examples, rollback.

5. **Added + Loaded** `/Users/openclaw/Library/LaunchAgents/ai.openclaw.notion-webhook-watchdog.plist`
   - Schedule: every 600s (10 min)
   - Label: `ai.openclaw.notion-webhook-watchdog`

## Validation Results

### Router + scripts
- `node --check /Users/openclaw/event-router/router.js` → **pass**
- `bash -n /Users/openclaw/.openclaw/workspace/scripts/notion_webhook_rotate_and_verify.sh` → **pass**
- `/usr/bin/python3 /Users/openclaw/.openclaw/workspace/scripts/notion_webhook_watchdog.py` → **pass**

### Runtime checks
- Event router restart + health check:
  - `GET /health` → **200**
- Self-test forward path:
  - `POST /selftest/forward` → **200**
  - Router log contains `[notion] selftest_forward ... "openclawStatus":200`

### Watchdog artifacts
- `tmp/notion-webhook-watchdog/health.json` generated and currently reports **healthy** after self-test.
- `tmp/notion-webhook-watchdog/alert.md` generated.

### Scheduler
- `launchctl print gui/$(id -u)/ai.openclaw.notion-webhook-watchdog` shows service **running**.

## Rollback

### Router rollback
- Restore prior router source from backup if available (`router.js.bak.*`) and restart:
  - `launchctl kickstart -k gui/$(id -u)/ai.openclaw.event-router`

### Token rotation rollback
- Automatic rollback notes are generated by rotation script at:
  - `tmp/notion-webhook-token-rotation/ROLLBACK-<timestamp>.md`
- Core rollback path:
  1. Restore `openclaw.json` from `tmp/notion-webhook-token-rotation/openclaw.json.<timestamp>.bak`
  2. `openclaw gateway restart`
  3. `launchctl kickstart -k gui/$(id -u)/ai.openclaw.event-router`
  4. Re-verify new/old token behavior and self-test forward status

## Notes
- Rotation script is implemented and syntax-validated; not auto-executed in this pass to avoid unrequested live token churn.
- No secrets are printed in outputs/logs by the new scripts.