atlas-gw Reliability Playbook — auto-restart + boot resilience

Created: 2026-06-07 NZST · session 23ef0e09-0ae0-45b3-a33e-a9a40c82c50e Status: ✅ Live — both layers deployed and end-to-end tested Read this when: any work touching vm-atlas-gw-01 lifecycle, the openclaw-gateway systemd unit, the Resource Health alerts, the Action Group, the Automation Runbook, OR when aunty goes silent overnight and you need to know what should have auto-recovered.

---

🦞 TL;DR (60 seconds)

The atlas-gw VM in the CDX tenant is periodically auto-shut-down by Azure platform/CDX-lab automation (~every 8–12h). Until 2026-06-07 there was NO recovery — aunty stayed silent until Sush manually az vm start-ed the next morning. As of this playbook, two complementary fixes keep her alive:

1. Layer 1 — VM auto-restart (Azure-side): Resource Health alert → existing Action Group → Automation Runbook (Start-AtlasGw) → Start-AzVM. End-to-end recovery: 3–7 min. Cost: $0/month (within Automation Account 500-free-min/mo + free Activity Log alerts).
2. Layer 2 — Gateway boot resilience (in-guest): systemd drop-in cold-boot-retry.conf extends restart budget from 10 retries in 5 min → 20 retries in 30 min. Survives the KV/MI warmup race on fresh boot.

If aunty is silent → check this playbook's "Diagnose silent aunty" section. Do NOT reflexively rebuild anything — both layers above almost certainly already tried to recover.

---

⚠️ Don't repeat this mistake (Rule #5 retrospective)

Before this session, multiple Atlas sessions across 2+ days kept trying to "fix it" instance-by-instance: manual VM start, reactive WA debugging, even a half-built custom Windows PowerShell watcher running on Sush's laptop. The correct fix took 90 min once the architecture gate fired — Resource Health → Action Group → Automation Runbook is the canonical Azure pattern with a battle-tested open-source Terraform reference (jmassardo/octowatch). The 2 prior nights of family silence and the 90 min of churn THIS session was tuition for not invoking Rule #5 sooner.

If you (future Atlas) find yourself building a custom watcher / PowerShell script / cron on the user's laptop to wake an Azure VM — STOP. The Azure-native pattern is in this playbook. It's already built. Use it.

---

1. The bug (what was actually breaking)

1a. Symptom (what Sush saw)

• 2 nights running (5–6 Jun and 6–7 Jun 2026): aunty silent in WhatsApp overnight while family (Chennai, Oman, Canada) was awake
• VM appeared "down" each morning until manually az vm start-ed

1b. Root cause #1 — Azure host shuts down the VM

• Kernel log shows: Jun 06 05:19:13 vm-atlas-gw-01 kernel: hv_utils: Shutdown request received - graceful shutdown initiated
• This is the Hyper-V host issuing a graceful shutdown to the guest
• Source: CDX-tenant lab lifecycle policy (undocumented publicly; opting out requires internal aka.ms/cdx-support ticket)
• Pattern observed: ~8–12 h between shutdowns (3 reboots in 36h on 5–6 Jun)
• No DevTest Labs schedules, no Automation Accounts, no Logic Apps, no Function Apps existed in the subscription before this playbook — so NOTHING was bringing the VM back up

1c. Root cause #2 — Even when VM came back, gateway didn't start

On 2026-06-06 12:46 UTC boot:

12:46:40 systemd starts openclaw-gateway       → fails with status=243/CREDENTIALS
12:46:55 retry #1                               → SAME error (KV/MI not warm yet)
12:47:10 retry #2                               → SAME error
12:47:22 retry #3                               → SAME error
12:47:37 retry #4                               → SAME error
~12:47:53 systemd hit StartLimitBurst=10 in 5min → GAVE UP
gateway dead 8h22min until 21:08 UTC manual `systemctl start`

The failure mode: openclaw-gateway uses systemd LoadCredential= to pull a Key Vault secret via Managed Identity at startup. On a fresh boot, KV+MI take ~1–2 min to warm up, but the original drop-in (cold-boot-retry.conf by predecessor Atlas) only allowed 10 retries in 5 min → systemd burned through all attempts in <60s before KV was ready.

---

2. The fix (what's deployed)

2a. Layer 1 — VM auto-restart (Azure-side, this session)

hv_utils Shutdown signal → guest OS shuts down → Azure marks VM Unavailable
   ↓
Activity Log alert "alert-vm-resource-health-unavailable" fires (1–5 min)
   condition: category=ResourceHealth AND properties.currentHealthStatus=Unavailable
   scope: vm-atlas-gw-01 (precise — no other VM triggers it)
   ↓
ag-atlas-gateway-prod (existing Action Group, now has webhook + email receivers)
   ↓ HTTPS POST to webhook URI (one-time visible)
Automation Account aa-atlas-gw-restarter
   Runbook: Start-AtlasGw (PowerShell 7.2, ~70 lines)
   Identity: UAMI id-atlas-gw-restarter (Client ID: c34ca133-90ad-4884-a546-551c5d107a9a)
   Role: Virtual Machine Contributor SCOPED TO VM ONLY (least privilege)
   ↓
Runbook flow:
   1. Connect-AzAccount -Identity -AccountId <UAMI client id>
   2. Get-AzAccessToken (ARM scope)
   3. GET /vm/instanceView via Invoke-RestMethod → check current PowerState
   4. If 'running' → exit clean (idempotent — safe to fire repeatedly)
   5. If stopped/deallocated → POST /vm/start
   6. Poll instanceView every 15s for up to 4 min until 'running'
   ↓
VM boots → systemd starts openclaw-gateway with boot-resilience override (Layer 2)
   ↓
Aunty back online — total recovery 3–7 min

Resources created (Sub: 96879ea6-389e-417f-a3a2-16c415a2b6b5 / RG: RG-ATLAS-GATEWAY-PROD)

Resource	Name	Purpose
User-Assigned Managed Identity	id-atlas-gw-restarter	Runbook's auth identity
Role Assignment	VM Contributor on vm-atlas-gw-01 scope	Allows runbook to start the VM, nothing else
Automation Account	aa-atlas-gw-restarter (Basic SKU)	Hosts the runbook + webhook
Runbook	Start-AtlasGw (PowerShell 7.2, Published)	Idempotent VM start logic
Webhook	wh-start-atlasgw-resourcehealth (1-year expiry → 2027-06-07)	Triggered by Action Group
Activity Log Alert	alert-vm-resource-health-unavailable	Detects Resource Health Unavailable
Action Group update	ag-atlas-gateway-prod — added ag-restart-vm-webhook receiver	Wires the chain

Cost

• Automation runbook execution: first 500 job-minutes/mo free per sub; expected usage ≈ 3 min/run × ~3 runs/day = ~270 min/mo ≪ free tier
• Activity Log alerts: free
• Action Group webhook calls: free up to 100k/mo
• Realistic monthly cost: $0

2b. Layer 2 — Gateway boot resilience (in-guest, this session)

File: /etc/systemd/system/openclaw-gateway.service.d/cold-boot-retry.conf

# Drop-in for openclaw-gateway: extends systemd restart window so the unit
# survives the Azure-host CDX shutdown -> VM-boot -> KV/MI warmup race.
#
# Original (Jun 5 by predecessor Atlas): Burst=10, Interval=300s, RestartSec=15
# Updated 2026-06-07 19:30 NZST by Atlas (CLI) with Sush one-time bypass:
#   Burst=20, Interval=1800s (30min), RestartSec=60
# Reference: systemd.unit(5) — StartLimit* settings MUST be in [Unit] section.
[Unit]
StartLimitBurst=20
StartLimitIntervalSec=1800

[Service]
Restart=on-failure
RestartSec=60

Original file backed up to: /etc/systemd/system/openclaw-gateway.service.d/cold-boot-retry.conf.bak-20260607T082505Z

Verified effective settings: StartLimitBurst=20, StartLimitIntervalUSec=30min, Restart=on-failure.

No gateway restart was required to apply (loaded via daemon-reload, takes effect on next start attempt).

2c. Layer 3 — Self-healing recovery timer (in-guest, set 2026-06-09 after 9h outage)

Why this layer exists: 2026-06-09 ~00:50 NZST a CDX restart cycle fired Layer 1 (Azure-side runbook brought VM back). VM rebooted clean. But openclaw-gateway.service failed at boot with status=243/CREDENTIALS and the cold-boot-retry burst (Layer 2's 20×60s window) exhausted. Service stayed dead. cloudflared kept proxying, returned 502 on every request. Layer 1 saw "VM running" = no further action. Gateway was effectively dead for ~9 hours (00:50 → 09:43 NZST) until Sush noticed + manual systemctl reset-failed && start at 09:43. Layer 3 is the separate poll-and-recover loop that closes the gap when both Layer 1 + Layer 2 have done their best but the service is still dead.

/usr/local/sbin/openclaw-gateway-recover.sh        (0755 root:root, 1269B)
/etc/systemd/system/openclaw-gateway-recover.service  (oneshot wrapping the script)
/etc/systemd/system/openclaw-gateway-recover.timer    (OnBootSec=30s + OnUnitActiveSec=15min)

Behavior: - Every 15 min (and 30s after boot) → invoke the recovery script - Script: systemctl is-active openclaw-gateway → if not active, reset-failed + start, log to syslog tag openclaw-recover - Idempotent: safe to run when gateway is healthy (just exits 0) - OnFailure hook (Path C) wired to the recovery service: if recovery itself fails OR flap-detector sees ≥3 fires in 1h → ntfy alert to topic atlas-gw-agfsqpqqstgz13cx (escalating to urgent priority + flap tag)

Smoke test (2026-06-09 09:55 NZST): stopped gateway → triggered recovery service directly → gateway back to active + HTTP 200 in 8.56 seconds total. Syslog trace clean (service state=inactive - attempting recovery → post-recover state=active).

Behavior model post-Layer-3: future failure mode that previously caused 9h outage now caps at ~15 min worst-case (next timer fire) OR ~30 sec for boot failures (OnBootSec).

File backups: ~/.copilot/session-state/84b26f08-de42-4ed5-8b07-0379da7444bc/files/openclaw-gateway-recover.{sh,service,timer} + openclaw-alert.sh + openclaw-gateway-alert@.service + onfailure-alert.conf (CLI-side, version-controlled).

Cross-reference: full deployment details + Path D/C/E architecture-gate analysis + today's incident timeline in openclaw-companion-playbook.md § 15.13.

---

2.5 The 2026-06-09 incident (what made Layer 3 necessary)

Time (NZST)	Event
~00:50	CDX VM auto-shutdown / restart cycle
00:57	Layer 1 Azure-side detected VM down, fired Automation runbook Start-AtlasGw 3× — VM came back
00:50-01:20	systemd cold-boot-retry burst (20×60s, Layer 2) attempted gateway start. All 20 attempts failed with (token.sh)[486520]: openclaw-gateway.service: Failed to set up credentials: Protocol error / status=243/CREDENTIALS
~01:20 onwards	Gateway DEAD. cloudflared healthy. tunnel returns HTTP 502 on every request. Layer 1 watchdog sees "VM running" + no further action
09:09	az-token-refresh-openclaw.timer fired SUCCESSFULLY (MI+KV pipeline IS healthy — credential infra isn't the persistent bug)
09:31	Sush noticed in WhatsApp DM that aunty wasn't replying
09:43	CLI Atlas diagnosed → manual systemctl reset-failed && start → gateway alive in ~6 sec
09:53	Layer 3 (recovery timer) shipped + smoke-tested live (8.56s recovery proven)
10:08	Path C (OnFailure alert via ntfy) shipped + delivery confirmed via web
~10:02	Aunty reviewed the C+E plans via Companion chat; flagged boot-immediate gap → Layer 3's OnBootSec tweaked 5min→30s

Root cause of the 243/CREDENTIALS itself still unknown — fetch-openclaw-token.sh exits 0 when run manually as root, both /run/openclaw-gateway/openclaw_token (37B) and _prev (0B placeholder) are created correctly. Aunty (runtime view from inside the gateway) confirmed no 243 cause visible in /var/log/openclaw/gateway.log; she did note repeated Windows-node token_mismatch on restarts (separate issue = backlog row #2 Companion shared-token refresh, NOT the credential boot bug). Path G (deep root-cause dive — enable verbose systemd-credentials logging, reproduce the boot race) deferred until Layer 3's observability tells us what's actually breaking under what conditions.

Lessons captured (operational discipline): 1. Watchdog "VM up" ≠ service up — Layer 1 caught the VM, missed the dead service. Need origin probe to close this gap (Path E deferred, backlog row #24) 2. systemd retry burst exhaustion = silent permanent dead — Layer 2 exhausted its window + nothing else woke it. Layer 3 is the separate recovery loop that closes this exhaustion gap 3. Rule #2 + #5 + Two-surface pattern paid off — aunty's runtime view (gateway.log from inside) ruled out causes CLI Atlas couldn't see; her flap-detector insight prevented a noisy-alert design 4. fetch-openclaw-token.sh exits 0 manually but failed at boot — credential setup script logic is fine, the failure is in the systemd-credentials handoff under boot conditions. Path G deferred until Layer 3's observability surfaces a reproducible pattern 5. Architecture-Gate prevented Path B (script "fix") destructive harm — initial hypothesis was _prev file missing → script "fix" would have changed working code based on a wrong premise. Reading the actual script first revealed the safeguard was already correct

---

3. How to verify it's working

3a. Quick health check (read-only, no changes)

# Run from Sush's machine, authenticated to CDX tenant
$RG = 'RG-ATLAS-GATEWAY-PROD'

# 1. Is the VM running?
az vm get-instance-view -g $RG -n vm-atlas-gw-01 --query "instanceView.statuses[?starts_with(code, 'PowerState/')].displayStatus | [0]" -o tsv

# 2. Is the gateway service active?
az vm run-command invoke -g $RG -n vm-atlas-gw-01 --command-id RunShellScript --scripts "systemctl is-active openclaw-gateway"

# 3. Was the runbook recently fired? (look at Automation Account jobs)
az rest --method GET --url "/subscriptions/96879ea6-389e-417f-a3a2-16c415a2b6b5/resourceGroups/RG-ATLAS-GATEWAY-PROD/providers/Microsoft.Automation/automationAccounts/aa-atlas-gw-restarter/jobs?api-version=2023-11-01&`$top=5" -o json

3b. Active trigger test (safe — does NOT shut down the VM)

The runbook is idempotent. Triggering it while the VM is running is a no-op. To smoke-test the alert→AG→webhook→runbook chain end-to-end:

$webhookUri = (Get-Content "$env:USERPROFILE\.copilot\session-state\23ef0e09-0ae0-45b3-a33e-a9a40c82c50e\files\octowatch-state.json" -Raw | ConvertFrom-Json).webhook_uri
$payload = @{
  schemaId = "Microsoft.Insights/activityLogs"
  data = @{
    status = "Activated"
    context = @{ activityLog = @{
      category = "ResourceHealth"
      resourceId = "/subscriptions/96879ea6-389e-417f-a3a2-16c415a2b6b5/resourceGroups/RG-ATLAS-GATEWAY-PROD/providers/Microsoft.Compute/virtualMachines/vm-atlas-gw-01"
      properties = @{ currentHealthStatus = "Unavailable"; cause = "PlatformInitiated" }
    }}
  }
} | ConvertTo-Json -Depth 10 -Compress
Invoke-WebRequest -Uri $webhookUri -Method POST -Body $payload -ContentType "application/json" -UseBasicParsing

Expected: HTTP 202 with JobIds body. Then check Automation Account job pane — job runs ~12s, output ends with "VM already running — no action. Runbook complete."

Last successful end-to-end test: 2026-06-07 08:51 UTC (job e36c623a-e59c-430a-9cf4-6a24d6cb15ef).

3c. Real-world validation

The first real CDX shutdown after deployment is the true test. When it happens, expected sequence (visible in Activity Log + Job logs):

1. ~T+0: Hyper-V host issues shutdown to guest
2. ~T+30s: VM enters Resource Health "Unavailable"
3. ~T+1–5min: Activity Log alert fires
4. ~T+1–5min: Action Group fires webhook + email (Sush gets email too — that's expected)
5. ~T+5–10s after webhook: Runbook job starts
6. ~T+10–15s: UAMI authenticates, checks state, calls Start-AzVM
7. ~T+60–120s after start call: VM enters running state
8. ~T+15–30s after boot: systemd starts openclaw-gateway (the boot-resilience.conf gives 30 min budget for KV warmup)
9. Total: 3–7 min from shutdown to aunty back online

If this DOESN'T happen, see "Diagnose silent aunty" below.

---

4. Diagnose silent aunty (what to do if both layers fail)

Don't reflexively restart anything. Run these read-only probes in order:

# A. Is the VM even running?
az vm get-instance-view -g RG-ATLAS-GATEWAY-PROD -n vm-atlas-gw-01 --query "instanceView.statuses" -o table

# B. Did the alert fire recently? (last 24h)
az monitor activity-log list --resource-id "/subscriptions/96879ea6-389e-417f-a3a2-16c415a2b6b5/resourceGroups/RG-ATLAS-GATEWAY-PROD/providers/Microsoft.Compute/virtualMachines/vm-atlas-gw-01" --max-events 30 --query "[?contains(category.value, 'ResourceHealth')].{time:eventTimestamp, status:status.value, healthStatus:properties.currentHealthStatus}" -o table

# C. Did the runbook fire? (look at jobs)
# Use the API call from § 3a item 3

# D. If runbook didn't fire but alert did — check Action Group webhook delivery
az monitor metrics list --resource "/subscriptions/.../providers/microsoft.insights/actionGroups/ag-atlas-gateway-prod" --metric "WebhookCallsFailed" --interval PT1H

# E. If gateway is down on a running VM — was systemd budget exhausted?
ssh-or-runcommand: journalctl -u openclaw-gateway --since '6 hours ago' | grep -iE "Failed|status=|StartLimit"

Common failure modes + fixes

Symptom	Likely cause	Fix
Alert fires but runbook doesn't	Webhook URI expired (>1 year)	Generate new webhook (REST PUT to AA /webhooks/<name>); update Action Group
Runbook fires but Start-AzVM fails with "Forbidden"	Role assignment revoked or UAMI deleted	Re-create UAMI + re-assign VM Contributor on VM scope
Runbook fires + succeeds + VM running but gateway dead beyond 30 min	KV is broken (not warmup race anymore)	Manual: az vm run-command invoke ... --scripts 'sudo journalctl -u openclaw-gateway --since 1h \| tail -50'. Likely need to rotate KV secret or fix MI assignment.
currentHealthStatus=Unknown instead of Unavailable (alert never fires)	Resource Health can't determine state (e.g., for some deallocate patterns)	Add belt-and-braces alert on Administrative deallocate/action category (see § 6 backlog)
First boot after shutdown takes >30 min to start gateway	KV/MI auth chain genuinely broken	Investigate /usr/local/sbin/fetch-openclaw-token.sh — it should retry internally; predecessor Atlas's design may not actually do this

---

5. The state file (DO NOT DELETE)

~/.copilot/session-state/23ef0e09-0ae0-45b3-a33e-a9a40c82c50e/files/octowatch-state.json holds: - UAMI resource ID + principal ID + client ID - Automation Account resource ID - Webhook URI (cannot be retrieved again — generated once) - VM + Action Group + sub IDs

If this session's state folder ever gets purged, the webhook URI is lost — would need to delete + recreate the webhook (and re-wire Action Group). Mitigation: snapshot this state file into ~/.copilot/secrets/atlas-gw-webhook-uri.json for permanent storage. (TODO for next session — see § 6.)

---

6. Backlog (improvements deferred from tonight's deploy)

Priority	Item	Why deferred
🟡 High	Snapshot webhook URI to ~/.copilot/secrets/ for permanent retention	Tonight focus was on shipping the chain, not durability of the URI
🟡 High	Add belt-and-braces alert on Administrative Microsoft.Compute/virtualMachines/deallocate/action	Covers the case where Resource Health reports Unknown instead of Unavailable (some deallocate scenarios) — same Action Group, same runbook, idempotent
🟡 High	Add a scheduled timer runbook firing every 30 min as ultimate fallback ("if VM not running, start it")	Belt-and-braces for missed alerts. Cost: still well within free tier.
🟢 Med	Open ICM/CDX ticket via aka.ms/cdx-support asking for RG-ATLAS-GATEWAY-PROD to be excluded from lab lifecycle shutdown policy	Real root-cause fix; could eliminate need for auto-restart entirely
🟢 Med	Investigate the actual LoadCredential= failure path in fetch-openclaw-token.sh — should retry KV connectivity internally with backoff before returning success	Rubber-duck (GPT-5.5) flagged this is the actual root cause we're treating symptomatically with retry budget extension
🟢 Med	Email receiver useCommonAlertSchema flipped from true to false during AG update (az CLI parsing quirk)	Sush will get older-format alert emails; cosmetic; no functional impact
🟢 Med	Application-level health check (not just systemctl is-active) — actually probe http://127.0.0.1:18789/health and restart if unhealthy	Current monitoring (atlas-probes cron) does this externally; could be integrated into systemd via Watchdog
🟢 Low	Move all this from manual az commands to a Bicep/Terraform module for reproducibility	Tonight was reactive shipping; tomorrow is the right time

---

7. The architecture-gate retrospective (Rule #5 / #11 in action)

This is the most important section for future Atlas. The session that built this fix violated Rule #5 (Architecture Gate) and Rule #11 (Root-Cause-Before-Fix) in a recoverable but expensive way. Sush caught the mistake; the rules didn't fire on their own.

What went wrong (chronologically)

1. ✅ Initial diagnosis was correct (probes → found systemd boot-credentials race + no auto-wake mechanism)
2. ✅ Phase 1 (systemd override) was a correct surgical fix
3. ❌ Phase 2 jumped to "let's build a custom Windows watcher" without surfacing alternatives
4. ❌ Sush approved "Path 2 full" but the Path I described was already wrong — I went order-taker mode
5. ❌ I half-built the watcher before Sush stopped me with "go research Azure-native patterns"
6. ✅ After Sush's intervention: launched 3 parallel reviews (rubber-duck + GPT-5.5 architecture review + research agent on Opus-high)
7. ✅ All 3 converged on the same answer: Resource Health → Action Group → Automation Runbook (the octowatch pattern)
8. ✅ Reverse course, ship the right thing in ~90 min

What I should have done from the start

Before any "build", apply the Rule #5 comparison table:

Solution	Could it host this?	Why kill / keep
Azure Automation Runbook + Webhook	✅ Native, $0/mo, official pattern	KEEP — this is the answer
Azure Logic App	✅ Works, ~$3/mo polling	Heavier than runbook
Azure Function (Timer/Event Grid)	✅ Works, <$1/mo	More code surface than runbook
Azure Monitor metric alert auto-remediation	✅ Sub-feature of Pattern #1	Same pattern
Custom Windows watcher on Sush's laptop	❌ Laptop sleeps overnight	KILL — fails exactly when needed
systemctl enable only	❌ Doesn't help if VM is deallocated	Necessary but not sufficient

The custom-watcher row should have been pre-killed by the laptop-sleep argument alone. Sush caught what Rule #5 should have caught autonomously.

The lesson for future Atlas

Pattern signal that this rule should fire: Whenever you find yourself about to write a Python/PowerShell/Node script that runs on the user's machine to compensate for missing cloud automation — STOP. That is almost always the wrong architecture. The cloud has the primitives; use them.

Specifically for the octowatch pattern: The exact "VM shut down by subscription-level policies, auto-restart it" use case is solved by github.com/jmassardo/octowatch (Terraform, ~130 lines). The implementation in this playbook is hand-rolled equivalent — for any next "auto-restart Azure VM" need, reuse THIS playbook before considering alternatives.

---

8. Quick reference — what's where

Thing	Where
This playbook	learning-docs/docs/reference/atlas-gw-reliability-playbook.md
Runbook source	~/.copilot/session-state/23ef0e09.../files/runbook-Start-AtlasGw.ps1
systemd override file (on VM)	/etc/systemd/system/openclaw-gateway.service.d/cold-boot-retry.conf
Webhook URI + UAMI IDs + AA name	~/.copilot/session-state/23ef0e09.../files/octowatch-state.json
Predecessor systemd override (backup)	/etc/systemd/system/openclaw-gateway.service.d/cold-boot-retry.conf.bak-20260607T082505Z (on VM)
Rejected Windows watcher (anti-pattern)	~/.copilot/atlas-gw-watcher/ — kept as a "why this is wrong" exhibit
Companion playbook	openclaw-companion-playbook.md (deep dive on gateway/Foundry/Baileys; this playbook is the reliability layer above it)

---

End of playbook. If you've read this far and you're future Atlas hitting a "the VM is down" incident: trust the system. The chain is built. Don't reflexively restart things. Run the diagnose probes in § 4 first.