Secret rotation runbook ​

Use this procedure whenever a secret is exposed — if it appears in a log, a transcript, a shared screen, or a repository — or on a periodic cadence, or when someone with access leaves. The same steps apply regardless of how the exposure happened.

When to rotate ​

Rotate any secret that has been:

  • Printed to a terminal or log that was shared or recorded
  • Committed to version control (even briefly)
  • Sent over a channel that isn't encrypted end-to-end
  • Viewed by someone who should not have access

When in doubt, rotate. The cost is usually a Railway redeploy (~30 seconds); the cost of not rotating can be far higher.

Cadence: rotate long-lived secrets roughly quarterly, and immediately on suspected compromise or staff offboarding. Guidance, not a hard policy.

Recommended order when rotating everything at once: do the disruptive ones in a maintenance window — DATABASE_URL / MIGRATION_DATABASE_URL (brief outage) and CONSOLE_SESSION_SECRET (logs every console user out). The rest (Anthropic, OAuth, Slack, error-reporter) are isolated and can be done any time. After each, run the Sanity sweep at the bottom.

Secrets never live in code — they're Railway environment variables, the tenants file, or the relevant provider console. "Update in VERA" almost always means Railway service → Variables → edit → Save (which auto-redeploys), except per-tenant Slack creds (tenants file / /admin).

1. ADMIN_SECRET ​

Gates /admin and the /status* dashboards.

  1. Generate: node -e "console.log(require('crypto').randomBytes(24).toString('base64'))"
  2. Update on Railway → ADMIN_SECRET → Save.
  3. Blast radius: /admin and /status reject the old secret immediately; no user-facing impact.
  4. Verify: /admin and /status accept the new secret.

2. DATABASE_URL (Supabase vera_app role) ​

Runtime connection used by lib/pg.js. The role is vera_app — NOBYPASSRLS, DML grants only. Do not reset the master postgres password here.

  1. Supabase → Settings → Database → reset the vera_app role password.
  2. Copy the new connection string (pooler mode).
  3. Update on Railway → DATABASE_URL → Save.
  4. Blast radius: full outage until the redeploy picks up the new value (the app can't reach Postgres).
  5. Verify: a console read (e.g. the project list) returns data. Confirm the old string fails: psql "old-string" -c "SELECT 1".

3. MIGRATION_DATABASE_URL (Supabase postgres role) ​

Privileged connection used by scripts/start.js to run migrations at boot. Role postgres — bypasses RLS, manages pgmigrations.

  1. Supabase → Settings → Database → reset the postgres role password.
  2. Copy the new connection string.
  3. Update on Railway → MIGRATION_DATABASE_URL → Save.
  4. Blast radius: runtime unaffected; only the next deploy's migrate step fails if this is stale.
  5. Verify: next deploy's boot log shows the migrate step succeeding. Update any local .env used for npm run migrate:up.

4. ANTHROPIC_API_KEY ​

Used for box-2 (analytical) model calls. Box-1 deterministic paths and the entire console do not call the model.

  1. Rotate in the Anthropic console (create a new key, then revoke the old).
  2. Update on Railway → ANTHROPIC_API_KEY → Save.
  3. Blast radius: analytical Slack replies fail until updated; deterministic logging/reads and all console pages are unaffected.
  4. Verify: /status Anthropic check goes green (it records the last successful messages.create).

5. CONSOLE_SESSION_SECRET ​

HMAC key for console session cookies, magic-link tokens, and OAuth state cookies (lib/console/auth.js). High blast radius — rotate deliberately.

  1. Generate: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
  2. Update on Railway → CONSOLE_SESSION_SECRET → Save.
  3. Blast radius: every active console session is invalidated and all outstanding magic links stop working — everyone must sign in again. Any OAuth flow in flight at the moment of rotation fails and must be restarted. If this var is ever unset, /console* returns 503 (fail-closed).
  4. Verify: sign in to /console fresh (OAuth or magic link) and confirm a session is established.

6. GOOGLE_OAUTH_CLIENT_ID / GOOGLE_OAUTH_CLIENT_SECRET ​

Google sign-in for the console. Only the secret rotates day-to-day; the client ID changes only if you recreate the OAuth client.

  1. Google Cloud console → APIs & Services → Credentials → the OAuth client → rotate the client secret.
  2. Update on Railway → GOOGLE_OAUTH_CLIENT_SECRET (and ..._CLIENT_ID if it changed) → Save.
  3. Blast radius: "Sign in with Google" fails until updated; existing console sessions survive. If both vars are unset, the Google button simply doesn't render.
  4. Verify: the login page shows the Google button and a Google sign-in completes.

7. MS_OAUTH_CLIENT_ID / MS_OAUTH_CLIENT_SECRET ​

Microsoft (Azure AD / Entra) sign-in. Same shape as Google.

  1. Azure portal → App registrations → the app → Certificates & secrets → new client secret; remove the old.
  2. Update on Railway → MS_OAUTH_CLIENT_SECRET (and ..._CLIENT_ID if it changed) → Save.
  3. Blast radius: "Sign in with Microsoft" fails until updated; existing sessions survive.
  4. Reminder: the callback URI registered in Azure must be ${CONSOLE_BASE_URL}/console/auth/microsoft/callback.
  5. Verify: a Microsoft sign-in completes.

8. ERROR_REPORTER_GITHUB_TOKEN ​

GitHub PAT with repo scope on talktalkmake/VERA, used to auto-file issues. Must be unset / no-op in dev and test.

  1. GitHub → Settings → Developer settings → Personal access tokens → revoke the old, create a new one with repo scope.
  2. Update on Railway → ERROR_REPORTER_GITHUB_TOKEN → Save.
  3. Blast radius: automatic issue filing stops until updated; the app is otherwise unaffected (the reporter is fire-and-forget and never blocks responses).
  4. Verify: a subsequent real error files/updates an issue (or test in staging).

9. Per-tenant Slack credentials ​

Each tenant has its own slack_bot_token (xoxb-…) and slack_app_token (xapp-…), stored in the tenants file (TENANTS_FILE / tenants.dev.json) and editable in /admin. These are not global env vars.

  1. Rotate in the Slack app configuration (the distributed app A0AE27UD6H3, or the tenant's own app) → regenerate the bot token / app-level token.
  2. Update in VERA via /admin (the tenant editor) or by editing the tenants file and reloading.
  3. Blast radius: that one tenant's Slack bot goes offline until updated; other tenants are unaffected (credentials are per-tenant).
  4. Verify: /status Slack-socket check for that tenant is green, and the bot replies to a DM.

Sanity sweep ​

Run after rotating anything:

  • [ ] /status shows green for Postgres and Anthropic
  • [ ] A console page loads and a fresh sign-in works (if CONSOLE_SESSION_SECRET or an OAuth secret was rotated)
  • [ ] /admin accepts the current ADMIN_SECRET
  • [ ] The bot replies to a Slack DM on at least one tenant (if any Slack/DB secret was rotated)
  • [ ] The next deploy's boot log shows migrations applying cleanly (if MIGRATION_DATABASE_URL was rotated)