Runbook: Rapidly Recovering User Identity Services After a Cloud Provider Incident

Hook: When identity fails, the whole cloud stops — recover it fast

Security and IT leaders tell us the same thing: when an IdP or SSO outage hits a cloud provider, investigation, remediation, and business continuity all grind to a halt. In 2026, with multi-cloud deployments, sovereign cloud rollouts, and new outage patterns (see late‑2025/early‑2026 incidents), you can't rely on manual improvisation. This runbook gives you a repeatable, tested sequence to restore IdP, SSO, and MFA quickly while preserving evidence, rotating credentials safely, and validating continuity.

Executive summary — what this runbook achieves

This runbook is a step‑by‑step playbook for technology teams to:

• Assess and triage identity outages within 15–30 minutes.
• Bring emergency access (break‑glass) online safely.
• Failover or restore IdP/SSO and re‑establish MFA continuity.
• Rotate compromised credentials and signing keys with no surprise downtime.
• Run fast continuity tests and capture chain‑of‑custody artifacts for legal/compliance needs.

Context: Why this matters in 2026

Late 2025 and early 2026 saw new patterns that directly affect identity availability and trust: global provider outages that cascaded into major SaaS failures, vendor fragmentation as organizations adopt sovereign cloud regions (for example, recent launches of dedicated EU sovereign clouds), and an uptick in identity‑focused attacks. These trends make IdP resiliency, emergency access planning, and credential hygiene top priorities for cloud incident response teams.

"Identity is the new perimeter: when it fails, true zero‑trust is impossible to enforce."

Runbook structure — phases at a glance

1. Preparation (pre‑incident): policies, break‑glass, secondary IdP, test plans.
2. Triage & containment (0–60 min): scope, severity, communication.
3. Emergency access & temporary control (15–120 min): enable break‑glass and emergency admin paths.
4. Recovery (1–8 hours): restore IdP/SSO functionality or failover to secondary control plane.
5. Credential rotation & MFA continuity (4–48 hours): rotate keys, re‑enroll MFA where needed.
6. Validation & testing (ongoing): synthetic logins, chaos testing, tabletop reviews.
7. Post‑incident: forensics, compliance reporting, runbook updates.

Phase 0 — Preparation: build the components you'll need

Preparation reduces mean time to recover. Implement these controls before an incident:

• Break‑glass accounts — physically separate admin accounts with credentials stored offline (hardware token + printed one‑time recovery codes locked in safe). Ensure these accounts have narrowly scoped privileges and are tracked in an auditable log. Consider the hardware and workstation policy for emergency consoles (see guidance on secure devices and auditable endpoints in operational fleets).
• Secondary IdP / federation plan — configure a standby IdP (cloud or on‑prem) and pre‑establish SAML/OIDC integration with critical SPs. Use automated IaC (Terraform, ARM, CloudFormation) to spin it up.
• Key & certificate lifecycle management — use a secrets manager (HashiCorp Vault, cloud KMS) and document rotation playbooks for signing keys and TLS certs that are compatible with rolling key use. Treat key versioning like model/version governance; see approaches to versioning and staged rollouts.
• MFA contingency mechanisms — ensure a mix of hardware tokens, platform authenticators (FIDO2), and backup OTP seeds stored in escrow for emergency re‑enrollment.
• Audit & log retention — centralize System and Auth logs (CloudTrail, Okta System Log, Azure AD sign‑ins) into a WORM store for at least 90 days to preserve chain‑of‑custody.
• Playbooks and runbook automation — keep scriptable runbooks in a secured repository with CI that validates syntax and dry‑runs key steps.

Prep checklist (mandatory)

• Register and test at least one break‑glass user per critical IdP.
• Maintain a warm standby IdP with pre‑configured SAML/OIDC metadata for key SPs.
• Store private signing keys in HSM or cloud KMS and maintain a rotation policy.
• Document step‑by‑step MFA re‑enrollment process and escalate path.

Phase 1 — Triage & containment: scope the incident in 15 minutes

When an outage or compromise is detected, rapid scoping prevents poor decisions. Use this checklist in the first 15 minutes:

• Identify impacted identity endpoints (IdP console, SSO login flow, MFA APIs).
• Classify the cause: provider outage vs. IdP compromise vs. configuration failure.
• Notify the incident response (IR) and executive callbacks per RACI.
• Open an evidence preservation task: snapshot logs, collect provider status pages, take screen captures of dashboards.

Quick diagnostics

• Check provider status pages (and Slack/Twitter for widespread outages).
• Run curl/test clients against OIDC and SAML endpoints to measure error types (5xx vs 4xx vs DNS failures).
• Query authentication logs for spike patterns or unusual token issuance.

Phase 2 — Emergency access: get admins in without widening blast radius

If normal admin flows fail, bring emergency controls online. Use the least privilege required and document every action.

Emergency access options

1. Break‑glass local console access: If IdP provider management is down, use a locally hosted management console (pre‑provisioned and secured) to manage critical settings.
2. Out‑of‑band MFA tokens: Use hardware tokens stored in custody or pre‑distributed FIDO2 keys to log in through alternative identity paths.
3. Privileged jump host: A hardened bastion with cached credentials and pre‑staged CLI tokens can provide temporary console access.
4. Emergency federation to alternate IdP: If primary IdP is unavailable but SPs can accept a second IdP, enable a pre‑configured standby IdP and validate claims mapping.

Operational steps (example)

1. Authorize break‑glass use via the incident commander and two‑person control.
2. Retrieve hardware token from custody and authenticate using the break‑glass identity.
3. Enable a read‑only audit mode for investigation, then escalate to limited write access if needed for fixes.
4. Document all commands, screenshots, and times — preserve in immutable evidence store.

Phase 3 — Restore IdP/SSO: stepwise recovery and failover

Decide: restore the primary IdP or failover to the standby. Your decision should be based on incident type and provider ETA.

Option A — Restore primary IdP

1. In read‑only mode, capture configuration snapshots: SAML metadata, OIDC keys, SCIM connectors.
2. Reconcile any partial changes and roll back recent config changes that may have caused the outage.
3. Restart identity services per vendor guidance; monitor auth success metrics closely.
4. Once stable, re‑enable normal admin access and monitor for anomalies for 24–72 hours.

Option B — Controlled failover to standby IdP

1. Promote standby IdP into active role using pre‑loaded SAML/OIDC metadata.
2. Switch SPs in small batches: route critical business units first, validate sign‑ins.
3. Use a staged approach with feature flags or DNS TTL adjustments to reduce user impact.
4. Keep original IdP logs preserved for post‑mortem and forensic analysis.

Sample failover checklist

• Ensure SCIM provisioning endpoints are reachable and mapped.
• Verify signing certificates are present in SP trust stores (or use dual‑key strategy described below).
• Coordinate with SaaS vendors to accept the alternate IdP metadata where necessary.

Phase 4 — Credential rotation without breaking services

Rotation must be deliberate. Key principle: introduce new credentials first, validate them, then retire old credentials. This minimizes downtime and supports rollback.

Key rotation sequence (signing keys, TLS certs, API secrets)

1. Generate new key/certificate in KMS/HSM and store metadata in your secrets manager.
2. Publish new public key/certificate to SPs alongside the existing key (dual key trust) if the protocol supports it (SAML allows multiple keys in metadata).
3. Update the IdP to sign with the new key for a subset of tokens to validate acceptance.
4. Monitor SP logs for verification successes; after a validation window, remove the old key from trust stores.

Service credential rotation (automation)

• Use IaC to rotate application credentials: issue new creds, update app config via CI/CD, validate, then revoke old creds.
• Script rotations with a canary group and health checks; maintain fallback scripts for emergency rollback.
• For short‑lived tokens, adopt OAuth2 client credentials with automated renewals and short lifetimes to reduce blast radius.

MFA re‑enrollment strategy

1. For users who lost MFA state, issue temporary emergency access codes with strict expiration and conditional access policies limiting resource scope.
2. Require re‑enrollment via supervised workflows: identity verification by helpdesk + temporary code + mandatory change of password plus hardware token issuance where possible.
3. Log every re‑enrollment action and flag for post‑incident review.

Phase 5 — Continuity testing: validate identity resilience

Recovery isn't complete until you validate. Run these tests immediately and schedule them periodically.

Immediate validation steps

• Perform synthetic sign‑ins for several user personas across apps (SSO, API clients, on‑prem apps federated to cloud IdP).
• Validate SCIM provisioning for a test user (create → update → delete) to confirm downstream user lifecycle works.
• Run MFA scenario tests: hardware token login, app‑based OTP, and FIDO2 authenticator checks.

Periodic resilience exercises

• Quarterly tabletop exercises with simulated outage of primary IdP and live failover to standby (see postmortem and incident comms templates for formatting).
• Monthly automated chaos tests in a staging environment that mimic provider outages and certificate expiries.
• Annual red/blue team tests focusing on identity takeover scenarios and MFA bypass attempts.

Phase 6 — Evidence preservation & compliance

Preserve artifacts for forensic, legal, and compliance activities. Time matters — start preservation during triage.

Artifacts to collect

• Full IdP system logs, SSO assertion logs, SCIM provisioning traces, MFA device enrollment logs.
• Snapshots of IdP configuration (SAML metadata, OIDC client registrations, attribute mappings).
• Cloud provider incident timelines, status pages, and communications from vendor support.
• Command logs and console screenshots from emergency access sessions.

Chain of custody checklist

1. Record collection time (UTC), collector identity, and collection method for each artifact.
2. Store copies in an immutable evidence repository (WORM or object lock) and restrict access.
3. Use signed manifests and hashing (SHA‑256) to ensure integrity for legal admissibility.

Case study: rapid IdP recovery during a 2025 provider outage (anonymized)

In late‑2025, a mid‑sized SaaS company experienced regional provider disruption that broke its primary IdP control plane. Because the team had implemented this runbook's principles, they executed the following in under 3 hours:

1. Authorized break‑glass and restored console access using a hardware token from escrow.
2. Promoted a pre‑configured standby IdP in a separate cloud region and updated SAML metadata via automation for core SPs.
3. Performed key rotation using a dual key trust model and re‑enrolled 1% of high‑risk users for MFA as a pilot before rolling out org‑wide.
4. Preserved logs and vendor communications for the post‑incident report; no customer data exposure was found.

The result: critical business workflows resumed with minimal user friction and a clear timeline for returning to the primary IdP.

Tools, templates, and snippets (operationally useful)

Use these patterns to automate recovery tasks.

IaC snippet (Terraform conceptual)

Pre‑define standby IdP resources and SP metadata as code so you can promote with a single pipeline run. Example conceptual steps:

1. terraform apply -target=module.standby_idp
2. terraform output sp_metadata > standby_metadata.xml
3. Upload metadata to SPs via API or vendor console automation

Key rotation checklist (scripted)

1. kms create‑key --usage SIGNING --alias idp‑signing‑v2
2. idp upload‑public‑key --file v2_pub.pem
3. idp set‑signing‑policy --use new_key_for_subset=true
4. monitor sp_verify_logs for 24h
5. if ok: idp remove‑old‑key --alias idp‑signing‑v1

Common pitfalls and how to avoid them

• Pitfall: No standby IdP metadata. Fix: Pre‑publish and test metadata in dev.
• Pitfall: Over‑privileged break‑glass. Fix: Limit to minimum scope and require two‑person authorization.
• Pitfall: Credential rotation that breaks integrations. Fix: Use dual keys and staged rollouts.
• Pitfall: Not preserving logs. Fix: Automate log export to immutable storage on incident detection.

2026 trends to incorporate into your identity recovery program

• Sovereign clouds and regional independence: More organizations host identity in regionally separate control planes — design failover across legal boundaries and understand data residency impacts. See architectural patterns for hybrid sovereign cloud architecture.
• FIDO2 + passkeys adoption: Passkeys reduce OTP dependence — incorporate hardware and platform authenticators in your MFA contingency planning.
• Short‑lived credentials and automated rotation: Embrace ephemeral credentials for service accounts and OAuth clients to minimize exposure.
• Identity observability: Invest in SIEM and identity threat detection platforms that can detect anomalous token issuance in real time. Also plan for edge and cache behaviours that can affect detection — techniques described in edge cost & optimization guides are helpful when designing observability for distributed control planes.

Actionable takeaways — what to do this week

1. Inventory your IdP dependencies and classify critical SPs (A/B/C). Prioritize A for immediate failover readiness.
2. Create or validate two break‑glass accounts per IdP and test recovery from physical custody.
3. Document and automate a key rotation playbook using dual key uploads and monitor windows.
4. Schedule a tabletop outage exercise that simulates a provider outage and failover to a standby IdP.

Final checklist — ready to run in an incident

• Triage and classify incident within 15 minutes.
• Authorize break‑glass and collect evidence.
• Choose restore primary or failover to standby IdP and execute in staged batches.
• Rotate keys and credentials using dual trust; re‑enroll MFA for impacted users with emergency codes as needed.
• Validate with synthetic logins and SCIM tests, then document and report.

Closing — keep identity resilient in 2026

Identity outages are no longer theoretical. With provider outages and new sovereign cloud architectures on the rise, you need a tested runbook that covers emergency access, safe credential rotation, and continuous validation. The steps above convert those requirements into operational actions your team can run under pressure.

Ready to operationalize this runbook? Download our incident‑ready IdP/SSO/MFA templates and automation examples, run a tabletop this quarter, and contact our cloud incident response team for a tailored resilience assessment.

Related Reading

• Postmortem templates and incident comms for large-scale service outages
• Hybrid sovereign cloud architecture for municipal data
• Data sovereignty checklist for multinational CRMs
• Hybrid Edge Orchestration Playbook for Distributed Teams — Advanced Strategies (2026)
• Builder Confidence Slumps — Tax Moves Homebuilders Should Make Now
• Don’t Clean Up After Your AI: Setting Governance Rules for Contract Drafting and Operating Agreements
• Airport Battery Rules: What Capacity Power Banks You Can Fly With
• Automating Vulnerability Triage: From Bug Reports to Fixes
• How Gmail’s New AI Changes Your Email Open Strategy (and What to Do About It)