System Admin Root Emergency Recovery

This runbook restores the one explicitly marked Root account in the independent System Admin control-plane database. Use it only when the normal Root session is unavailable, including an SSO admission outage or loss of both the OTP device and recovery codes. It does not apply to customer endpoint databases.

Preconditions

1. Have an approved Infra incident and an operator identity independent of System Admin.
2. Confirm the exact control-plane host, database, schema, and environment from the deployment record. Do not infer the target from a developer shell's default database settings.
3. Take and verify a database backup. Use a read-only connection for the checks below, then open a separate transaction for the recovery.
4. Verify migration SystemAdminAccessManagement1894000000007 is applied. Do not run a partial or hand-written schema migration during an account recovery.
5. Obtain the Root UUID from the protected account inventory. The UUID is the recovery selector; do not promote an account by matching an email address.

If the selected Root UUID is unavailable, inspect the unique Root record and compare it with the protected inventory before writing anything:

SELECT "id", "emailNormalized", "status", "loginPolicy", "authorizationVersion"
FROM "admin_users"
WHERE "isRoot" = true AND "deletedAt" IS NULL;

Proceed only if exactly one result is independently confirmed as the intended Root. If there are zero or multiple results, stop and escalate; do not seed another administrator or guess by email.

Review legacy local accounts before upgrading

For a database created before the isRoot marker, review this read-only inventory before deployment. Confirm the initial Root by UUID with the account owner; do not infer Root from email or automatically select the first administrator.

SELECT u."id", u."emailNormalized", u."status", u."loginPolicy",
       credential."mustChangePassword", credential."lockedUntil"
FROM "admin_users" u
JOIN "local_credentials" credential
  ON credential."adminUserId" = u."id" AND credential."deletedAt" IS NULL
WHERE u."deletedAt" IS NULL
ORDER BY u."id";

Migration 1894000000007 changes only the schema. It does not select a Root, inspect local accounts, update isRoot, or write an audit row. After a legacy database is structurally migrated, use tools/system-admin/server/db/seeds/assign-existing-root.sql with the independently confirmed UUID. This manual SQL validates that the account is active, allows local login, has an existing local credential, and holds a permanent global assignment to the active system SUPER_ADMIN role. It preserves password hashes, OTP factors, recovery codes, login policy, authorization version, sessions, and role assignments.

If active local accounts exist but the UUID is missing, malformed, or ineligible, the manual Root script stops before changing data. If another active local account remains, review it separately rather than silently disabling it. An empty database uses bootstrap-local.sql after the catalog script. Never rerun an account script to repair a different account.

Environments that already executed the old version of migration 1894000000007 must not rerun it. Confirm the Root UUID independently and use the reviewed manual SQL script to set isRoot=true, preserving the existing credential and factors. Verify there is no other Root, record the change, and check the permanent global assignment before committing. Do not mark migration history as unapplied or roll back unrelated schema changes to force a rerun.

Inspect the control plane

Bind :root_id to the confirmed UUID in the database client. Check the account, credential, factors, sessions, challenges, and current global Super Administrator assignment:

SELECT "id", "isRoot", "status", "loginPolicy", "authorizationVersion", "deletedAt"
FROM "admin_users"
WHERE "id" = :root_id AND "isRoot" = true;

SELECT "id", "mustChangePassword", "passwordChangedAt", "failedAttemptCount", "lockedUntil"
FROM "local_credentials"
WHERE "adminUserId" = :root_id AND "deletedAt" IS NULL;

SELECT "id", "factorType", "status", "confirmedAt", "deletedAt"
FROM "mfa_factors"
WHERE "adminUserId" = :root_id AND "deletedAt" IS NULL;

SELECT "id", "scopeType", "endpointId", "organizationId", "expiresAt", "revision", "deletedAt"
FROM "admin_role_assignments"
WHERE "adminUserId" = :root_id;

SELECT role."id", role."code", role."status", permission."code"
FROM "admin_roles" role
LEFT JOIN "admin_role_permissions" link
  ON link."roleId" = role."id" AND link."deletedAt" IS NULL
LEFT JOIN "admin_permissions" permission
  ON permission."id" = link."permissionId" AND permission."deletedAt" IS NULL
WHERE role."code" = 'SUPER_ADMIN' AND role."deletedAt" IS NULL;

Confirm there is one active SUPER_ADMIN system role and that its active permission links include sysadmin_access:manage.global. If the role or permission mapping is missing, use the controlled catalog reconciliation command from the System Admin bootstrap runbook; do not improvise system role rows by hand.

Restore account access

Use one transaction for all database changes and its application audit record. Lock the SUPER_ADMIN role before the Root user, matching the application lock order. Keep the old password and OTP when they are still available.

BEGIN;

SELECT "id"
FROM "admin_roles"
WHERE "code" = 'SUPER_ADMIN' AND "status" = 'active' AND "deletedAt" IS NULL
FOR UPDATE;

SELECT "id"
FROM "admin_users"
WHERE "id" = :root_id AND "isRoot" = true AND "deletedAt" IS NULL
FOR UPDATE;

UPDATE "admin_users"
SET "status" = 'ACTIVE',
    "loginPolicy" = 'local_only',
    "authorizationVersion" = "authorizationVersion" + 1,
    "updatedAt" = now()
WHERE "id" = :root_id AND "isRoot" = true AND "deletedAt" IS NULL;

INSERT INTO "admin_role_assignments"
  ("adminUserId", "roleId", "scopeType", "endpointId", "organizationId", "expiresAt", "assignedBy", "revision")
SELECT :root_id, role."id", 'global', NULL, NULL, NULL, NULL, 0
FROM "admin_roles" role
WHERE role."code" = 'SUPER_ADMIN'
  AND role."status" = 'active'
  AND role."deletedAt" IS NULL
  AND NOT EXISTS (
    SELECT 1
    FROM "admin_role_assignments" assignment
    WHERE assignment."adminUserId" = :root_id
      AND assignment."roleId" = role."id"
      AND assignment."scopeType" = 'global'
      AND assignment."endpointId" IS NULL
      AND assignment."organizationId" IS NULL
      AND assignment."deletedAt" IS NULL
  );

UPDATE "admin_sessions"
SET "revokedAt" = now(), "updatedAt" = now()
WHERE "adminUserId" = :root_id AND "revokedAt" IS NULL;

UPDATE "auth_challenges"
SET "consumedAt" = now()
WHERE "adminUserId" = :root_id AND "consumedAt" IS NULL;

INSERT INTO "admin_audit_logs"
  ("actorKind", "adminUserId", "authMethod", "action", "targetType", "targetId",
   "endpointId", "organizationId", "outcome", "requestId", "metadata", "ipAddress", "userAgent", "createdAt")
VALUES
  ('system', NULL, NULL, 'auth.root.db_recovered', 'admin_user', :root_id,
   NULL, NULL, 'success', gen_random_uuid(),
   jsonb_build_object('mode', :recovery_mode, 'reason', :incident_reason),
   :operator_ip, NULL, now());

-- Verify the updated account and assignment inside the transaction before commit.
SELECT user."id", user."status", user."loginPolicy", user."authorizationVersion",
       assignment."scopeType", assignment."expiresAt"
FROM "admin_users" user
JOIN "admin_role_assignments" assignment ON assignment."adminUserId" = user."id"
JOIN "admin_roles" role ON role."id" = assignment."roleId"
WHERE user."id" = :root_id
  AND user."isRoot" = true
  AND user."deletedAt" IS NULL
  AND user."status" = 'ACTIVE'
  AND user."loginPolicy" = 'local_only'
  AND assignment."scopeType" = 'global'
  AND assignment."endpointId" IS NULL
  AND assignment."organizationId" IS NULL
  AND assignment."expiresAt" IS NULL
  AND assignment."deletedAt" IS NULL
  AND role."code" = 'SUPER_ADMIN'
  AND role."status" = 'active'
  AND role."deletedAt" IS NULL;

COMMIT;

The SQL uses named bind placeholders; adapt only the placeholder syntax to the approved database client. Check that the Root update affected exactly one row and the verification query returns exactly one permanent global assignment. If a precondition or row count differs, roll back and investigate. Do not hard-delete the user, role assignments, factors, sessions, or audit rows.

If the Root password and all OTP recovery methods are lost

Do not disable MFA for an ordinary login. Generate a one-time random temporary password on the secured operator workstation and hash it with the application helper hashSystemAdminPassword from tools/system-admin/server/security/password.ts (bcrypt cost 12). Keep the cleartext only in the approved secret-delivery channel. The account's forced-password-change step will require a new password before TOTP enrollment.

Within the recovery transaction above, after locking the Root account and verifying the existing credential row, also run the following statements. Bind :temporary_password_hash to the application-generated bcrypt hash. Use recovery mode credential_and_factor_reset for the audit record.

UPDATE "local_credentials"
SET "passwordHash" = :temporary_password_hash,
    "mustChangePassword" = true,
    "passwordChangedAt" = NULL,
    "failedAttemptCount" = 0,
    "lockedUntil" = NULL,
    "updatedAt" = now()
WHERE "adminUserId" = :root_id AND "deletedAt" IS NULL;

UPDATE "mfa_factors"
SET "status" = 'revoked',
    "deletedAt" = now(),
    "deletedBy" = :root_id,
    "updatedAt" = now()
WHERE "adminUserId" = :root_id AND "deletedAt" IS NULL;

UPDATE "mfa_recovery_codes"
SET "consumedAt" = now(),
    "updatedAt" = now()
WHERE "adminUserId" = :root_id AND "consumedAt" IS NULL;

Require the credential update to affect exactly one row. If the credential row is absent, stop and follow a reviewed database repair procedure rather than adding an unreviewed credential. After commit, the Root signs in with the temporary password, completes the required password change, and enrolls a new OTP through the normal protected login flow. The new OTP must be confirmed before a normal session is issued. Save the newly generated recovery codes in the approved secret store; do not put passwords, hashes, TOTP secrets, provisioning URIs, or recovery codes into the incident audit metadata.

Verify and close the incident

1. Have the Root log in through the local Root form. Confirm the temporary password cannot be reused after the forced change when this path was used.
2. Confirm TOTP works and the recovery-code count is nonzero. Confirm a recovery code cannot be consumed twice.
3. Confirm the Root has the permanent global SUPER_ADMIN assignment and can open System Admin access management.
4. Confirm every pre-recovery session and authentication challenge is revoked or consumed.
5. Retain the auth.root.db_recovered database audit row, transaction result, protected backup reference, operator identity, incident reason, and independent Infra incident record.
6. Do not broaden Google admission rules as a side effect of a Root recovery. If SSO policy caused the incident, repair the exact Workspace/email-domain pair separately and verify Workspace-enforced MFA with its administrator.