System Admin Authorized Backend Reads

This runbook configures the signed connection between the independent System Admin control plane and a business API. The business API keeps its own authentication and tenant data; System Admin only calls the explicitly authorized service procedures.

Environment contract

Infra configures the stable backend identity and internal route on both deployments. The value is an opaque runtime identity; it is not a database UUID and it is not generated by a migration.

# System Admin
- name: SYSADMIN_ENDPOINTS
  value: |
    [
      {
        "id": "example-backend",
        "name": "Example Backend",
        "baseUrl": "http://api-main.example.svc.cluster.local",
        "env": "production",
        "csReportPrefix": "EXMPL"
      }
    ]

# Corresponding business backend
- name: SYSADMIN_ENDPOINT_ID
  value: example-backend

id must be lowercase letters, digits, or hyphens (1–50 characters) and must be unique in the Sysadmin registry. Every replica of one backend uses the same value. Multiple hospital codes and organizations may share one backend. baseUrl is the origin reachable from the Sysadmin Pod; include a port only when the Service requires it, and do not append /api or /trpc.

SYSADMIN_ENDPOINTS is required. A JSON [] is valid for an intentionally empty registry. Malformed JSON, duplicate IDs, duplicate CS prefixes, or unsafe origins stop Sysadmin startup. A missing SYSADMIN_ENDPOINT_ID makes internal service verification fail closed while ordinary business routes remain available.

Signing configuration

Generate one signing key pair in the Sysadmin workspace. Keep the private key only in Sysadmin and copy the public key to the business backend. Configure the same issuer on both services:

# System Admin
SYSADMIN_SERVICE_SIGNING_PRIVATE_KEY="...PKCS#8 private PEM..."
SYSADMIN_SERVICE_TOKEN_ISSUER=urn:hereby:works:sysadmin:production

# Business backend
SYSADMIN_SERVICE_TOKEN_ISSUER=urn:hereby:works:sysadmin:production
SYSADMIN_SERVICE_TRUSTED_PUBLIC_KEY="...generated public PEM..."
SYSADMIN_ENDPOINT_ID=example-backend

Do not add endpointInstanceId or a separately maintained UUID for this identity. SYSADMIN_ENCRYPTION_KEY remains the separate Sysadmin secret for local credentials and OIDC; it is not the backend identity. Existing endpoints.id UUIDs remain Sysadmin foreign-key targets for permissions, hospital codes, publications, and audit history.

Database association and first deployment

After migrations, Sysadmin creates a missing association row for each configured id inside a serializable transaction. The row contains no account, grant, hospital code, or publication seed. Existing active rows are matched by endpoints.code and keep their UUID. An ambiguous or soft-deleted match stops startup and requires a reviewed operator SQL adjustment.

For an existing installation, review the mapping before rollout:

SELECT id, code, "legacyId", "instanceId", "baseUrl", "registrationStatus",
       "appApiBaseUrl", "appApiPublicationStatus", "appApiPublicationRevision"
FROM hereby_works.endpoints
WHERE "deletedAt" IS NULL
ORDER BY code, id;

If the reviewed ENV ID does not exactly match the existing active code, edit and run tools/system-admin/server/db/seeds/reconcile-endpoint-env-identities.sql in the DB tool. The script aborts for missing/soft-deleted UUIDs, duplicate active codes, or an incomplete update. It changes only the active code association and preserves the endpoint UUID and all business data. Do not insert an account, identity, approval timestamp, or endpoint UUID in a migration. Hospital codes and App publication remain separate operations in the App backend page.

Runtime behavior

• Sysadmin reads URL, name, environment, membership, and identity only from its immutable ENV registry.
• The database supplies the existing UUID relationship and business metadata such as hospital codes, permissions, public App URL, publication revisions, health records, and audit history.
• A configured endpoint is selectable under the existing permission grants even when its health check is unavailable. Health never grants authorization.
• Removing an ID disables routing, health probes, mutations for that backend, and public discovery. Its DB row and business data remain. Re-adding the same ID reuses the UUID and saved publication state.
• An internal URL change triggers fresh health results only. It does not clear the public App URL or force republication. A changed ID is a new identity and requires a reviewed cutover.
• Capability requests still verify RS256 signature, issuer, audience, expiration, replay, UUID endpointId, configured ID, action, and scope.

The browser calls Sysadmin. Sysadmin performs the signed request from its own Pod or process network:

GET <configured baseUrl>/trpc/systemAdminRead.capabilities

Endpoint page

The endpoint page is a status view. It shows the Infra display name, environment, health status, last check, sanitized failure reason/request ID, and CS prefix synchronization status. The page does not show internal URLs or identities and does not allow endpoint create, edit, approval, rejection, disable, identity rotation, or manual CS prefix mutation. Old mutation URLs return 405 ENDPOINT_CONFIG_MANAGED_BY_INFRA.

csReportPrefix is optional in SYSADMIN_ENDPOINTS. When present, Sysadmin synchronizes it after a successful capabilities check using the existing monotonic version and idempotency rules. Omission does not clear an existing value. Duplicate or conflicting prefixes fail synchronization without changing the last successfully applied value. The only way to request a new value is to update ENV and roll the deployment.

App backend and discovery

The App backend page continues to manage only the public HTTPS App URL, evidence/confirmation, publication revision, and hospital codes. It does not manage the internal route or backend identity. Multiple hospital codes may resolve to one configured backend.

Public discovery requires an enabled code, current ENV membership, a saved published public URL, matching verified URL/ENV identity/revision, and the existing CORS/rate-limit checks. The response backendInstanceId now contains the configured ENV ID (for example example-backend), not the old DB UUID.

Diagnostics and rollout checks

Use the request ID shown by the endpoint page to correlate the health result with Sysadmin structured logs and audit rows. A 401 or 403 from the capabilities route proves DNS, routing, and the HTTP listener are reachable; it does not prove the signing configuration is correct.

Before production rollout verify:

1. Both deployments have matching id/SYSADMIN_ENDPOINT_ID, issuer, and signing/trusted public-key settings.
2. Migrations finish before Sysadmin starts; no seed or identity bootstrap command is run.
3. The endpoint page lists the configured backend without exposing its internal URL.
4. A manual health check succeeds and the CS prefix status converges when a prefix is configured.
5. The App public URL can be saved and published, multiple hospital codes resolve to the same backend, and organization-scoped reads remain isolated.
6. Existing local admin credentials, Root/MFA state, business login, permissions, and audit history are unchanged.

During a normal EKS rolling deployment, old and new Pods may briefly use different ENV registries. Do not add a registry generation table or DB fallback. Validate final routing and CS prefix convergence after both rollouts complete.