Sign Webhook ESignRecord Binding Race — Implementation Plan

Status: implementation complete; migration and live manual QA pending Date: 2026-08-06 Projects: hereby-dhub + hereby-sign Execution rule: obtain Eron's explicit approval before changing code or local data. Commit and push require separate approval.

1. Problem

The annual-leave promotion contract is successfully created and sent, but the first Sign webhook can arrive before DHub stores the returned Sign contract ID in ESignRecord.

Observed event:

contractId: 81
requestKey: annual-leave-promotion:first:cbaa9703-0a93-4f95-8370-61337d2086a4:884fa537-04e4-4d0e-a10b-b50a9068e484
statusCode: signer_waiting

Observed sequence:

10:26:06.935  Sign ticket use succeeded (201)
10:26:06.965  Sign produced signer_waiting webhook
10:26:06.966  create-and-send-from-template returned (201)
10:26:06.980  DHub webhook returned 404 Record not found
10:26:06.988  DHub deadline synchronization succeeded (201)

The contract send itself succeeded. This bug loses or delays the first status event and produces a misleading error log.

Do not retry contract creation for contract 81 as a way to repair this webhook failure.

2. Confirmed Root Cause

DHub creates an ESignRecord with externalRequestKey before calling Sign, but herebyContractId is still null.

Current DHub webhook lookup:

recordRepo.findOne({
  where: { herebyContractId: contractId, deletedAt: IsNull() },
});

Sign emits the signer_waiting webhook during finalization, before create-and-send-from-template returns. DHub therefore has not yet executed:

ESignRecord.update(eSign.id, { herebyContractId: sent.contractId, ... });

The webhook route cannot find the pre-created record and returns 404.

3. Required Design

3.1 Stable tenant reference

Modern DHub-created Sign requests must carry a stable externalOrganizationId from DHub to Sign.

• DHub sends its organizationId in create-and-send-from-template.
• Sign verifies that the external organization mapping belongs to the authenticated API credential/corporation.
• Sign stores the value with dhub_external_contract_requests.
• Every modern webhook echoes externalOrganizationId inside the signed payload.

Do not trust this value before webhook signature verification. It is a lookup scope, not authentication by itself.

3.2 DHub webhook resolution order

For modern DHub events:

1. Validate the required event envelope shape:

   • externalOrganizationId
   • requestKey
   • contractId
   • eventId, eventType, occurredAt, statusVersion

2. Load the active Hereby Sign credential using organizationId = externalOrganizationId.

3. Verify the timestamp and HMAC signature with that organization's webhook secret.

4. Query ESignRecord with the tenant scope:

   organizationId = externalOrganizationId
   externalRequestKey = requestKey
   deletedAt IS NULL

5. Bind herebyContractId after successful authentication:

   • null → conditionally set it to the webhook contractId;
   • same value → continue idempotently;
   • different value → reject and log a security/integrity conflict.

6. Insert the webhook inbox event.

7. Return 200 only after the inbox insert is durable or the same eventId already exists.

The route must not perform an unscoped ESignRecord query by requestKey. All organization-owned queries must include organizationId.

3.3 Inbox processing

The route must finish the authenticated contract binding before inserting the inbox entry. The worker can then continue using:

organizationId + contractId

The worker must preserve the current rules:

• duplicate eventId is processed once;
• older/equal statusVersion cannot overwrite a newer state;
• a mismatched requestKey fails safely;
• failures remain retryable in the inbox.

3.4 Sign delivery durability

The receiver-side fix closes the current race. Sign should also treat webhook delivery as an at-least-once operation:

• persist outbound webhook events before delivery;
• retry non-2xx responses with bounded exponential backoff;
• reuse the same eventId for every retry;
• do not block or roll back a successfully sent contract because the callback is temporarily unavailable.

If a durable Sign webhook outbox does not already exist, implement it as a separate follow-up ticket rather than hiding retries inside an in-memory timer.

4. Hereby Sign Changes

Repository:

/Users/joshua/Project/FIG/hereby/hereby-sign

Expected areas:

apps/contract-server/src/app/contract/dto/contract.external.dto.ts
apps/contract-server/src/app/contract/contract.service.ts
apps/contract-server/src/database/migrations/
apps/contract-server/src/app/contract/__tests__/contract.external.service.spec.ts

Implementation requirements:

1. Add externalOrganizationId to the create-and-send DTO.
2. Validate it against external_integration_mappings for:
   • provider dhub;
   • authenticated corp_id;
   • authenticated open_api_credential_id;
   • exact external_organization_id.
3. Snapshot it in dhub_external_contract_requests using a forward migration.
4. Include the stored value in all DHub webhook events.
5. Include it in the HMAC-signed JSON payload.
6. Never accept a caller-provided organization reference that does not match the authenticated mapping.

Migration requirements:

• add external_organization_id varchar(64);
• backfill existing DHub rows from external_integration_mappings using the authenticated user/corporation mapping;
• fail the migration if a row is ambiguous instead of guessing;
• make the column non-null after successful backfill if all supported rows are DHub-owned.

Do not edit the existing applied migration. Add a new migration.

5. DHub Changes

Repository:

/Users/joshua/Project/FIG/hereby/hereby-dhub

Expected areas:

packages/api/src/services/hereby-contract.service.ts
packages/api/src/routes/hereby-contract-webhook.routes.ts
packages/api/src/routes/hereby-contract-webhook.routes.test.ts
packages/api/src/jobs/sign-webhook-inbox.job.ts
packages/api/src/jobs/sign-webhook-inbox.job.test.ts

Implementation requirements:

1. Send organizationId as externalOrganizationId in the Sign request.
2. Extend the webhook body schema/type with the new required field for modern events.
3. Resolve the credential by the supplied organization scope, then verify the signature.
4. Resolve the record by organizationId + externalRequestKey.
5. Bind herebyContractId with a conditional update after authentication.
6. Re-read or validate the conditional update result to handle concurrent DHub response processing.
7. Save the inbox event only after the record is bound.
8. Keep the existing legacy webhook branch unchanged; do not introduce another compatibility path.

Recommended conditional binding semantics:

UPDATE e_sign_records
SET herebyContractId = :contractId
WHERE id = :recordId
  AND organizationId = :organizationId
  AND externalRequestKey = :requestKey
  AND deletedAt IS NULL
  AND (herebyContractId IS NULL OR herebyContractId = :contractId)

The implementation must use TypeORM's correctly quoted database column names and verify the affected row count.

6. Tests

Sign unit tests

• accepts a matching external organization mapping;
• rejects an organization reference belonging to another mapping;
• persists externalOrganizationId on the request;
• emits it in the webhook payload;
• signs the exact payload containing it;
• idempotent retry keeps the same organization reference and eventId.

DHub route tests

• pre-created record has externalRequestKey but no contract ID;
• correctly signed webhook resolves it by organization + request key;
• route binds the contract ID, stores one inbox row, and returns 200;
• duplicate event returns 200 without another inbox row;
• wrong organization cannot resolve another tenant's record;
• valid signature from organization A cannot bind organization B's record;
• conflicting existing contract ID is rejected;
• invalid signature cannot bind a contract ID;
• missing required modern event fields is rejected.

DHub worker tests

• newly bound record is processed successfully;
• duplicate/older status version does not regress state;
• request-key mismatch becomes a retryable failed inbox item.

7. Verification Commands

Sign:

pnpm --filter hereby-contract-server test -- --runInBand src/app/contract/__tests__/contract.external.service.spec.ts
pnpm --filter hereby-contract-server build

DHub:

pnpm --filter @hereby/api exec vitest run src/routes/hereby-contract-webhook.routes.test.ts
pnpm --filter @hereby/api exec vitest run src/jobs/sign-webhook-inbox.job.test.ts
pnpm --filter @hereby/api build
pnpm lint:check

Then run the repository-required full build before handoff.

8. Local Manual QA

Use a new test employee/request after both services and migrations are restarted.

1. Send one annual-leave first notice.
2. Confirm Sign ticket usage happens once.
3. Confirm one contract is created.
4. Confirm the first signer_waiting webhook returns 200, not 404.
5. Confirm ESignRecord.herebyContractId is bound to that contract.
6. Confirm one inbox entry exists for the event ID.
7. Confirm the inbox worker processes it.
8. Complete the employee signature.
9. Confirm the completion webhook also returns 200 and DHub becomes completed.
10. Repeat the same API request key and confirm no extra contract, ticket charge, message, or inbox event is created.

Before any local send, use only designated test employee contact data. Do not use a real employee phone number or email address.

9. Existing Contract 81

Perform read-only checks first:

• DHub ESignRecord.herebyContractId = 81;
• request key matches the observed request;
• status is sent/signer_waiting;
• Sign request is linked to contract 81;
• no duplicate contract was created after the successful send.

If these values are already correct, no data mutation or resend is required. The missing first webhook can be treated as superseded by DHub's synchronous send result.

If they are inconsistent, write a separate one-time recovery procedure with exact row predicates and obtain Eron's approval before executing it.

10. Acceptance Criteria

• [x] A webhook arriving before the create-and-send response binds the pre-created DHub record.
• [x] The first webhook returns 200 and is stored durably.
• [x] No organization-scoped query is performed without an organization filter in the modern webhook branch.
• [x] Cross-tenant request-key or credential substitution is rejected.
• [x] Invalid signatures cannot mutate ESignRecord.
• [x] Duplicate delivery does not duplicate inbox processing.
• [x] Contract creation remains idempotent and consumes one ticket only.
• [x] Contract 81 is not resent merely to repair the missed webhook.
• [x] Targeted tests and builds pass.
• [ ] Migration execution, repository lint, and live manual QA pass.

Implementation notes:

• Sign now requires and validates externalOrganizationId, snapshots it on dhub_external_contract_requests, and includes it in the signed webhook payload.
• DHub verifies the modern signature before resolving by organizationId + externalRequestKey, conditionally binds herebyContractId, and only then inserts the inbox event.
• The new Sign migration intentionally fails before any DDL if historical DHub request rows cannot be mapped to exactly one organization. It has not been run against local data in this change.
• Repository-wide DHub lint still reports pre-existing employee-app CSS and maintenance-script diagnostics; the changed DHub files pass targeted Biome checks.
• Existing contract 81 was not resent or mutated.

11. Out of Scope

• changing annual-leave business copy;
• changing template PDF or field coordinates;
• automatic daily sending;
• retrying contract creation to compensate for webhook delivery;
• a broad rewrite of legacy webhook authentication.