Multi-Org CRM Platform

• No transactions across organisations, and no ETags or idempotency keys inside one. Correctness had to be built on top.
• A vendor API with undocumented behaviour. I found several of those quirks in production, and each one became a test.
• Six organisations in production from the start. One slow or broken organisation must not take down a page that also reads from the other five.
• A security model that had to change while people were using it, without locking anyone out.
• When the outcome of a call is uncertain, the system has to fail closed rather than guess.

A Java 21 and Spring Boot platform, with Postgres and Firestore for coordination state, running on Cloud Run. I was the primary developer; one teammate contributed a handful of commits. Here is the shape of it, then the four pieces I think matter most.

Receipts for writes that span two organisations

One user action can create a record in one organisation and then a linked deal in another. If the second call times out, I cannot tell whether it landed. A blind retry might create a duplicate. Giving up might leave half the work done.

So every operation gets a durable receipt in Firestore. The receipt moves through a sealed set of 12 transitions. Two of its states are explicitly unknown: the outcome of an external call is not known. The only way out of an unknown state is a read-only reconciliation that looks at the CRM and proves what happened. Before every external call, the platform writes a dispatch marker saying it is about to make that call.

marker → create in org A → marker → create in org B → COMPLETE → 201

Claims on a receipt use compare-and-set with leases and attempt fencing, so two concurrent resumes cannot both dispatch. The caller gets its 201 only after COMPLETE is written durably. The Camel entry points never redeliver, so retries only ever happen through the receipt.

A write fence for an API with no ETags

Required-field rules per pipeline stage have to hold on every write. The CRM’s update call has no If-Match, so two instances could each check the rules against a stale read and then overwrite each other. I built the missing concurrency control in Postgres, with one fence row per record.

reserve → read → evaluate → write → finalise

The reserve step claims the record. The write is a single update, with no database connection held while it runs. The finalise step is a compare-and-set that only succeeds if the fence is still ours. Rules live in a versioned store that reconciles itself against the CRM’s metadata by content hash, and checks fail closed if the metadata version does not match. After launch I added a release path for reservations that failed before they could finalise.

Switching permission engines in production

Access control had to move from fixed roles to configurable policies: brand scope, own, team or reporting-tree records, parent-record readability and masked fields. I did not want a big-bang switch across six live organisations.

• The new engine is deny-by-default at the catalog level and masks fields it should not show.
• A shadow comparator runs the new engine next to the old one on real requests, stores only the differences and never affects the response.
• Readiness gates decide when a resource may cut over, so nobody gets locked out on the strength of a hunch.
• Ownership resolves from either a platform user id or the CRM owner’s email, so no data backfill was needed.
• List pages use batch evaluation rather than one check per row and per field. The story of why is in the hard problems below.

Performance across six organisations

Pages that read from six organisations used to pay each one’s latency in sequence. A bounded parallel fan-out replaced that, so a page now waits only for its slowest organisation, and a broken organisation degrades its own part of the page and gets reported instead of sinking the response. Board pagination moved into a single query in the CRM’s own query language, which took a board interaction from about 25 CRM calls down to 1 or 2 at roughly 5,000 projects.

An audit log nobody can forget to write

Rather than asking every code path to remember to audit, a Spring post-processor wraps every CRM client in a proxy. Each successful create, update and delete is captured with its before-state and request context. Database triggers reject UPDATE and DELETE on the audit rows, so the log is append-only even against my own code.

• CRM calls per board interaction went from about 25 to 1 or 2, at roughly 5,000 projects.
• A page across six organisations waits only for its slowest one, not the sum of all six.
• A 200-row page used to spend about 38 seconds in rule context before batch evaluation replaced per-row checks.
• A bug that created a duplicate project on every ingestion run is gone.
• The backend test suite went from nothing to more than 2,300 tests in four months, including concurrency tests against embedded Postgres and the Firestore emulator.

I am deliberately not quoting latency after the fixes. I do not have those measurements yet, and I would rather leave the number out than guess it.

• Measure. Record p50 and p95 for the endpoints fixed in the slowdown, count CRM calls per page view, and see how often receipts actually end in an unknown state.
• Handle rate limiting properly. The hand-written CRM clients repeat the same timeout and retry logic, and none of them understands a 429 or a retry-after header. I would pull that into one shared layer.
• Give the dead-letter queues a consumer. They are written to today, but nothing replays them.
• Cache the organisation graph and metadata snapshots, which are re-read several times per request, and bound the fan-out queue.
• Give the first record in a two-step create the same idempotency treatment as the second. A lost response on that step still needs manual handling.