Why your integration keeps breaking at month-end

Why your integration keeps breaking at month-end

Integration work is easy to underestimate because the happy path is simple. Call the endpoint, map the fields, store the result. The happy path is not what runs on the last working day of the month.

Slow is different from down

A system that is down fails fast and obviously. A system that is merely slow holds your connection open, exhausts your pool, and takes your service down with it. If your timeout is longer than your caller's patience, an upstream slowdown becomes your outage.

Set aggressive timeouts and a circuit breaker. Failing quickly and clearly is nearly always better than waiting hopefully.

Retries need idempotency to be safe

Retrying a timed-out request is only sensible if the other side can recognise it as a repeat. Otherwise the thing you added for reliability starts creating duplicate orders, duplicate invoices and duplicate payments, under the load that made the timeout happen.

  • Send a stable idempotency key with anything that creates or changes state.
  • Back off exponentially, with jitter, so retries do not arrive as a synchronised wave.
  • Cap the attempts and route what is left to a dead-letter queue a person can inspect.

Reconcile, do not assume

Both sides should be able to answer "how many records did you see today, and what were their totals?" A daily reconciliation that compares those answers turns a silent divergence into an alert on the day it starts.

Test against the bad day

Point the integration at a stub that is slow, that returns partial responses, and that fails one call in twenty. Almost everything that will hurt you in production shows up within an hour of doing that, and it is far cheaper to find it on a Tuesday afternoon than at month-end.