Before the invoice button went live, we built the thing that stops it
The system was about to start emailing customers. Nothing prevented it from emailing all of them.
A Dutch sun-shading installer
Automated invoicing was days from going live. We stopped and built the blast-radius control first: one funnel every outbound message must pass through, with idempotency claimed in the database before the provider is ever called.
The challenge
Outbound email is the point where a software defect stops being internal. A bug in a report is embarrassing; a bug in a send loop reaches every customer you have, permanently, in about four seconds.
The risk was not hypothetical. When we measured the running system - reading the live container rather than the configuration file - the outbound gate was already open. What was actually preventing customer email was a missing environment variable. That is not a control. That is luck.
The standard mitigations are also the ones that fail under exactly the conditions you need them: a retry that duplicates, a rate limiter that resets itself, an allowlist that fails open when the lookup errors.
What we did
One funnel, no exceptions
Every outbound path was converted to a single guarded function, and a test that scans the source keeps the number of send sites fixed - a new one cannot be added quietly.
Claim before you call
A database unique constraint claims the send key before the email provider is contacted. If the claim fails, the provider is never called. Duplicates become impossible rather than unlikely.
A breaker that does not heal itself
Rate breaches trip a circuit breaker with no automatic recovery. A person must clear it and record a reason. An auto-resetting breaker just spreads an incident over a longer window.
Fail closed, always
If the allowlist cannot be evaluated, nothing sends. Every override is possible but audited, so bypassing the guard leaves a record.
Rehearse against the real schema
The migration was rehearsed on a scratch copy of the production schema, and every constraint was tested negatively - proving it actually refuses bad input, not merely that it exists.
How it works
Ordered precedence
Checks run in a fixed order - invalid, dry run, approval required, allowlist, provider disabled, breaker - so the reason a message did not send is always unambiguous.
Database-level idempotency
The uniqueness guarantee lives in the database, not in application logic, so it holds across processes, restarts and concurrent workers.
Explicit enablement
Live sending requires positive configuration. An absent setting means nothing sends, rather than everything sending.
Audited override
There is a way to force a send, because operationally there has to be. It is logged with its justification.
Concurrency proven, not assumed
Twenty simultaneous calls to send the same message produce exactly one provider call. This is asserted by a test, not argued in a document.
Rate limits proven
A loop of fifty attempts reaches the provider at most ten times and leaves fifty audit entries - so both the limit and the logging are demonstrated.
Negative constraint testing
Each constraint was tested by trying to violate it. A constraint that exists but does not refuse is worse than none, because it is trusted.
Gaps written down
What had not yet been done at handover - migration not applied, no live send performed - was stated explicitly rather than left implied.
Technologies
The result
Five outbound send sites were consolidated behind one guarded funnel, held there by a test that fails if a sixth appears. Thirty-five new tests cover the guard specifically, within a suite of 2,347.
The measured starting state is the part worth remembering: the gate was open, and what was holding email back was an unset variable. Reading the running system rather than the configuration file is the difference between believing you have a control and having one.
Nothing about this makes a demo. It is the work that means the invoice feature can be switched on without anybody holding their breath.
No infrastructure identifiers, provider credentials, allowlist contents or environment details are published here.
Related work
408 messages in a week. Thirty-three were actual customers
One shared inbox was absorbing the whole company. Sorting it had quietly become somebody's job.
Twenty-five containers, two nodes, and an AI agent with a key to all of it
Letting an AI operate production infrastructure is either reckless or well-governed. The difference is entirely in the boundaries.