Twenty-five containers, two nodes, and an AI agent with a key to all of it
Letting an AI operate production infrastructure is either reckless or well-governed. The difference is entirely in the boundaries.
Our own platform
We run the platform our own work is delivered on, and we let AI agents operate it. That required deciding, concretely, what an agent is allowed to touch - which turns out to be the same governance question every organisation adopting AI is about to face.
The challenge
The interesting question about AI in operations is not whether a model can run a command. It obviously can. It is what happens the first time it runs the wrong one.
Most answers to that are aspirational - a policy document, a review step, a promise that a human is watching. Under real operational pressure, those degrade quickly.
We wanted an environment where AI agents genuinely help operate infrastructure, and where the consequences of a bad decision are bounded by architecture rather than by attention. That means the boundaries must be real: separate nodes, firewall rules, scoped credentials, audited access - the same controls an enterprise would demand, applied to a platform small enough to change quickly.
What we did
One control plane, explicit targets
A Model Context Protocol server exposes a defined set of operational targets - virtualisation, containers, databases, DNS, monitoring - rather than handing over shell access and hoping.
Isolate the workloads that matter
Client-facing workloads run on physically separate hardware with firewall boundary rules and narrowly scoped reverse-proxy exceptions, so a mistake in one context cannot reach another.
Infrastructure as code, throughout
Containers, DNS, monitoring and deployment paths are defined in version control, so what an agent changes is visible as a diff rather than as a surprise.
Observability as a precondition
Unified metrics, dashboards and alerting exist before automation is granted access. An automated action you cannot observe is not an automation, it is a rumour.
Write down the incidents
A reboot-ordering dependency caused a real outage. The failure, the diagnosis and the fix are documented, because undocumented incidents get repeated.
How it works
Two nodes, twenty-five containers
A virtualised estate running application, database, monitoring and inference workloads, plus a dedicated backup target.
Shared inference host
A GPU host provides local model inference to every other service, so AI workloads run on infrastructure we control rather than being sent elsewhere.
Unified observability
Metrics collection, dashboards as code and alerting across the estate, validated on every change.
Automated health reporting
Scheduled checks report the state of the estate on a fixed cadence, so drift surfaces on a schedule instead of during an incident.
Physical separation for client work
Client workloads are not neighbours of internal experiments. They run on separate hardware behind their own firewall rules.
Scoped, enumerated access
Agents reach a defined list of operational targets. Capability is granted deliberately rather than inherited from a shell.
Everything observable
Automated actions land in the same monitoring and version control as human ones, so review is possible after the fact.
Decisions recorded
Architectural choices are written as decision records with their context and consequences, not just their outcome.
Technologies
The result
The estate runs across two virtualisation nodes and roughly twenty-five containers, with client workloads isolated on separate hardware, infrastructure defined in version control, and unified monitoring across everything.
We are deliberately transparent that this is our own platform rather than a client engagement. That is exactly what makes it useful to talk about: it is the environment where we can show the whole thing - the architecture, the decision records, and the incident we caused and then fixed.
If you are working out how to let AI agents near your infrastructure without handing over the keys, this is the conversation we most enjoy having.
No addresses, hostnames, container identifiers, credentials or topology details are published here. Security-relevant configuration is described only in general terms.
Related work
Before the invoice button went live, we built the thing that stops it
The system was about to start emailing customers. Nothing prevented it from emailing all of them.
The first automated read looked perfect. Every field that mattered was empty
An extraction that returns a complete-looking result with nothing important in it is worse than one that fails.