Orchestrating 130+ autonomous agents in production

Context

Trilogy runs an enterprise SaaS platform that finds and applies cost savings across customer AWS estates. It has contributed to more than $2B in aggregate cloud savings, and the mechanism behind it is a fleet of autonomous agents that inspect real customer infrastructure and act on what they find.

That last part is the whole problem. An agent that suggests something is a demo. An agent that is trusted to run unattended against someone else's production account is an engineering problem with very little margin.

The problem

Writing one agent is not hard. Writing the hundred and thirtieth is not hard either, if the first hundred and twenty-nine taught you nothing.

The actual problem was that each agent arrived with its own scaffolding. Its own retry logic, its own error handling, its own idea of what a valid result looked like, its own way of reporting what it did. Onboarding a new agent took around two weeks, most of it spent rebuilding infrastructure that already existed somewhere else in a slightly different shape.

At that rate the fleet does not scale, and worse, neither does trust in it. Every agent is a separate argument about whether its output can be believed.

Constraints

• Unattended execution. No human in the loop on the hot path. If an agent is wrong, it is wrong in a customer's account.
• Multi-tenant blast radius. Failure has to be contained to one evaluation, not one customer, and certainly not the fleet.
• Non-deterministic components. LLM calls fail, time out, and occasionally return confident nonsense. The architecture has to assume this rather than hope against it.
• Cost. Millions of evaluations daily, each potentially involving model calls. Per-invocation cost is a first-class design constraint, not an afterthought.

Approach

The move was to stop treating agents as programs and start treating them as workloads on a platform, with the platform owning everything that is not agent-specific logic.

Orchestration as the substrate. AWS Step Functions holds the workflow: 50+ state transitions, explicit retry policies, dead-letter queues for anything that exhausts them. Lambda runs the short work, ECS Fargate the long work. An agent author describes what their agent does. The platform supplies everything about how it survives.

Structured validation on every output. No agent result is trusted because it came back without an exception. Every output passes through schema validation before anything downstream consumes it, which is what makes a non-deterministic component safe to build on. This is the single most important decision in the system. An LLM that returns malformed or implausible output is a caught error, not a silent corruption.

LLMs where judgment is needed, not everywhere. Models handle classification, natural-language output generation, and automated validation of findings. The deterministic parts stay deterministic. Using a model where a conditional would do is how per-invocation costs quietly become the largest line item.

Gateways for governance. Model routing runs through gateway layers, Portkey and TrueFoundry, which enforce access controls, cost and latency policy, and give unified observability across providers. Without this you cannot answer basic questions like which agent spent what, or what happens when one provider degrades.

Two patterns on one layer

Once the substrate existed, it turned out to carry two quite different ways of working:

1. A maintenance pipeline. Agents perform root-cause analysis on existing products and ship the fix themselves, end to end.
2. Parallel feature delivery. Several agents implement different pieces of the same feature simultaneously from a single authoritative spec, instead of one engineer working sequentially through a backlog.

Both plug into the same orchestration layer with no custom scaffolding per team. That is the part I am most pleased with, because it was not designed for the second pattern. The second pattern fit because the layer was drawn at the right boundary.

Outcome

What I took from it

The agents are the easy part. The durable engineering is in the operating model around them: execution policies, tool and service access scoping, retry and failure semantics, and observability that gives engineering leadership real visibility into what the fleet did and why. That work is what separates a pilot from a platform, and it is almost always underestimated.

Validation is the load-bearing wall. Every serious AI-in-production conversation eventually arrives at the same place: you cannot trust the output, so you have to verify it structurally. Teams that design this in from the start scale past the pilot. Teams that add it after the first bad output spend a quarter retrofitting it.

Draw the platform boundary early. The onboarding time dropping from two weeks to three days was not the result of making agents simpler. It came from moving everything that was not agent logic out of the agent.