An AI agent spends much of its life waiting, which can carry a substantial infrastructure cost. A conversational chatbot typically waits for a person between turns. An agent can initiate the next step as soon as a tool returns, creating a sustained sequence of dependent requests. A chatbot handles one prompt at a time, and the pause before the next one is a person thinking, so the serving layer can treat each request as self-contained and scale by adding replicas.
An agent resolving a customer’s billing dispute works in a loop: It reads the ticket, calls the billing system, waits, reasons again, calls another tool, and then responds. Each pause is a machine waiting on another machine, each turn depends on the one before, and tasks can run for minutes or hours.
The infrastructure has to hold the agent’s state across those pauses and keep its context ready for the next step. Buying infrastructure for agents as if they were chatbots means organizations are paying to wait.

Why faster chips alone do not solve the problem
Adding parallel hardware does not remove dependencies between turns. The waiting creates three distinct costs.
- Latency. Ten four-second database queries add 40 seconds to a task. A faster accelerator does not make those queries return sooner.
- Accelerator memory. While an agent waits, its conversation's working memory (the KV cache) can remain on the accelerator, occupying capacity other requests could use. It can move to slower memory for later retrieval or be discarded and rebuilt, repeating computation.
- Idle compute. An accelerator doesn’t have to wait for one agent to finish its task, as the serving software can handle other requests in the meantime. An agent’s execution environment can still reserve CPU and memory while it waits for a tool response.

According to recent Omdia research, 81% of organizations stated that agentic AI is a strategic priority across IT and cloud, and 80% reported that ensuring positive ROI for their AI initiatives is a top concern.1 Omdia research also found that 81% of organizations considered scaling AI agents into full production more complex than they initially expected, while 61% identified agent costs as a major barrier to adoption.2
Four questions every CIO must answer before agents reach production
Full-stack co-design refers to engineering chips, memory, networking, storage, serving software, and orchestration together around the workload. Four questions show why evaluating accelerators alone misses where the money goes. Google's AI Hypercomputer is the example for each.
- Does the gateway know what the model already holds in memory? The best server for an agent's next turn is often the one that already holds the conversation's context because another server may need to retrieve or rebuild that context. A gateway that knows this routes accordingly, and one that does not makes the organization pay for unnecessary rebuilding across turns. For example, GKE Inference Gateway routes this way, and Google reports that its predictive routing cuts time to first token (TTFT) by up to 70%.
- What happens to an agent's state while it waits? An agent that executes generated code needs its own isolated environment, and an environment that sits idle while a tool call returns still costs money. The decision is whether to pay for idle agents or to suspend them and accept a short delay when work resumes. For example, GKE Agent Sandbox provides the isolation, and Google reports allocating 300 sandboxes per second at sub-second latency. Agent Substrate, released as open source in May 2026, suspends idle agents and restores them with sub-second responsiveness. It is available on GKE for evaluation and non-production use.
- Which stage of the loop runs on which silicon? Most of an agent's loop is orchestration and tool calls, which run well on ordinary CPUs. Inference accelerators handle the model's prompt processing and token generation, and training hardware handles model improvement. Running the whole loop on the most expensive hardware means paying accelerator prices for work a CPU could do. Google's eighth-generation TPUs come in a training variant (TPU 8t) and an inference variant (TPU 8i), both expected later in 2026, alongside Axion CPUs and NVIDIA GPUs. Framework support eases moving workloads onto TPUs, but portability still has to be tested with the buyer's own models.
- Where does the agent's context live, and how fast does it get back to the accelerator? Long-running agents can accumulate more context than available accelerator memory can hold. Retaining that context for reuse can require moving some of its KV cache to cheaper memory or storage. The slower that round trip, the longer the agent waits before its next inference step. In Google's tests, keeping the KV cache in cheaper memory tiers cut TTFT by 44% for 10,000-token prompts. Offloading context to Managed Lustre shared storage cut GPU hours by nearly 60% for 50,000-token prompts.
These decisions interact: Keeping context close may speed the next turn but consume memory, while releasing resources can introduce restoration delays. Co-design makes those trade-offs explicit across the system instead of optimizing each component independently.

How to run the proof
The performance figures above are Google's own. CIOs should ask for reproduction on their own workloads, including the costs of retaining, moving, and rebuilding state. The four questions double as the test plan. Measure cache hit rates at the gateway and resume times for suspended agents, useful work per hardware pool, and time spent waiting on storage.
Co-design within one provider reduces integration work but can create another silo if components only work well together there. Open components, including llm-d and Agent Substrate, help address that concern. Whether they deliver comparable performance elsewhere remains the test.
The procurement question, therefore, changes: Instead of asking which accelerator wins a benchmark, organizations should measure successfully completed tasks per dollar at the required quality and response time. Vendors should answer with measurements from the buyer's own environment.
Learn how full-stack system co-design powers the next generation of AI agents with AI Hypercomputer.
References