NEW YORK — Model routing is emerging as a key tool for enterprises looking to curb AI costs amid major investments into the technology’s adoption.
Bank of America plans to double its AI budget next year with a focus on triaging more advanced dialogue with its AI-enabled financial assistant Erica, said Hari Gopalkrishnan, chief technology and information officer at Bank of America. The bank is also working to improve the AI tool’s institutional memory and ability to conduct more complex transactions.
More advanced capabilities will need to be facilitated by more advanced AI models, but that’s not the case for every interaction, Gopalkrishnan said during the Evident AI Symposium last week. The bank’s AI routing layer, Orchestra, uses a small, classifier model to route queries to the most financially-effective model, he said.
Managing token spend and routing to lower cost models through Orchestra is an important and effective cost management tactic for the bank, he said.
“We didn’t want to be beholden to any given model,” he said. “We wanted to be flexible, to have the existing small language models, the existing fine-tuned models. We expected fully — and it turned out — that there are simpler classifier models that do very easy decision things that you don’t need a large language model for.”
Smaller, fine-tuned models have also benefited Capital One, said Prem Natarajan, chief enterprise AI officer and chief scientist at Capital One.
The bank distilled its open weight models, fine-tuned with proprietary data, into smaller models supporting Chat Concierge, a multiagent application that helps car buyers navigate the process. The smaller models helped Capital One reduce latency by five times and increase customer engagement for dealerships by 50%, he said during the event.
Vendors are adding their own model routing tools and features to help enterprises manage output quality and costs. Snowflake, for example, added dynamic model routing to its Cortex AI Gateway that automatically picks the appropriate AI model to handle specific queries.
Agent guardrails and data
Bolstering AI agent guardrails and keeping data ready are also top of mind for banking executives.
A key piece of Bank of America’s AI strategy heading into 2027 includes highly-controlled guardrails that ensure agents stay within their allotted runtime and aren’t allowed to complete multiple tasks autonomously, Gopalkrishnan said.
“The instrumentation, telemetry, being able to have a feedback loop, those are some of the things we are evolving,” he said.
For Capital One, keeping a strong data foundation will continue to play a key role in supporting the bank’s AI journey, Natarajan said. The bank’s early move to a modern tech stack in the cloud and use of machine learning over the years taught Capital One the criticality of data to unlocking insights, he added.
“We have copious amounts of well-managed, well-curated data,” he said.
Generative AI and agents raise expectations of how ready the data ecosystem needs to be, he said. While Capital One has made significant investments in its foundational data layer, it will continue to elevate data capabilities to support AI, he said.
“The engineering and architectural decisions you make lower down in the stack are either massive enablers or massive constrainers of what you can do up top,” he said. If parts of the architecture need to be redone, Capital One doesn’t hesitate, he added.
“We go back and we rebuild from the bottom up,” he said.
