ORLANDO, Fla. — As CIOs lead their organizations through AI-driven change, runaway costs and accuracy hiccups continue to threaten momentum, analysts said during the opening keynote of Gartner’s IT Symposium/Xpo Monday. The stakes are high for IT chiefs to solve the adoption puzzle, and it’s not an easy feat.
“This is the moment where people will look back and say CIOs and AI leaders either made a mistake or they helped put us on a path to greatness,” said Daryl Plummer, distinguished VP analyst and chief of research at Gartner.
With more than 7,000 IT executives in attendance, Gartner analysts said CIOs must create an "AI survival kit" that includes measures to mitigate accuracy and cost risks while supporting innovation.
For now, most organizations rely heavily on human-in-the-loop oversight to catch AI-generated mistakes. This approach has proven useful thus far, but employees are unlikely to keep up as AI adoption grows.
“The human-in-the-loop equation is collapsing off itself,” Gartner VP Analyst Alicia Mullery said.
Establishing formal metrics to test AI outputs can help organizations keep pace with the sheer amount of AI-generated content. CIOs could also adopt a two model fact checking system, Gartner recommended. This approach enables one model to check another’s outputs.
To ensure accuracy isn’t slipping below standards, IT leaders can establish a measure of when AI accuracy is good enough for a specific initiative. “Good enough may be harder to achieve than you think, because we hold AI to a higher standard," Mullery said.
Costs are part of the equation, too. Gartner sounded the alarm on cost-related challenges last year, characterizing the magnitude of the risk by comparing it to security. The risk has continued, unabated.
“For every AI tool that you buy, there will be 10 ancillary costs you didn’t anticipate, such as managing access credentials for autonomous agents, acquiring new data sets to ground AI or packing your accuracy survival kit,” Plummer said.
Gartner estimates that AI implementation costs can near $2 million on day one. Training takes 25% more effort, and change management can add 200% more effort to AI initiatives.
“If previous tech introductions had upfront transition costs, AI has a transition mortgage that you just keep paying,” Mullery said.
Measuring readiness
To get a sense of an organization’s AI readiness, CIOs must take several factors into account, including technical and workforce maturity.
For the technology to be ready does not mean a model that can pass the world’s toughest test or outthink a human, Mullery said, warning leaders of getting wrapped up in vendor-driven hype. CIOs should choose tools that can solve the business problem they want to remedy, rather than select the latest toy to hit the market. Enterprises that can drown out noise and instead focus on value opportunities will be better positioned to make inroads, according to Gartner analysts.
CIOs will need to help get their organization’s workforce up to speed as well by investing in training and upskilling opportunities.
“AI readiness grows much faster than human readiness,” Mullery said. “If all the vendors stopped innovating with AI today, we’d still have years before we could catch up — that’s why we can’t keep up.”
Most organizations are contending with low workforce readiness. More than 70% of CIOs said their workforce is not ready for AI, according to a Gartner survey.
“AI unleashes a toxic mix of a steep learning curve and the primal fear that AI is going to replace us,” Mullery said.
The AI-driven chaos is distracting, especially as big tech companies rework their talent pools to position more roles around AI. CIOs also need to be aware of potential AI-driven behavioral byproducts, which represent the unintended changes to human skills and actions.
“If you’re not using your coding skills every day, you are losing coding skills everyday,” Plummer said. “AI is stealing your skills, and it’s happening faster than you can imagine. That is skills atrophy, and it’s a behavior byproduct you need to watch for.”
