The focus of Chen Goldberg’s job has shifted in the last six months.
As GM and VP of engineering at Google Cloud, Goldberg has a front-row seat to the ongoing rush to deploy large language models and generative AI tools in the cloud.
Microsoft, through its partnership with ChatGPT developer OpenAI, took the lead, introducing generative AI search capabilities in February. Google and its parent company Alphabet responded in March with a suite of new, cloud-based generative AI tools.
“The demand from customers has definitely accelerated efforts that we already had underway,” Goldberg said.
Technological breakthroughs create deployment challenges, particularly at the enterprise level.
Advances in generative AI have raised specific data security and user safety concerns, leading a number of large companies to enact internal restrictions, while researchers and executives warn of broader global consequences in the absence of regulatory oversight.
Goldberg, who leads Google Cloud’s Kubernetes and serverless team of over 1,000 engineers, is focused on how to integrate generative AI into an existing ecosystem and what enterprise customers will want to do with the technology.
Efficiency gains, whether through summarizing emails or coding assistance, are one promising category of use cases, Goldberg said. The new models also have cybersecurity and product development potential.
Goldberg’s team recently trained Google’s PaLM2 model to identify patterns of malicious code and is exploring natural language application development.
“There are opportunities to create new experiences and new products that actually would not have been possible before,” Goldberg said.
Right-sizing AI
As the new technology proliferates, infrastructure will have to adjust.
LLMs chew through vast quantities of data and it takes massive processing power to support enterprise-grade generative AI workloads.
Hyperscalers AWS, Microsoft and Google Cloud have begun retooling data centers, installing graphics processing units and other chip technologies capable of handling the anticipated surge.
“To meet that demand, we are working on making sure that GPUs are easily available and that we have storage features that allow you to easily do training,” Goldberg said.
Model training presents several engineering challenges. Checkpoints, which provide snapshots of a model's progress for backup purposes, eat up additional storage, adding cost to an already expensive operation.
Devising practical ways to package the technology for manageable enterprise use cases is an engineering problem with solutions.
“We don’t think that one model rules them all or that the bigger the model, the better it is,” said Goldberg. “We are helping customers find the right size at the right cost per use case.”
For most organizations, start-to-finish training won’t be necessary. Feeding additional data to pre-trained foundation models will yield a broad range of applications, according to Goldberg.
“What we will see more and more is that enterprises will want to take their data, fine-tune models and come up with new products, new solutions and see what's possible,” said Goldberg. “Some will be internal use cases and some will be for end users.”
Safeguarding proprietary data has already emerged as a major concern, even among those most eager for early adoption. Integrating new tools with existing tech at a time when recruiting and retaining skilled technicians remains difficult, is also a looming issue.
“Understanding those challenges is critical because we're not building products and services just for the sake of the technology,” Goldberg said.
Solutions engineered by Goldberg’s team are regularly put to practical tests internally.
Earlier this year, a data center-bound chip development infrastructure team at Alphabet completed its migration to Google Cloud. Prior to that, Goldberg coordinated a hybrid multicloud solution for the company’s backend systems.
“We talk about drinking our own champagne,” Goldberg said. “It’s not always easy, but within Google we are using our own cloud technology. It's a very good way of learning.”
