AI data centers have spent heavily on faster accelerators, denser racks and liquid cooling to keep that hardware running. The data pipeline that feeds those processors has not kept up, and expensive compute ends up waiting on memory and storage.
That gap is getting more attention as AI infrastructure bottlenecks become central to data center planning. The fix is to rethink the external memory and storage tiers behind the GPU, according to Hao Zhong (pictured), co-founder and chief executive officer of ScaleFlux Inc.
“GPU and TPU performance has been advancing incredibly fast, almost more than 1,000 times in the last eight years,” Zhong said. “And we need to keep up with the data pipeline. Otherwise, we are not going to get enough efficiency out of that.”
Zhong spoke with theCUBE Research’s John Furrier at the AI Data Pipeline Forum, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed AI memory bottlenecks, liquid cooling and the shift toward inference and agentic workloads. (* Disclosure below.)
Looking past liquid cooling to the AI data pipeline
Zhong describes AI infrastructure as a pipeline with several stages. Processors with high-bandwidth memory make up the first stage, and external DRAM and SSDs form the second and third, he explained. This year, ScaleFlux unveiled new controllers built for those tiers: one for PCIe Gen 6 SSDs and one for Compute Express Link memory.
“What ScaleFlux is helping our ecosystem do is build the controller chips or little processors for the second and the third stage data pipeline,” Zhong said.
The workloads are changing too. As AI applications move from training toward inference and agentic work, systems need much more data close at hand, according to Zhong. That puts capacity, bandwidth and performance in external DRAM and SSDs at the center of infrastructure decisions.
“That’s why the data storage and memory now becomes extremely important and literally become the bottleneck of the future AI,” Zhong said.
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the AI Data Pipeline Forum:
(* Disclosure: Scaleflux sponsored this segment of theCUBE. Neither Scaleflux nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)