AI factories run on graphics processing units that cost a fortune and work at blistering speed, yet those chips increasingly sit idle waiting for data. As the bottleneck moves from compute to memory and storage, NAND flash is being asked to do far more than hold files.
That shift is putting data movement constraints at the center of AI infrastructure planning. A GPU operates at about 20 to 22 terabytes per second, and every tier of the hierarchy below it adds delay, according to Pichai Balaji (pictured), vice president of products and marketing at ScaleFlux Inc.
“When you move to DRAM, it is almost 37 to 40 times slower than what the GPU wants. You go even further down to the storage side, it is almost 800 times slower,” Balaji said. “A bottleneck is created all throughout when the data is moving from the GPU into the memory, into the storage. The bottleneck just gets amplified.”
Balaji spoke with theCUBE Research’s John Furrier at the AI Data Pipeline Forum, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed AI factories, smarter controllers and using NAND flash as active memory for AI workloads. (* Disclosure below.)
Rethinking NAND for AI factories
ScaleFlux builds both SSD controllers and Compute Express Link memory controllers, pairing the silicon with firmware and a full reference architecture. Its AI-optimized SSD platform targets key-value cache offload, the store of conversational context that inference workloads lean on. In that role, flash can stand in for memory rather than serve as a passive archive, Balaji explained.
“This is where the NAND device need not be used more as a storage device, it can be used as a memory device. A good intelligent SSD controller or a CXL memory controller can take advantage of the NAND characteristics and use it in different ways,” Balaji said.
ScaleFlux has demonstrated more than 200 flexible data placement streams and 7 to 10 drive writes per day on standard triple-level-cell NAND, he noted. Because the company is NAND agnostic, customers can also build around whatever flash they can source in a tight supply market.
“We can enable that NAND, we can give the full reference design, we can bring in our contract manufacturing partner and put it all together to scale,” Balaji said.
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the AI Data Pipeline Forum:
(* Disclosure: ScaleFlux sponsored this segment of theCUBE. Neither ScaleFlux nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)