Enterprises pushing AI from pilot to production are learning that data, not GPUs, is the hardest part, and open table formats such as Apache Iceberg are part of the answer. Demand for AI infrastructure shows few signs of slowing, but much of the information companies need is still scattered, ungoverned and out of reach.
That pressure is turning storage into an active layer that classifies, governs and serves data to models. Unstructured files such as logs and medical images hold intelligence AI can now extract, according to Arun Prasad Raman (pictured, left), vice president of hybrid cloud and AI infrastructure at Hitachi Vantara Corp.
“Storage is not seen as just storing the data,” Raman said. “You need to bring in more intelligence natively rather than moving the data to another replication and bringing it back for, say, indexing. If it can do it natively, that’s where the value add is.”
Raman and Jeff Lundberg (right), principal product and solutions manager at Hitachi Vantara, spoke with theCUBE Research’s John Furrier and Gemma Allen at SC26, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed AI’s pilot-to-production gap, Apache Iceberg and how Hitachi Vantara is adding intelligence to storage. (* Disclosure below.)
Apache Iceberg and the push for intelligent storage
Reference architectures for AI factories and data platforms are largely settled. Most projects stall because data is spread across on-premises systems, software-as-a-service applications and the cloud, while access control, governance and shifting regulations add friction, according to Lundberg. Hitachi Vantara is responding with industry-specific solutions built on its Hitachi iQ stack and its sister businesses’ rail and energy experience, he explained.
“It’s about the outcome more so than the technology that we’re trying to put into place,” Lundberg said. “That’s an enabler to get you to the solution you’re trying to get to.”
Much of that work now happens in storage. Last week, Hitachi Vantara expanded its VSP One platform with new object storage and data lakehouse capabilities. A service in its object store flags files containing personal information, such as Social Security numbers. Data scientists can then keep that data out of specific models or block its replication to another country, Raman explained.
“That’s the intelligence we are adding into the storage itself,” Raman said. “So that’s where the evolution of storage is making it more intelligent.”
Analytics follow the same logic. CT scan metadata can be stored in Apache Iceberg format on the company’s object store and queried directly. An application could list every abdominal scan from the past year without a separate extract, transform and load or ETL process or database, Raman noted.
“Whether it’s a backup or it’s 10 years old, it still carries some intelligence,” Raman said. “As long as it carries intelligence, it becomes important to AI.”
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of SC26:
(* Disclosure: Hitachi Vantara Corp. sponsored this segment of theCUBE. Neither Hitachi Vantara Corp. nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)
