How to build an AI stack on the fly

Welcome to Runtime! Today: four perspectives on building AI infrastructure that can launch quickly and stand the test of time, ServiceNow buys an enterprise AI assistant company, and the latest funding rounds in enterprise tech.

(Was this email forwarded to you? Sign up here to get Runtime each week.)

Intelligent design

LAS VEGAS — Enterprise tech spent the last 15 years perfecting infrastructure techniques built around the flexibility of cloud computing and the ubiquity of open-source software, but we're in a new era these days. Apps built for or around generative AI have different hardware and software requirements than their older cousins, and that requires anyone who wants to deploy such an app to learn a few new tricks.

That was the message from a panel discussion moderated by yours truly Monday at the HumanX conference called "Infrastructure 2.0: Building the foundations for AI's next leap." Over the course of 45 minutes the panel — Andrew Feldman of Cerebras Systems, Robert Nishihara of Anyscale, Guillermo Rauch of Vercel, and Sharon Zhou of Lamini AI — offered some advice on what to do and what not to do when drawing up infrastructure plans for AI.

The first step requires companies to understand that generative AI is a whole new world, and the tools and techniques perfected for earlier worlds won't cut it.

• "What's happening with AI is that the compute paradigm is changing very dramatically and the kinds of products that people are building are changing quite dramatically," Rauch said.
• "Historically, when you had big changes at the application layer, what was happening underneath at every different level of infrastructure is new demands were being placed … and when new demands are placed on the hardware, it's very hard for legacy hardware to meet those new demands," Feldman said.
• And even companies that are already using GPUs for some AI workloads face a learning curve: "Doing that at the scale at which these models need it — at such a much, much larger scale than traditional machine learning workloads — creates its own challenges," Zhou said.

Companies also need to rethink their approach to data management and storage, according to the panelists. Modernizing its data stack years before ChatGPT dropped helped Liberty Mutual hit the ground running in the generative AI era, but most companies were not that prepared.

• "One shift that we're seeing is the whole process of going from data that your organization has is switching from being sort of a SQL-centric workload on CPUs to being an AI-centric workload on accelerators. That's a massive shift," Nishihara said.
• Those SQL-centric workloads were built to process structured data, but generative AI unlocks all sorts of possibilities for unstructured data, which organizations have in spades but have never really known what to do with it.
• "I feel like the most magical thing is bringing that all together, the unstructured data with the structured data, and being able to reason through that" by expanding the number of people who can query a database from the data science team to basically the whole company, Zhou said.

No conversation about infrastructure would be complete without discussing scale. Right now companies are throwing themselves into generative AI development and building AI stacks to get up and running that might one day prove to be bottlenecks depending on the choices made in those early days.

• If you're going to scale to a large size, you need to be thinking about data, you need to be thinking about compute, and the model. And big models aren't right for everything," Feldman said, pointing out that smaller models are increasingly capable of getting the job done in lots of situations without incurring the expenses required to run large models.
• Companies adopting generative AI stacks "should be able to get all the benefits of scale, of the cloud, of different accelerators without having to think about the underlying distributed systems challenges, scheduling, [and] failure handling," Nishihara said.
• "The thing I recommend is, go with what is fast but be very knowledgeable about how you can scale, whether it be around costs or whether around capacity, because you might need to do that at some point," Zhou said.

Move it move it

Right now just about every enterprise software company is jockeying for position to become the preferred agentic AI vendor, and ServiceNow decided this week to bulk up its product lineup. On Monday it announced plans to acquire Moveworks, which has built a series of enterprise-oriented AI assistants, for $2.85 billion.

Moveworks' AI Assistant works across "enterprise messaging apps, web browsers, intranets, service portals, and more" to allow companies and employees to search for unstructured data and complete internal tasks. ServiceNow has been trying to expand beyond its core business in IT service management into other areas of business software, and "the majority of Moveworks' current customer deployments already use ServiceNow as an important system of action to access enterprise AI, data, and  workflows, pointing to a seamless integration for the companies," it said in a press release.

Despite nearly a year's worth of hype, most companies are taking their time when it comes to adopting AI agents, as Salesforce acknowledged during its earnings results two weeks ago. "This deal enables ServiceNow to challenge Salesforce more effectively, provided it successfully navigates cultural integration challenges and fully leverages Moveworks’ capabilities," Constellation Research's Prabhu Ram told CIO.

Enterprise funding

Reflection AI launched with $130 million in seed and Series A funding to allow the former members of Google's DeepMind to build "fully autonomous" coding assistants.

Turing raised $111 million in Series E funding for its AI infrastructure tools, which are used by companies like OpenAI to train their own models.

SpecterOps scored $75 million in Series B funding that will allow it to grow its BloodHound Enterprise security product, used to detect and eliminate identity-based security threats.

Norm AI landed $48 million in Series B funding to build AI agents for compliance departments.

Crogl launched with $25 million in Series A funding and released its "knowledge engine," which uses AI to comb through security alerts and elevate the real threats.

The Runtime roundup

CoreWeave signed a $12 billion compute deal with OpenAI, which helps both companies reduce their reliance on Microsoft as a strategic partner.

Oracle missed Wall Street's guidance for its third quarter and issued lower-than-expected guidance for the current quarter, but swore that its cloud infrastructure business is otherwise going great.

Meta is testing a custom-designed AI training chip that could help it reduce its reliance on Nvidia, according to Reuters.

Asana co-founder and CEO Dustin Moskovitz announced that he will step down upon the selection of a new CEO but remain chair of the board.

Thanks for reading — see you Thursday!