Cloudflare's plan for the AI era

Welcome to Runtime! Today: how Cloudflare thinks it can outmaneuver the big clouds on the edge, Progress Software reveals another major vulnerability in a file-transfer product, and the quote of the week.

(Was this email forwarded to you? Sign up here to get Runtime each week.)

Livin' on the edge

In between baffling attempts to defend some of the worst people on the planet, Cloudflare has been building a very interesting approach to cloud computing that looks a little different than the traditional strategy followed by the Big Three. It updated that strategy for the AI boom this week during the observation of its 13th birthday.

"Our job isn't to be the final destination for your data, but to help it move and flow," Cloudflare co-founders Matthew Prince and Michelle Zatlyn wrote in their annual birthday letter to shareholders and customers. Rather than the hub-and-spoke centralized cloud computing model built around massive data-center complexes in dedicated regions, Cloudflare built and maintains computing and networking infrastructure that connects relatively small data centers with modest amounts of computing power across a larger number of locations around the world.

Cloudflare is making a bet that inference will be the most strategic enterprise AI workload in the long run, a sentiment echoed by several AWS execs in Seattle earlier this month.

• Developing AI models requires training them on massive data sets, which requires incredible amounts of computing power and is mostly responsible for the global run on GPUs this year.
• Once those models have been trained, putting them into action inside applications where they can respond to prompts is known as inference.
• Inference is sensitive to latency — or the amount of time it takes your prompt to run through the LLM and back with an answer — and less resource-intensive, which means it could be ideal for edge networks like Cloudflare's.
• "Since inference is required upon every single invocation (rather than just once), we expect that inference will become the dominant AI-related workload," Cloudflare wrote in a blog post.

The company rolled out several new services this week in hopes of capitalizing on this trend.

• Workers AI builds on the serverless Workers service introduced in 2017 to let customers run several well-known LLMs on managed Cloudflare servers with Nvidia's GPUs (it didn't say which ones.)
• Edge and serverless computing have been hyped for years, but AI applications look like they're going to make the concept mainstream thanks to the cost (in both dollars and latency) of running those apps on traditional cloud servers.
• Cloudflare also announced AI Gateway, an observability service that could make it easier to understand how AI applications are performing in the real world.

It also introduced a new vector database, which is fast becoming an important part of any AI application.

• Vector databases make it easier for LLMs to answer prompts without having to scour their entire data set each time a common prompt is entered.
• Vectorize is Cloudflare's take on the emerging technology, designed to
  "power semantic search, classification, recommendation and anomaly detection use cases directly with Workers, improve the accuracy and context of answers from LLMs, and/or bring-your-own embeddings from popular platforms, including OpenAI and Cohere," the company said.
• Also on the database front, it introduced Hyperdrive, a new service that Cloudflare claims will speed up queries to traditional databases when running across its network.

Two years ago Prince told me that Cloudflare was "aiming to be the fourth major public cloud." While it hasn't made huge inroads against the major cloud providers when it comes to traditional applications, all bets are off as enterprise tech tries to figure out its AI strategy.

Same story, better outcome

As the fallout from the MOVEit vulnerability continues to grow, Progress Software disclosed Thursday that it had identified several new vulnerabilities in a different file-transfer product made by the company.

Eight vulnerabilities — two of them critical — were recently identified in the WS_FTP product by third-party security researchers and Progress Software itself, according to The Record. Thankfully, in this case, Progress said it did not believe that the vulnerability has been exploited, and patches are available for customers to prevent that from happening.

These types of file-transfer products, which allow sensitive corporate information and personal data to move around corporate networks constantly, are clearly a ripe target for the criminal hacking community. It's much harder for enterprise security teams to defend against attacks exploiting flaws in their software stack than ones launched when employees click on a funny-looking link in that weird email.

Quote of the week

"I think that the worry and sort of the fear that I think from a practitioner level that you see is, maybe I don't mind if myself or my team are looking at me individually. But I know that the minute those (metrics) end up in a dashboard, or in a system, they're eventually going to get rolled up and now an executive is going to make a choice.

And they're just going to draw a line without any context for understanding and there are going to be people above and below that line. And that has some potentially really dire consequences for everyone involved, including that executive." — Google's Nathen Harvey, on why individual software developer productivity metrics designed by shady consulting firms can do more harm than good.

The Runtime roundup

In the weeks before Anthropic signed a new investment and cloud computing deal with AWS, Google Cloud was having trouble maintaining an Nvidia H100 cluster used by the LLM startup, according to Big Technology.

Meta slipped out a new version of its Llama 2 LLM this week that it claims can outperform similar LLMs from OpenAI and Anthropic.

Thanks for reading — see you Tuesday!