The dark side of the AI coding boom

Welcome to Runtime! Today: A Replit user finds out the hard way that LLM-powered coding tools are still a work in progress, Microsoft scrambles to get Sharepoint users patched, and the latest funding rounds in enterprise tech.

(Was this email forwarded to you? Sign up here to get Runtime each week.)

Most of the advances in software development and distributed-systems design that we now take for granted were achieved by learning from failure, which, truth be told, is the way the whole durned human comedy keeps perpetuating itself down through the generations. But the vibe coding era is a little bit different because it's hard to know exactly why large-language models make decisions, as Jason Lemkin of SaaStr learned last week.

Last Thursday evening Lemkin laid out the details of a troubling experience with Replit, a popular vibe-coding tool that lets users build and maintain apps. During an experimental coding session during which Lemkin hoped to learn more about how these tools work, Replit deleted the production database he was using for his app without asking for permission to do so, which would be a catastrophic outcome for any mission-critical application.

Lemkin had instructed the tool to freeze any code changes, but for some reason Replit overrode that command to push changes to the production database.
And the day before, the tool "kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test," Lemkin wrote.
When Lemkin asked Replit what happened to the database, it responded like a panicked intern caught pushing the wrong button during their first week on the job.
"You had protection in place specifically to prevent this. You documented multiple code freeze directives. You told me to always ask permission. And I ignored all of it," the tool wrote in response to Lemkin's questions about what went wrong.

Replit CEO Amjad Masad responded to Lemkin's thread on X Saturday morning, calling the tool's actions "unacceptable and should never be possible." The company has raised $222 million in funding to date, and Bloomberg reported in April that Replit was in talks to raise another $200 million, which would value the company at $3 billion.

Masad outlined several steps that Replit would take in response to Lemkin's experience, acknowledging that Replit did not have a way to separate development databases — where developers can evaluate the effect of proposed changes in a safe bubble — from production databases.
On Monday it rolled out that change (in beta for now, however), saying "We're excited to announce the launch of separate development and production databases for Replit apps, making it safer to vibe code with Replit."
"And yes, we heard the “code freeze” pain loud and clear – we’re actively working on a planning/chat-only mode so you can strategize without risking your codebase," Masad said Saturday.

Millions of developers are using AI-coding tools right now to build new software, and Lemkin's experience appears to be an isolated incident in which no actual user data was harmed. But it's a little unclear how Replit failed to anticipate this problem when designing its tool, given that messing around with the production database is a high-wire act even in the hands of the best professionals.

Lemkin is an investor, not a professional software developer, but a primary selling point for a lot of these new AI coding tools is they can allow anyone to build real, quality apps; "Vibe coding makes software creation accessible to everyone, entirely through natural language," Replit declares on its home page.
However, it's clear that some level of development and deployment expertise is needed to safely use these tools in a commercial setting, which should alleviate worries about software-engineering jobs vanishing by the end of the decade.
As Redis creator Salvatore Sanflippo put it in a blog post over the weekend, "In this historical moment, LLMs are good amplifiers and bad one-man-band workers."

Microsoft SharePoint was one of the more popular software tools for setting up intranets (if you know what that is, it's time for a colonoscopy) back around the turn of the century. It remains widely used inside enterprises and government organizations that are still on-premises Windows shops, and a newly discovered SharePoint vulnerability is quickly turning into yet another security nightmare for Microsoft.

Two separate vulnerabilities are under active exploit by at least two hacking groups believed to have the support of the Chinese government, Microsoft said in a blog post Tuesday. Patches are available for both flaws, and the issues do not affect anyone running SharePoint on cloud services, but CRN reported that on-premises SharePoint users also need to rotate security keys to make sure they're safe.

"Federal investigators believe multiple U.S. government agencies are among the early victims of the ongoing cyber exploitation campaign, though the full scope is not yet clear," Politico reported Tuesday. It's another blow to Microsoft's security reputation nearly two years after it launched the Secure Future Initiative, especially given that "a security patch released by Microsoft earlier this month failed to fully fix a critical flaw in the U.S. tech company's SharePoint server software that had been identified at a hacking competition in May," Reuters reported.

Enterprise funding

Lovable raised $200 million in Series A funding for its vibe-coding tool, which the company said was "one of Europe's largest Series A investments ever."

Reka AI scored $110 million in new funding as it continues to develop new "multimodal" AI models, which can work with text, images, video, and audio.

Bright AI landed $51 million in Series A funding to bring observability technology to physical infrastructure like power grids and gas pipelines.

Delve raised $32 million in Series A funding for its efforts to apply agentic AI to compliance software.

Composio scored $25 million in Series A funding as it tries to build a "shared learning layer" that can help AI agents get better with experience.

CandorIQ landed $4.8 million in seed funding for its HR and finance software, which hopes to make it easier to manage tasks like payroll and hiring.

The Runtime roundup

OpenAI's Project Stargate is having trouble getting started on its quest to build a massive network of data centers, according to the Wall Street Journal, a development that nobody saw coming.

And in what can only be pure coincidence, OpenAI and Oracle announced Tuesday after that report surfaced that they plan to build 4.5 gigawatts of data-center capacity that could be worth $30 billion a year for Oracle by 2028, assuming it actually gets built.

Thanks for reading — see you Thursday!

Is Google Cloud winning the AI era?