Sabotage in the software supply chain

Welcome to Runtime! Today: after dodging a potential catastrophe, the debate over how to  strengthen the open-source software supply chain enters a new chapter, Microsoft 365 is no longer a Teams sport, and the latest funding rounds in enterprise tech.

(Was this email forwarded to you? Sign up here to get Runtime each week.)

500ms away from disaster

An audacious attempt to compromise the security of the servers that run enterprise tech was thwarted late last week thanks to a sequence of events that will be hard to duplicate at scale. The incident validated some of the best practices in open-source software and revealed some of its biggest weaknesses, and needs to be a wake-up call for governments, vendors, and tech buyers.

Thanks to some dedicated sleuthing by a Microsoft engineer, Linux maintainers were able to stop a two-year effort by someone posing as an eager-to-help developer to insert a backdoor into production Linux systems. The vehicle was the open-source XZ Utils data-compression tool, and a compromised version of that tool made its way into new, experimental builds of Linux but was detected before it could make its way downstream into commercial distributions.

A quick recap:

• On Friday morning, Microsoft's Andres Freund (who should never be allowed to buy a beer at any tech conference ever again) posted his discovery of a backdoor in XZ Utils that he noticed after snooping through the code somewhat miffed that it was executing 500ms slower than expected.
• He notified maintainers of the Debian Linux operating system that some experimental, newer builds had incorporated the compromised tool, and after double checking, Red Hat also issued an "urgent security alert" for beta versions of Fedora that had been compromised.
• The exploit could have allowed whoever controlled it to execute code remotely on compromised Linux servers, which make up the vast majority of servers used in enterprise tech.
• As the weekend unfolded, security researchers discovered that the backdoor was inserted over a two-year process by someone who volunteered to help write patches for the project.

"This might be the best executed supply chain attack we've seen described in the open, and it's a nightmare scenario: malicious, competent, authorized upstream in a widely used library," said open-source maintainer Filippo Valsorda, as noted by Ars Technica. 

• The attacker took advantage of a widely discussed problem in open-source software: there are countless software libraries and tools at the heart of the world's infrastructure software that are maintained by unpaid, overworked individuals.
• Starting in late 2021, multiple people (or someone pretending to be multiple people) pressured the lead maintainer of the project to accept legitimate patches written by the attacker to fix legitimate issues, convincing the lead maintainer that the attacker was just another developer trying to help.
• After about a year, the attacker was formally added to the list of maintainers for XZ Utils and spent another year helping write new releases until adding the backdoor to the project two months ago.

A world so reliant on enterprise infrastructure software can no longer expect hero hackers or dedicated-but-weary maintainers to save the day by detecting and fixing all the problems in open-source code. But there are no quick, easy solutions here.

• One of the biggest problems is the sheer number of open-source libraries and tools that are layered by the dozen to assemble modern software; simply putting together a list of all those projects is an immense undertaking, let alone inspecting them.
• The Biden administration's push for government suppliers to maintain a software bill of materials helps, but involving the government too deeply in open-source software maintenance feels like a different kind of disaster waiting to unfold.
• Some want maintainers to receive compensation, but even Tidelift's Luis Villa, who has been at the forefront of those efforts, warned that "paying maintainers is not a magic bullet."
• "Planning for the scenario in which the worst case has happened and understanding the outcomes and recovery process is everyone’s homework now, and making sure you are prepared with tabletop exercises around zero days," wrote Docker's Justin Cormack.

You can go your own way

After pressure from European regulators, Microsoft announced Monday that it would no longer automatically include Microsoft Teams in its Microsoft 365 bundle, which contains Windows and Office, for all customers globally. Current customers outside Europe will be able to continue paying for Teams as part of that bundle if they like, but new enterprise customers will need to purchase two separate products.

Teams was once a central part of Microsoft's attempt to prevent companies like Slack and Zoom from making inroads into its dominance over the market for office-productivity software. But competitors argued that by folding teams into the popular Microsoft Office bundle, the company was making it harder for Microsoft shops to pick other collaboration or conferencing tools, and last year European regulators agreed with that argument.

"Globally consistent licensing helps ensure clarity for customers and streamline decision making and negotiations," Microsoft said in its announcement. It also sets up an interesting price comparison; Morgan Stanley analysts (as noted by Techcrunch) pointed out that standalone Teams is now far cheaper than Slack and new Microsoft customers will actually pay more to use both Microsoft 365 and Teams.

Enterprise funding

Hailo raised $120 million in an extension of an earlier Series C round to further its work on energy-efficient AI processors for edge computing.

Zafran scored $30 million in seed and Series A funding for its cybersecurity risk management tools that detect vulnerabilities and determine how exploitable they might be.

Skyflow landed $30 million in an extension of an earlier Series B round as it tackles the problem of helping companies secure customer data as they introduce LLMs.

Read AI raised $21 million in Series A funding to increase the capabilities of its meeting-summary technology.

The Runtime roundup

Rubrik filed for an IPO, revealing that it lost an eye-popping $354 million on revenue of $628 million last year.

NIST attributed the growing backlog of updated software vulnerability data to "a change in interagency support" in a statement over the weekend.

AWS will give the latest batch of Y Combinator startups $500,000 in credits for its Amazon Bedrock AI model service, according to The Register.

Microsoft 365 users who shelled out for its Copilot assistant can now tap into GPT-4 Turbo and upload more files for analysis.

Slack's Noah Desai Weiss has left the company, according to Fortune, leaving Salesforce execs fully in charge of Slack's product strategy.

Thanks for reading — see you Thursday!