Software development has a package-management crisis

Welcome to Runtime! Today: Software package managers are under attack once again in a campaign to steal credentials, Instructure appears to have paid off the hackers behind the ransomware attack on the Canvas educational software platform, and the latest funding rounds in enterprise tech.

Please forward this email to a friend or colleague! If it was forwarded to you, sign up here to get Runtime each week, and if you value independent enterprise tech journalism, click the button below and become a Runtime supporter today.

Package problems persist

Weeks after SAP developers were hit with a software supply chain attack targeting packages stored on npm's platform, the Mini Shai-Hulud attackers struck again Monday evening and managed to compromise open-source packages catering to several different types of developers. It's just another sign for developers that the days of relying on centralized software package managers — as convenient as they are — might be numbered.

The new attacks surfaced Monday and targeted both npm and PyPI, a popular source of packages for Python developers, according to Google's Wiz. "Wiz assesses with high confidence that this is the work of TeamPCP, the same operators behind the SAP, Checkmarx, Bitwarden, Lightning, Intercom, and Trivy compromises," the company said in a blog post.

• Tanstack, a popular library among developers working with React, was hit first and hardest after the attackers published 84 malicious components across 42 Tanstack packages on npm, the open-source project said in a postmortem published Monday.
• But packages designed for developers working with UiPath's RPA tools were also compromised, and Microsoft said it was investigating what appeared to be very similar problems impacting the PyPI repository and packages related to MistralAI's services.
• Compared to the SAP attack from last month, "this wave does not just look like someone manually publishing bad versions. The malware is built to run inside build systems, steal npm and GitHub access, and abuse trusted publishing paths to push new compromised packages," Aikido Security said in a blog post.

The good news is that platforms like npm and security companies are getting good at detecting malicious packages and responding quickly before they can be downloaded by a huge number of developers; Tanstack said StepSecurity detected the bad packages within 20 minutes. The bad news is the attackers used novel exploits to compromise other packages maintained by its targets and even built a self-destruct button into the payload.

• The malicious payload was designed to steal login credentials to popular CI/CD tools like GitHub Actions, GitLab, and CircleCI, as well as the Big Three cloud providers, according to Wiz.
• "Mini Shai-Hulud is a true worm: after stealing credentials from one CI/CD pipeline, it enumerates every package that maintainer controls and publishes infected versions of each," StepSecurity said in a blog post.
• The attackers also built a "dead-man's switch" in compromised packages, which was set to ping an external server every minute and if the system owner revoked the GitHub authentication token stolen by the package, it would run a command that deletes all the files in that user's home directory, according to Snyk.

All the security company blog posts linked here provided detailed instructions on how to search potentially compromised systems for signs of infection and take action to remove the malicious packages. But while cleaning up after the fact is obviously important, solving the root causes of these compromises will require companies to change the way they use software package managers.

• Tools like npm and PyPI allow developers to build and update applications without having to rewrite all the essential plumbing code every time, but automatically downloading those packages from an easily compromised source in the name of speed or convenience is starting to look real dangerous.
• "The important lesson is not just that more packages were compromised. It is that the malware is built around the way modern release systems work," Aikido said.
• Gergely Orosz offered several suggestions that developers and software organizations could take in order to get ahead of future attacks, but acknowledged that they all involve compromises between speed, cost, and complexity that will force difficult decisions.
• "Package management has conflicting goals: security wants lockdown, developer experience wants flexibility, maintainer sustainability wants minimal friction. These tensions aren't solvable, but the threat landscape calls for us to rebalance and make informed trade-offs," Wiz said in a guide to working with packages published last week.

What did we learn

As ransomware attacks proliferated over the last several years, security professionals and companies generally advised one initial course of action when responding: Don't pay the ransom. But different companies make different decisions for different reasons, and Instructure appears to have struck some sort of deal with the group that forced its popular Canvas educational software tool to its knees last week.

"With that responsibility [to customers] in mind, Instructure reached an agreement with the unauthorized actor involved in this incident," the company said in a statement Monday. It did not specify the nature of that agreement, but said that "the data was returned to us, we received digital confirmation of data destruction (shred logs), [and] we have been informed that no Instructure customers will be extorted as a result of this incident, publicly or otherwise."

It's almost impossible to imagine that ShinyHunters, the group believed to be behind the attack, would simply hand over that data without some sort of financial compensation, and TechCrunch reported Tuesday that a ShinyHunters representative seemed satisfied with the terms of the deal. And although Instructure appeared to acknowledge the risks of cutting deals with criminals in its initial statement, writing "while there is never complete certainty when dealing with cybercriminals, we believe it was important to take every step within our control to give customers additional peace of mind, to the extent possible," according to The New York Times, that sentence no longer appears on its web site.

Enterprise funding

Amp launched with $1.3 billion in new funding to build an AI computing grid around unused capacity for startups and research institutions.

Cowboy Space raised $275 million in Series B funding for its orbital data center strategy, which will be based around a collection of satellites launched into space by the company's own rocket technology.

Exaforce scored $125 million in Series B funding for its Ai-powered security-operations center strategy.

Tessera Labs raised $60 million in Series A funding for its AI-powered (what else) enterprise migration software, which puts it in direct competition with hyperscalers like AWS.

XBOW added $35 million in new funding to a previous Series C round, as it builds out AI-powered (seriously) penetration testing software.

White Circle landed $11 million in seed funding for its AI model testing technology, which helps companies building custom models understand where they need better data protection guardrails or prompt-injection defenses.

The Runtime roundup

Google is considering a partnership with SpaceX to launch orbital data centers, according to The Wall Street Journal, although experts aren't sure if space-based data centers will ever make economic sense.

Alon Haimovich, the former general manager of Microsoft Israel, has left the company after an internal investigation into the Israeli military's use of its Azure cloud services, according to The Guardian.

Thanks for reading — see you Thursday!