Red Hat is probably the most successful open-source enterprise software company in the history of tech. For many years that success has existed in tension with the open-source software community, and this summer that tension rose to a new level.
The trigger was Red Hat's decision in June to stop providing an exact copy of each new release of Red Hat Enterprise Linux to an open portal where anyone could download the software and use it as they pleased. It provided this "downstream" copy of RHEL — an open-source term used to describe new software built using some code from existing open-source projects, or "upstream" code, and some original code — out of a sense of tradition, after it acquired the CentOS project in 2014.
For decades, CentOS was a very popular community-supported downstream version of RHEL that was used by individual developers and small teams. Big corporations like Facebook and Salesforce also used CentOS in production environments or as a low-cost test and development server for production applications that often ran on RHEL.
But in 2020 Red Hat turned CentOS into essentially a beta upstream version of RHEL and warned CentOS users that previously released versions would no longer be updated after 2024. Companies that had made CentOS a significant pillar of their tech stack — with the expectation it would receive security and feature updates for ten years, just like new versions of RHEL — faced a hard deadline to make a decision about their next steps.
"The biggest issue with that end-of-life (decision) was that first off, it was mid-release cycle," said Greg Kurtzer, one of the founders of the original CentOS project, in a recent interview. "So everybody was counting on it, who doesn't want excitement in their job life — they want stability, they want boring — everybody was affected by that. And all of a sudden, they're thinking, 'what am I going to run on this giant chunk of my infrastructure?'"
Red Hat, of course, presented RHEL as the answer to that question. But around the same time two new projects sprung up — Rocky Linux, run by Kurtzer, and AlmaLinux — that used the free copy of RHEL to build their own downstream versions of Linux that they advertised as "bug for bug" compatible with RHEL, which requires expensive licenses and support contracts to use directly from Red Hat. Those projects also found commercial backers in CIQ, which Kurtzer founded and runs as CEO to support Rocky Linux, and CloudLinux, which supports AlmaLinux.
They also found customers: an unnamed "very large enterprise" is using AlmaLinux to manage a fleet of delivery trucks, said Benny Vasquez, chair of the board of the AlmaLinux OS Foundation, in an interview. And Rocky Linux is mostly used by "professional IT teams," Kurtzer said.
However, with the June announcement, Red Hat declared that it was no longer interested in making life easy for the "rebuilders" by releasing a free copy of RHEL. In an interview, Red Hat's Mike McGrath, vice president of core platforms engineering, made it clear that the company was no longer interested in creating commercial opportunities around downstream versions of RHEL.
"I'm not aware of anywhere else in open source where a downstream rebuild exists solely to duplicate the upstream just with a different name and logo," McGrath said. "Let's just say it: A lot of people want to use RHEL, and they don't want to pay for it."
Free as in beer
Open-source software has always been a messy, passionate sector of the tech world where capitalism and socialism struggle to co-exist.
Smeared in 2001 as a "cancer" by former Microsoft CEO Steve Ballmer, who did not have a great track record predicting the future of technology, open-source software would go on to dominate enterprise tech over the next two decades. As it turned out, software produced by committees would prove to be more nimble and resilient to changing mores in technology than software produced in top-down fashion by big corporations, and a generation of internet infrastructure was built around open-source projects.
Founded in 1993, Red Hat was instrumental in advancing Linux as a lower-cost alternative to servers running Unix and Windows, making money by writing software packages that helped companies integrate its version of Linux into their workflows and providing technical support. That commercial success (Red Hat's 1998 IPO set a record at the time) was often in conflict with those who felt software should be free and open to anyone.
Over the years a sort of realpolitik settled in as open-source software became an enormous business. But from time to time, disputes would arise over whether or not commercial companies were living up to their responsibilities under the GPL, the seminal software license that governs the use of the Linux kernel and requires developers who incorporate the Linux kernel into their products to release the code behind those products to their users.
Critics of Red Hat's decision last month argue that the company is trying to walk a fine line by providing the RHEL source code — which everyone involved agrees was derived in part from other open-source projects — only to paying Red Hat customers, rather than the general public.
"For approximately twenty years, Red Hat … has experimented with building a business model for operating system deployment and distribution that looks, feels, and acts like a proprietary one, but nonetheless complies with the GPL and other standard copyleft terms," wrote Bradley Kuhn, policy fellow at the Software Freedom Conservancy, in response to Red Hat's decision last month. Kuhn does not believe Red Hat violated the GPL in this case, but argued that projects like CentOS, Rocky Linux, and AlmaLinux served as a check on Red Hat's willingness to test the boundaries of the license.
"In this situation, however, this completes what appears to be a decade-long plan by Red Hat to maximize the level of difficulty of those in the community who wish to “trust but verify” that RHEL complies with the GPL agreements," Kuhn wrote.
Fork in the road
Undeterred by Red Hat's decision to restrict access to RHEL, Rocky Linux and AlmaLinux are making plans for the future.
AlmaLinux announced plans earlier this month to create a proper fork of RHEL, based on code from the CentOS Stream project with plans to work on its own bug fixes and possible enhancements, Vasquez said.
Following the 2020 decision "our goal immediately was to serve a community that had been using CentOS Linux that was not going to be served by CentOS Stream, to provide an enterprise Linux solution for people who had been using CentOS downstream and were no longer going to be able to," she said. "This shift just supports that. It's continuing to work within the agreements and expectations that Red Hat has put in place, and is giving our community a solution that they can continue to work with."
Future versions of AlmaLinux will be "application binary interface" compatible with RHEL, which is a lower standard than "bug for bug" and will require the AlmaLinux project to take on more work, but Vasquez said the community is excited about moving forward on its own terms.
Rocky Linux, on the other hand, is taking a more confrontational approach. It announced plans to work with Red Hat rival SUSE on its own fork of RHEL, and will continue to work on future versions of Rocky Linux that maintain the "bug for bug" standard.
Kurtzer believes that future versions of the project will be able to live up to the compatibility promise by obtaining RHEL source code through methods that don't require the project to agree to Red Hat's contracts and licenses, which grant customers access to the RHEL software packages under an agreement that they won't distribute those packages themselves.
"We need to figure out, is there another way that we can obtain these sources in binaries without signing that contract?" he said. So far, the project is looking at two ways of doing that.
The first involves downloading the Red Hat Universal Base Image from a container repository like Docker Hub. "This base image is freely redistributable," according to Red Hat, but Kurtzer said he won't be surprised if Red Hat decides to rebuild that image around CentOS Stream in the future, which means the image wouldn't include the final RHEL code because versions of CentOS Stream are built before subsequent versions of RHEL are finalized.
The other plan sounds like a lawsuit waiting to happen. Kurtzer argued that simply by launching a version of RHEL on a cloud provider, anyone could gain access to the source code.
"You didn't actually have to abide or sign or click through (on) any of Red Hat's terms and conditions. And as a result, you now have access to GPL and copyleft licensed source code, or binaries, which means you must be able to get access to the source code," he said.
Red Hat took a dim view of this option. "Red Hat subscription terms apply to customers regardless of the channel through which they purchase," the company said in a statement in response to questions about that approach.
Red and blue
The elephant in the room during this discussion is IBM, which acquired Red Hat for $34 billion in 2018 to provide a growth engine for its aging software business.
Ironically, IBM was one of the open-source community's biggest supporters among the tech establishment during the contentious early days of Linux. But many open-source watchers saw the hidden hand of IBM behind Red Hat's decision to cut off the commercial opportunities of the rebuilders, believing that Red Hat was under pressure to grow at all costs to justify the acquisition.
In multiple conversations with current and former Red Hat employees over the last month, that doesn't seem to be the case. However, there's no question that Red Hat is a business like any other business that has targets to hit during a year in which enterprise tech buyers have grown very selective about their expenditures.
In today's cloud-first world, businesses have lots of options if they want to build Linux applications; cloud providers have their own Linux distributions, and several commercial options compete with Red Hat. But according to McGrath, the rebuilders weren't actually trying to compete with Red Hat.
"Usually when people in the open source community can't agree on something, one of them will fork and run, and they will maintain it. And since forever, these rebuilders have not done that," he said. "And I think the reason is because they don't want the code — they can get the code now — but they want to be able to put Red Hat's promise about what we're going to do with the future of that code in their systems as well."