Why developer productivity is a team sport

Welcome to Runtime! Today: why evaluating software developers like salespeople is misguided, IBM promises legal protection for generative AI customers, and the latest moves in enterprise tech.

(Was this email forwarded to you? Sign up here to get Runtime each week.)

Metrics all the way down

If we assume that every company has become a software company, that means software developers have become some of the most valuable (and expensive) employees on the payroll. How do companies know they're getting the most out of their investment in those employees?

It's an old debate in information technology that took on new life last month after McKinsey (yep, that McKinsey) published an article claiming it had developed a new metrics-based approach to measuring individual developer productivity. It was met with an immediate backlash from several corners of the software world.

• It turns out that the vast majority of engineering managers believe they get the best out of their employees when they treat them less like coding automatons and more like, well, people.
• "The activity of building software is still a highly creative and human-centered collaborative exercise, which doesn't lend itself to easy measurement," said Brendan Humphreys, head of engineering at Canva.
• It's no coincidence that McKinsey's article appeared at a time when engineering managers are being asked to rank their developers should the business determine that it needs to cut costs across the board.
• "You should be measuring productivity at a team level to achieve the outcomes that you want to achieve," said Milin Desai, CEO of Sentry. "Because otherwise you end up with busy work without impact."

Most engineering organizations measure some individual metrics; the number of lines of code written by a developer, the number of times their code is committed to the live production environment, or how long it takes them to resolve issues that are assigned to them, said Arvind Jain, CEO of Glean.

• However, "we've never felt that we could just look at these metrics and make a judgment on anyone," Jain said.
• Dylan Etkin co-founded Sleuth, a developer productivity measurement startup that makes a team-productivity dashboard based around the principles of the DORA project.
• "We recognized very early on that developers are very concerned about being measured on things that don't really matter," Etkin said.

DORA focuses on four key team-oriented metrics including deployment frequency — overwhelmingly cited by those interviewed for this article as the most important productivity metric they worry about — and time to resolve issues.

• "It really seeks to understand what makes up high-performing teams when it comes to using software for delivering business value," said Nathen Harvey, a developer advocate for both DORA and Google Cloud, which acquired DORA in 2019.
• The McKinsey article acknowledged the usefulness of DORA and another team-oriented developer productivity framework known as SPACE, but argued "while deployment frequency is a perfectly good metric to assess systems or teams, it depends on all team members doing their respective tasks and is, therefore, not a useful way to track individual performance."
• But that's why companies hire engineering managers, said Camille Fournier, managing director and global head of engineering and architecture for J.P. Morgan Chase and Co.'s Corporate Investment Bank division, and author of the popular guide for a generation of tech professionals, The Manager's Path.
• "My expectation is a line manager of individual contributors shouldn't generally need to look at a lot of metrics to figure out whether people are productive or not," she said.

The potential to improve developer productivity has actually been one of the more compelling aspects of the generative AI boom.

• "The way I look at it is, does it boost my top developers to be more creative?" Sentry's Desai said.
• Fournier believes coding assistants could also play a very interesting role helping developers program in languages in which they lack experience, greatly reducing the time it takes them to get up and running on something new.
• But if businesses decide to bring AI development tools into their organizations in search of productivity while also adopting McKinsey's "coders should code" metrics-driven productivity advice, they'll have just automated the process of gaming those metrics, Desai said.

McKinsey's system seems to be designed for companies that don't trust their technology leaders, which is, of course, a sign of a much deeper organizational problem. Should it appear at your workplace, it's probably time to test the market.

Read the full story on Runtime.

Lawyer up

There's something quite telling about the fact that enterprise tech vendors are lining up to offer legal protection to the early adopters of their newest products.

IBM joined Microsoft Thursday by promising to indemnify customers using its generative AI products against copyright or other lawsuits. IBM said this was part of "its standard intellectual property protection," but it's hard to imagine that most IBM customers are all that worried about legal issues when using its other software and hardware products.

AWS CEO Adam Selipsky declined to directly answer a question regarding whether AWS would offer similar protection two weeks ago in Seattle, and it's not clear where Google Cloud stands. If generative AI is really going to transform the enterprise to the degree that all these zillion dollar companies insist it will, the legal issues behind this technology will need to be resolved sooner rather than later.

Enterprise moves

Prat Moghe is the new CEO of TripleBlind, an AI privacy startup, after a year and a half as executive vice president at Cloudera.

John Rayfield is now corporate vice president of AI silicon at AMD, following almost three years at archrival Intel in a similar role.

Mike Campfield is the new chief revenue officer at Uptycs, a cloud application security startup.

The Runtime roundup

AWS announced that its Bedrock AI marketplace service is now generally available, and unveiled plans to let enterprise customers train its CodeWhisperer coding assistant on their own internal code base.

Mistral AI released a new open-source LLM that it favorably compared to Meta's Llama 2 model, which is kinda-sorta open source.

Did you know Snap had an enterprise division? Anyway, it's gone.

Akamai announced seven new cloud infrastructure computing regions, as it continues to build on its acquisition of Linode last year.

Microsoft is looking into lower-carbon concrete mixes that could help it reduce carbon emissions when constructing new data centers.

Workday introduced new generative AI features, but lowered its revenue guidance for the year and Wall Street freaked out.

Thanks for reading — see you Saturday!