I've only been working on the Linux kernel for a few months. Before that, I worked with proprietary source control at work and common tools like GitHub at home. The concept of the mailing list seemed obtuse to me. If I noticed a problem with some program, I'd be willing to open an issue on GitHub but not to send an email to a mailing list. Who still uses those, anyway?

Starting out with the kernel meant I had to figure this email thing out. git format-patch and git send-email take most of the pain out of formatting and submitting a patch, which is nice. The patch files generated by format-patch open nicely in Emacs by default, showing all whitespace and letting you pick up any irregularities. send-email means you can send it to yourself or a friend first, finding anything that looks stupid before being exposed to the public.

And then what? You've sent an email. It gets sent to hundreds or thousands of people. Nowhere near that many will read it. Some might miss it due to their mail server going down, or the list flagging your post as spam, or requiring moderation. Some recipients will be bots that archive mail on the list, or publish information about the patch. If you haven't formatted it correctly, someone will let you know quickly. If your patch is important or controversial, you'll have all sorts of responses. If your patch is small or niche, you might not ever hear anything back.

I remember when I sent my first patch. I was talking to a former colleague who didn't understand the patch/mailing list workflow at all. I sent him a link to my patch on a mail archive. I explained it like a pull request - here's my code, you can find the responses. What's missing from a GitHub-esque pull request? We don't know what tests it passed. We don't know if it's been merged yet, or if the maintainer has looked at it. It takes a bit of digging around to find out who's commented on it. If it's part of a series, that's awkward to find out as well. What about revisions of a series? That's another pain point.

Luckily, these problems do have solutions. Patchwork, written by fellow OzLabs member Jeremy Kerr, changes the way we work with patches. Project maintainers rely on Pathwork instances, such as https://patchwork.ozlabs.org, for their day-to-day workflow: tagging reviewers, marking the status of patches, keeping track of tests, acks, reviews and comments in one place. Missing from this picture is support for series and revisions, which is a feature that's being developed by the freedesktop project. You can check out their changes in action here.

So, Patchwork helps patches and email catch up to what GitHub has in terms of ease of information. We're still missing testing and other hooks. What about review? What can we do with email, compared to GitHub and the like?

In my opinion, the biggest feature of email is the ease of review. Just reply inline and you're done. There's inline commenting on GitHub and GitLab, which works well but is a bit tacky, people commenting on the same thing overlap and conflict, each comment generates a notification (which can be an email until you turn that off). Plus, since it's email, it's really easy to bring in additional people to the conversation as necessary. If there's a super lengthy technical discussion in the kernel, it might just take Linus to resolve.

There are alternatives to just replying to email, too, such as Gerrit. Gerrit's pretty popular, and has a huge amount of features. I understand why people use it, though I'm not much of a fan. Reason being, it doesn't add to the email workflow, it replaces it. Plaintext email is supported on pretty much any device, with a bunch of different programs. From the goals of Patchwork: "patchwork should supplement mailing lists, not replace them".

Linus Torvalds famously explained why he prefers email over GitHub pull requests here, using this pull request from Ben Herrenschmidt as an example of why git's own pull request format is superior to that of GitHub. Damien Lespiau, who is working on the freedesktop Patchwork fork, outlines on his blog all the issues he has with mailing list workflows and why he thinks mailing lists are a relic of the past. His work on Patchwork has gone a long way to help fix those problems, however I don't think mailing lists are outdated and superceded, I think they are timeless. They are a technology-agnostic, simple and free system that will still be around if GitHub dies or alienates its community.

That said, there's still the case of the missing features. What about automated testing? What about developer feedback? What about making a maintainer's life easier? We've been working on improving these issues, and I'll outline how we're approaching them in a future post.