MaddyIt being Hacktoberfest, we thought we’d share some bits from behind the scenes of the CouchDB 3.4.0 release. After all, this is no ordinary release: it has just the elements of drama and suspense you’d expect from any decent Halloween episode!
This article is adapted from a talk given at the Neighbourhoodie CouchDB Meetup. All talks are available on demand.
Before 3.4
If you haven’t upgraded yet, you’re probably running CouchDB 3.3.3. These numbers indicate the major release line, feature releases, and bugfixes respectively:
A new release line is warranted when upgrading is considered “difficult”. Think significant dependency changes, new document write and read behaviour, or big updates to users. To give you an idea of how often these come around, release line 3 came out toward the end of February 2020 (yes... that month... and no, this is not where the suspense starts so you can relax your shoulders). Release line 2 was announced in September 2016.
Isn’t that a bit too infrequent? You wouldn’t be the first to think that because it’s the result of a significant way in which CouchDB is different. To sum that difference up in a word: Stability.
Changes are designed to ensure existing applications just keep working, which often doesn’t match the picture most folks have of their database technology. Remember this, it’ll be important later on.
Feature releases come out with greater frequency, currently around once a year. Bug fixes come out, well, as often as they need to!
To give you a very brief idea of what goes into adding a flagship feature, I’ll talk briefly about Nouveau. You can read more about Nouveau on our blog to discover its advantages over its predecessor, Clouseau. We also have a guide to getting started with Nouveau for CouchDB. What our blog won’t tell you, is what it took to add Java capabilities that made building Nouveau happen. As it turns out, it’s not just adding the code. It’s adding Java code and tooling to:
- the build process
- the local development process (Linux, macOS, FreeBSD, Windows)
- the CI setup (Debian, RockyLinux for RHEL support, Ubuntu, FreeBSD, macOS, Windows)
- source release procedure
- binary release procedure (.deb, .rpm, .app, Docker, arm64, x84_64)
No, you are not the only one sweating looking at that 🥵 House of horrors door squeak, anyone? If you want to learn more about what went into Nouveau, and CouchDB’s new JavaScript engine, QuickJS, check out the talk we gave about it.
But, was there a 3.4.0? 👻
If you follow CouchDB news then you’re probably pulling the covers a little closer to your eyes, because you already know... there was no 3.4.0. Dum dum dum!
Once a maintainer utters (types) the words “Hey, I think this is a release” they have essentially initiated the process of submitting a release candidate to the community for consideration.
The Apache release protocol, which you can become acquainted with via the Apache release policy, takes 72 active hours to complete with several days in between. It covers everything from QA to legal stuff and by doing so, puts a safe and reliable version of CouchDB in your hands. By the time it’s completed, binaries for your operating system are up and ready to be downloaded, and source code is safe to fork.
So what happened to 3.4.0?
We realised, less than 100 minutes out from publishing and announcing the release on Monday, 23rd September, that something wasn’t right.
Remember when I said if CouchDB is one thing, it’s reliable? A feature release is considered safe to upgrade when new features are not on by default, so that downgrading or rolling back is relatively trivial and doesn’t break things.
And that’s where the problem lay: password hashing with PBKDF2-SHA256 (which replaced -SHA1) was on by default and ran automatically. If rolled back, passwords hashed with -SHA256 would now be unreadable. That’s capital-B Bad.
3.4.1
The undesirable password hashing behaviour meant that it was not considered safe to upgrade to 3.4.0. This was decided on Monday, a bit before midday.
But the protocol had been completed, and there was no turning back. It seemed there was one thing to do: let 3.4.0 pass unannounced into Monday night, a ghost ship in source files only, while maintainers worked on a fix.
By Thursday, September 27th, CouchDB 3.4.1 had come to the rescue, and was announced. Spoiler, we wrote about that too!.
Lessons learned
We could have also called this section “we did it the hard way (and then wrote about it) so you don’t have to.”
change : risk
This shouldn’t be groundbreaking. But if you, like me, tend to err on the optimistic side when it comes to anticipating what could go wrong, you may still need to learn the spectral lesson…
plan accordingly
Schedule the time for the changes and their implications. And then for the fun of it, add a buffer. 72 hours is a good start!
solve issues for the whole supply chain as early as possible
From the code in the source repository to making binaries for multiple platforms and architectures and everything in between. In the words of Jan Lehnardt:
“I wish we fixed that early on so we know the whole extent of the issues instead of discovering each new part of the problem at each step of the process.”
And last but not least:
Volunteer-driven OS projects don’t work on a schedule, but making frequent releases is still important. Stay nimble so you can always release.
Thank you for reading, and happy spooky season!
This article has been adapted from a CouchDB Meetup Talk presented by Jan Lehnardt on October 9, 2024. Watch the full video, and join the Neighbourhoodie mailing list to find out when new Meetups — and releases — are announced.
