There’s an undeniable shift happening in the open source community right now. A quick news search of open source -AI returns headlines like, “licensing firestorm”, “open source vulnerability”, and “battle for the future”. But the outlook isn’t all doom and gloom — especially when you talk to the people hacking away every day, for decades, to further the cause of free and open source software, and making the experience of using open source software more accessible for everyone. In fact, some of them might even say that the future looks bright!

Recently, I had a chance to chat with just one of those people — introducing Dirk Müller, who works in SUSE’s Linux group as Distinguished Engineer, and is a major contributor to openSUSE Tumbleweed and other openSUSE or open source projects. There’s a lot of exciting things going on in the openSUSE community — they’ve augmented their mirror downloads infrastructure with Fastly, making downloads up to 20% faster in areas without reliable mirrors, and just a few weeks ago openSUSE announced they would fork Red Hat Enterprise Linux (RHEL) after Red Hat put access to the binaries for it behind a paywall.

I wanted to learn more about these two developments in the openSUSE community, and learn more about Dirk too, so that’s enough preamble. Here’s our chat!

Hannah: Thanks for chatting with me, Dirk! How did you get involved with open source and openSUSE?

Dirk: I don't quite remember all of it, but I am pretty sure I got involved before the term "open source" was created in February 1998. Some time in the early 1990s I started using DKBTrace (which got superseded by POV-Ray on my Amiga computer. The sources of POV-Ray were available and I started reading and modifying it for myself. I eagerly ported my changes over to newer versions of POV-Ray when they were released and followed the main developers closely. Back then it wasn't a thought of mine to start contributing any of my changes back.

This changed with GoldED, which was a messaging application for FidoNet. This was a Shareware application for MS-DOS that I enjoyed using, but it had a few bugs that annoyed me. As the source code wasn't available, I started modifying the actual executable by applying patches on the machine code level and eventually started mailing Odinn, the main author, my modifications. After a while, he started responding and asked me whether I wouldn't feel more comfortable with modifying the source code directly. So eventually he mailed me an archive of the complete sources and I contributed a few patches. He also mailed me a copy of the GNU General Public License (GPL) v2.0 and asked me to read it and think about helping with a Linux port of GoldED in order to release it in source form under GPL v2. Reading that license I got interested. I was unaware of Free Software and Linux back then; and it took me a while to find an early copy of Linux on a set of physical CDs in a local library (coincidentally those CDs, that I still own, were made by SUSE). I didn't have access to the Internet nor was there good documentation as easily searchable as it is today, so it took me forever to make progress on that effort. Also, Odinn and I made the mistake of developing it behind closed doors and only releasing it as open source when it was "ready", which took far too long, and only happened when we lost interest in the entire project. However, thanks to Odinn’s brave decision to open source it, others took over and the software is still in active use today (in a fork called GoldED+). Seeing that the project survived even though all of the original authors stopped, this "infected" and intrigued me with the principles of open source.

When I started studying at university, I got in contact with a few folks that were starting to work on a free desktop environment called KDE and I became one of the very early contributors to it, for many years to follow. As a community contributor, I also sometimes helped SUSE to solve issues in KDE, as well as reported issues in the SUSE distribution, so I was eventually invited to join an "early beta tester" community of upcoming SUSE releases. Around a decade later, I joined SUSE shortly before the openSUSE project was announced. So in a way I have beencontributing to openSUSE from the very beginning.

Hannah: Incredible, Linux on a CD! Okay, so let’s address it — SUSE announced they would fork RHEL. How has this affected your work? Where do you think Enterprise Linux goes from here?

It does not affect the openSUSE project in any way, also specifically not in regards to the download infrastructure or CDN rollout. There is no change in the openSUSE community mission or near term plans. We recently published a FAQ.

I would hope that Enterprise Linux continues to open up and follow the community development models, as well as buyers updating their buying guidelines to prefer options that give you the choice rather than trying to lock you into a particular commercial offering. Customers look for reliable solutions; open source is the trustable, innovative and most sustainable approach to that need.

Several years ago SUSE's Linux Enterprise Server (SLES), the flagship enterprise Linux product from SUSE, was donated to the openSUSE community. If you use openSUSE Leap today, you are running the exact identical binaries to those available to SLES customers and the sources and all updates are available publically available via the same route. We've observed an uptake of usage of both openSUSE, as well as SLES as part of that.

Hannah: Should people be worried about what's going on?

Whenever something surprising or worrisome happens, it is a good idea to look at what your values are and what new choices to make going forward. SUSE has added with their announcement a choice to the available options on the table. Everyone has the freedom to choose, and if in doubt, I think people should choose what gives them the most freedom. Publicly available, open source gives you a freedom that restricted or proprietary software cannot provide; and it comes with the backing of decades of experience behind it that has shaped "enterprise linux" since late October, 2000.

Hannah: Dirk, you’re the voice of reason! We almost always have a choice, and change is hard, but it’s always worthwhile to choose freedom. Thanks for putting my anxiety to rest.

Now let’s talk about how you’re making the experience of using open source better for everyone. You previously built your own CDN to manage your download infrastructure. Can you explain how you did that? What were some benefits and drawbacks to that approach?

Dirk: In the beginning, there were a few dozen (now a few hundred) mirrors of the openSUSE distribution on community sponsored mirrors (university FTP sites and similar). These are pulling new content at regular intervals via rsync from a staging server that the openSUSE infra team is operating. Users had a web page where they had to pick a mirror from and then hope their choice of mirror remained updated and available. That hope was not always justified.

Eventually openSUSE users were loud enough so that an effort to address those issues started. We realized that these community-provided mirrors can be partial mirrors (e.g., the site admin by themselves decided to not mirror some parts of our trees), outdated or even occasionally down or overloaded, so a former colleague started a project called MirrorBrain which got later revamped in a new project called MirrorCache. Both are basically a two-tiers open source framework to use a few trustworthy, reliable redirector proxies to HTTP redirect to the existing, partial, and potentially outdated or unreliable mirrors all over the world. For that, it continuously keeps track of which mirrors are currently available, what part of the content they serve and how up to date they are. When a client is contacting the main redirector, it does a client IP geo-lookup and redirects the request to a nearest available mirror that has the file.

This setup has served us fairly well, and even with the addition of the Fastly CDN remains in place for the foreseeable future. The addition of Fastly is going to help us in three areas:

• We more frequently ran into a situation when a client was requesting a particularly new file, there was no mirror available (because it hasn't been mirrored yet by any of the mirrors that we do not have direct influence over) or no good mirror nearby. In both cases we ended up serving the file from the main redirector, which took away the bandwidth that was shared with the mirrors who tried to fetch the updates themselves, so that we could leverage the mirror to offload from our bandwidth use. This often caused hours-to-days-long problems when particularly large snapshots of our premier rolling openSUSE Tumbleweed distributions were released, as users jumped on it before any mirror caught up on it.
• We realized we have in some areas of the world an abundance of fast, reliable mirrors, while in other areas where we have emerging users we have no mirrors at all or only somewhat limited mirrors. Fastly's large and well balanced CDN network is going to help us with serving those users better.
• Due to the growing popularity of the project, our primary redirector proxies are about to hit scalability limits due to the number of incoming requests that they need to process. The caching features of Fastly's CDN will help us with both reducing the incoming request rate as well as cutting down their latency significantly.

We're in a cautious rollout of the CDN, so both systems are currently in use in parallel. We see the benefits of the ability to offload a lot of load peaks to the CDN and will continue to refine the usage over time.

Hannah: What prompted you to want to switch to using Fastly? Any improvements you've seen in performance so far? Any other benefits?

We've done a set of controlled experiments with the Fastly trial program from various locations on the globe, and Fastly's CDN helped us to reduce request-handling latency in several regions by a factor of 5 and more. Due to a variety of reasons, the "redirect latency" is very visible for end users, so we are confident that this leads to a better experience for our users.

Also, as it improved the availability, reliability and geo-distribution of our mirrors, for many "remote" locations Fastly provides a significantly better bandwidth for our users when files are served from the Fastly CDN cache directly, although those effects are harder to quantify, in many locations that improved transfer times by about 20%, with bigger effects in locations where we, so far, had no mirrors at all.

Hannah: Twenty percent improvement! That’s great. I can’t wait to hear what you build on Fastly next. Where should people go if they want to get involved with openSUSE?

openSUSE.org has links on how to get in touch with the openSUSE community. news.opensuse.org is a good place to start with just following the news of the project. Coincidentally it is 18 years of openSUSE project this month :).

—

Happy birthday, openSUSE!! Cheers to 18 more years of free as in freedom.

Now let’s go build the good internet — together ⏩