Background

When implementing SQL Review for PostgreSQL, we introduced pg_query_go as the PostgreSQL parser. The pg_query_go uses the native PostgreSQL parser via C bind, which naturally requires CGO support.

In the Bytebase 1.2.1 release, we found that the GoReleaser did not work correctly, and the error message pointed to pg_query_go. Fortunately, this version did not use pg_query_go. So we first added the "!release" Golang tag to ensure a successful release. Then we started the long road to fight against GoReleaser and CGO.

Clue One

build constraints exclude all Go files in /go/pkg/mod/github.com/pganalyze/pg_query_go/v2@2.1.2/parser

This error message may seem a bit puzzling, but it points to exactly what’s wrong. So let's go to the package "pg_query_go/parser" and find out.

The package "pg_query_go/parser" contains only one Go file, parser.go, but otherwise, it's all C code. The "parser.go" is a CGO file that imports "C." It's easy to guess that the Go parser ignores this file when not set CGO_ENABLE=1.

We can easily verify it by Googling or by manual testing. There, we got our clue to the next intersection: set CGO_ENABLED=1 in GoReleaser.

Clue Two

To open CGO in GoReleaser, you only need to add "CGO_ENABLED=1" to the corresponding "env" entry in the GoReleaser configuration file.

Let's retry.

This error looks strange. We can only know that there is something wrong with CGO. It seems that GoReleaser does not support CGO. After looking it up in the GoReleaser docs, my suspicions were confirmed.

But the docs shed some light: the "This project" mentioned is actually goreleaser/goreleaser-cross, so let's go to goreleaser-cross next!

Clue Three

The goreleaser-cross repository provides a Docker image that contains a GoReleaser and some C/C++ cross-compiler toolchains.

The cross-compiler is used in scenarios where binaries must be compiled on platform A but executed directly on platform B. For example, in embedded and operating system development scenarios, our development machine is often an x64 Linux platform, but the runtime environment may be an arm/arm64 Linux platform.

So why do we need this here? You may have noticed that we have specified four target platforms in the GoReleaser configuration file.

Our build environment is Ubuntu x64 on GitHub Action. Before turning on CGO, we only need to handle the parameters for cross-platform Go compilation. And this step is handled by the Go compiler and GoReleaser. But after CGO is introduced, we also need to compile C/C++ code, so we need the corresponding C/C++ cross-compilation toolchain.

In other words, we may need four compilation toolchains for the target platform. Fortunately, goreleaser-cross supports them all.

Now it looks like the problem is solved, but it is not perfect. The goreleaser/goreleaser-cross repo has only 26 stars, while goreleaser/goreleaser has 10k+ stars. It could mean that using goeleaser-cross have some risks.

If the cross-compilation toolchain is well maintained, the risk is relatively small. The main problem is when the target platform is Darwin. Anyone who has tried to cross-compile from Linux to Darwin knows this is a challenging task. The difficulty is that there is no readily available, well-maintained cross-compilation toolchain. You often need to build your cross-compilation toolchain. It's had to verify the compatibility of the cross-compilation toolchain, and some of them are not well-maintained. The o64-clang/oa64-clang cross-compilation toolchain used by the GoReleaser-cross is built based on the osxcross.

So is there a way to avoid this risk? Yes.

Let’s Try Another Way

As mentioned before, we are using Ubuntu x64 on GitHub Action. The easiest way to avoid using the cross-compilation toolchain is to compile on the same platform. Does GitHub Action provide any other environment? Yes, it does!

Note that the macOS supported are all x64 architecture. We still need to compile across the architectures but no need to cross the platforms!

So the next step becomes: to compile the Linux binary for both architectures on the Linux platform and compile the Darwin binary for both architectures on the macOS platform.

For the Linux platform

• Compiling from x64 to x64 requires a gcc/g++ toolchain, which we usually use.
• Compiling from x64 to arm64 requires a well-maintained toolchain, "aarch64-linux-gnu-gcc". You can get it directly from the Ubuntu package manager:

sudo apt-get -y install gcc-aarch64-linux-gnu

For the Darwin platform, it's even easier. The clang supports cross-architecture natively. Thanks to LLVM!

After some work, the GitHub Action configuration looks like this.

Two jobs are used to compile binaries on different platforms.

For the Darwin platform GoReleaser configuration file, we just need to open CGO.

For the Linux platform GoReleaser configuration file, it's a bit more complicated, and we need to use the corresponding compilation toolchain for the corresponding architecture.

We used GoReleaser's "overrides" feature here.

Up to this point, we can use GoReleaser to compile when CGO is on, but the story is not over yet. The ultimate goal of GoReleaser is to release, and if we use GoReleaser twice as we currently do, we will have two release messages. But this step is easier than before.

Release!

The main idea here is simple: separate the building and the release.

• In job one, build Linux binaries.
• In job two, build macOS binaries.
• Job three depends on job one and job two. Then it does the final release.

Job one and two skip the release with the skip-publish argument. Job three skips the building and uses the files generated in jobs one and two with the extra_files setting.

🚀End of story!

A fun fact

Bytebase already depended on go-sqlite3, which is also a CGO-dependent package. So why didn't it cause problems before?

The thing is CGO was not opened in GoReleaser. And go-sqlite3 did a mock when CGO was not opened, which means that the go-sqlite3 compiled without CGO was an empty package.

The reason why we didn't encounter problems is that Bytebase had already migrated from SQLite to PostgreSQL before GoReleaser was introduced. So it really didn't use go-sqlite3 and GoReleaser at the same time.

Summary

Don't try to introduce a cross-compilation toolchain unless you absolutely have to. It will incur additional verification and maintenance costs.

Cross-architecture and cross-platform complicate things and make life harder. Please stay away from them as much as possible.

Thank you, LLVM-Clang.