Remember that Java slogan, Write once, run anywhere? Go compiles directly to binaries, which target specific architectures. There’s no intermediate byte code. You cross-compile, and once you produce a binary for the target architecture: GOOS=linux GOARCH=amd64 go build, it just works. Well, at least usually.

With Cgo, go can interop with C code. There’s some magic involved, the toolchain behaves differently depending on your path (see below), but you usually don’t need to worry about it at all.

The cgo tool is enabled by default for native builds on systems where it is expected to work. It is disabled by default when cross-compiling as well as when the CC environment variable is unset and the default C compiler (typically gcc or clang) cannot be found on the system PATH. You can override the default by setting the CGO_ENABLED environment variable when running the go tool: set it to 1 to enable the use of cgo, and to 0 to disable it.

A simple upgrade goes wrong

Context:

• I have a go app; CI builds the binary and puts it in a docker image as one of the components of a larger system.
• The large system is based on some enterprisey LTS distro like Centos 7.
• The application uses a C library via Cgo.
• The application is built with go 1.19.6 and I wanted to upgrade it to 1.20.7.

Go is very good when it comes to backward compatibility. An upgrade is often painless and usually involves only changing FROM golang:<version-a>@sha256.. to FROM golang:<version-b>@sha256...

In my case, after the version upgrade, I hit some test failures due to more strict handling on the Host header (see 312920c), which was an easy fix.

Then, the container bootloops with the following message:

/my/app: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /my/app)

Looks like there’s a shared library–`GLIBC_2.34’–required by the produced binary, which is unavailable on the target.

The simplest solution when you don’t need cgo in your app would be to force CGO_ENABLED=0. In my case I need it.

Bisect to the rescue

How to debug what happened and when? We can use ldd to find out which shared libraries are required by a binary. We can then build the binary with different go images and see which one fails:

docker run --rm -it \
	-v $(pwd):/pwd \    # mount the current directory to the docker container
	my-docker-registry/my-app/my-app:<current-version> \
	ldd -v /pwd/my-app  # where my-app is built with the golang image you want to test 

Compare golang:1.19.6, the version that “works” …

	linux-vdso.so.1 =>  (0x00007ffdc6780000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f334b725000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f334b509000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f334b13b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f334b93f000)

	Version information:
	/pwd/my-app:
		libresolv.so.2 (GLIBC_2.2.5) => /lib64/libresolv.so.2
		libpthread.so.0 (GLIBC_2.3.2) => /lib64/libpthread.so.0
		libpthread.so.0 (GLIBC_2.2.5) => /lib64/libpthread.so.0
		libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.16) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
..

… with golang:1.20.7:

/pwd/my-app: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /pwd/my-app)
..
		libc.so.6 (GLIBC_2.34) => not found
..

Now, we can bisect this until we know which version we can easily upgrade to:

Problem solved, we’re now running on 1.20.4.

What about 1.20.7?

Upgrading to 1.20.4 is only a partial solution:

• we don’t understand what has changed,
• we’re missing three patch versions.

More importantly, we won’t be able to upgrade if an important hotfix lands in 1.20.8.

Looking at the recent changes to docker-library/golang (the go toolchain image), we find this: Add debian:bookworm distro for 1.20 and 1.19 · Pull Request #456 · docker-library/golang .

OK, so let’s check the debian versions between the images:

❯ docker run --rm -it golang:1.19.9 cat /etc/issue
Debian GNU/Linux 11 \n \l

❯ docker run --rm -it golang:1.19.10 cat /etc/issue
Debian GNU/Linux 12 \n \l

For reference:

Aha, the debian base for the go toolchain container changed between 1.19.9 and 1.19.10!

Luckily, bullseye is not gone for good (yet!), so we can force it for now:

FROM golang:1.20.7-bullseye@sha256:74b09b3b6fa5aa542df8ef974cb745eb477be72f6fcf821517fb410aff532b00
..

Won’t we have the same problem in a year, when bullseye-based images are no longer produced? Probably yes, but then bullseye has EOL a year from now. I expected that in a year we will migrate the base image to something different. If this was not the case, we will need to manually put the right version of glibc on the target. Out of scope here, but this stack overflow page discusses this https://stackoverflow.com/questions/55450061/go-build-with-another-glibc.

Summary

We’ve briefly discussed cgo, the challenges with shared libraries, and how to debug them. We’ve found a simple solution for my current problem, i.e., forcing a specific base image for the go toolchain container.