Golang 1.20 upgrade failed due to GLIBC_2.34 not found


Tags: go

Remember that Java slogan, Write once, run anywhere? Go compiles directly to binaries, which target specific architectures. There’s no intermediate byte code. You cross-compile, and once you produce a binary for the target architecture: GOOS=linux GOARCH=amd64 go build, it just works. Well, at least usually.

With Cgo, go can interop with C code. There’s some magic involved, the toolchain behaves differently depending on your path (see below), but you usually don’t need to worry about it at all.

The cgo tool is enabled by default for native builds on systems where it is expected to work. It is disabled by default when cross-compiling as well as when the CC environment variable is unset and the default C compiler (typically gcc or clang) cannot be found on the system PATH. You can override the default by setting the CGO_ENABLED environment variable when running the go tool: set it to 1 to enable the use of cgo, and to 0 to disable it.

A simple upgrade goes wrong


Go is very good when it comes to backward compatibility. An upgrade is often painless and usually involves only changing FROM golang:<version-a>@sha256.. to FROM golang:<version-b>@sha256...

In my case, after the version upgrade, I hit some test failures due to more strict handling on the Host header (see 312920c), which was an easy fix.

Then, the container bootloops with the following message:

/my/app: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /my/app)

Looks like there’s a shared library–`GLIBC_2.34'–required by the produced binary, which is unavailable on the target.

The simplest solution when you don’t need cgo in your app would be to force CGO_ENABLED=0. In my case I need it.

Bisect to the rescue

How to debug what happened and when? We can use ldd to find out which shared libraries are required by a binary. We can then build the binary with different go images and see which one fails:

docker run --rm -it \
	-v $(pwd):/pwd \    # mount the current directory to the docker container
	my-docker-registry/my-app/my-app:<current-version> \
	ldd -v /pwd/my-app  # where my-app is built with the golang image you want to test 

Compare golang:1.19.6, the version that “works” …

	linux-vdso.so.1 =>  (0x00007ffdc6780000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f334b725000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f334b509000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f334b13b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f334b93f000)

	Version information:
		libresolv.so.2 (GLIBC_2.2.5) => /lib64/libresolv.so.2
		libpthread.so.0 (GLIBC_2.3.2) => /lib64/libpthread.so.0
		libpthread.so.0 (GLIBC_2.2.5) => /lib64/libpthread.so.0
		libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.16) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6

… with golang:1.20.7:

/pwd/my-app: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /pwd/my-app)
		libc.so.6 (GLIBC_2.34) => not found

Now, we can bisect this until we know which version we can easily upgrade to:

Go version Released on Works with my-app/my-app image? Comment
1.19.6 2023-02-14 Yes Currently in use
1.19.9 2023-05-02 Yes Latest 1.19 we can upgrade to
1.19.10 No
1.20.4 2023-05-02 Yes Latest 1.20 we can upgrade to
1.20.5 No

Problem solved, we’re now running on 1.20.4.

What about 1.20.7?

Upgrading to 1.20.4 is only a partial solution:

More importantly, we won’t be able to upgrade if an important hotfix lands in 1.20.8.

Looking at the recent changes to docker-library/golang (the go toolchain image), we find this: Add debian:bookworm distro for 1.20 and 1.19 · Pull Request #456 · docker-library/golang .

OK, so let’s check the debian versions between the images:

❯ docker run --rm -it golang:1.19.9 cat /etc/issue
Debian GNU/Linux 11 \n \l
❯ docker run --rm -it golang:1.19.10 cat /etc/issue
Debian GNU/Linux 12 \n \l

For reference: Debian versions, debian 11 nicknamed bullseye, and 12 nicknamed bookworm

Aha, the debian base for the go toolchain container changed between 1.19.9 and 1.19.10!

Luckily, bullseye is not gone for good (yet!), so we can force it for now:

FROM golang:1.20.7-bullseye@sha256:74b09b3b6fa5aa542df8ef974cb745eb477be72f6fcf821517fb410aff532b00

Won’t we have the same problem in a year, when bullseye-based images are no longer produced? Probably yes, but then bullseye has EOL a year from now. I expected that in a year we will migrate the base image to something different. If this was not the case, we will need to manually put the right version of glibc on the target. Out of scope here, but this stack overflow page discusses this https://stackoverflow.com/questions/55450061/go-build-with-another-glibc.


We’ve briefly discussed cgo, the challenges with shared libraries, and how to debug them. We’ve found a simple solution for my current problem, i.e., forcing a specific base image for the go toolchain container.

>> Home