Fun with docker networks
How to find out a container responsible for a given open port on a host.
Recently, I’ve debugged an issue with a server running multiple docker containers. They interact with each other by exposing TCP ports on the host and communicating over these ports.
In this post:
- I describe a surprising discovery about host network mode containers (spoiler: they don’t show under
- Show to find a container responsible for a given port on the host,
- Link to a repo where I investigate docker networking in much more detail.
Let’s say we have the container exposing
:32080 on the host:
$ docker run -d --name my-service-17 -p 32080:80 nginx $ curl localhost:32080 .. <h1>Welcome to nginx!</h1> ..
We found out that
my-service-4 reports this error in the logs:
ERROR Unexpected response from https://<container-host>:32080/_important
How do you tie it back to
docker ps reports the published ports. You can do for example:
$ docker ps -a | grep 32080 abfa35be07f8 nginx "/docker-entrypoint.…" 7 minutes ago Up 7 minutes 0.0.0.0:32080->80/tcp my-service-17
At some point of the debug session, I looked for another port, say
docker ps -a | grep 32567 was empty – no results. To add insult to the injury there was indeed something on the other side:
$ curl localhost:32567 .. I'm here!
I was scratching my head. There are of course other tools in software engineers’ 🧰, e.g.
ss. We can find the PID of the process and then trace it back to its container:
$ sudo netstat -ntlp | grep ':32567' tcp 0 0 0.0.0.0:32567 0.0.0.0:* LISTEN 884710/nginx: maste # 884710 is the process PID $ for i in $(docker ps -a -q); do docker inspect -f ' ' $i | grep 884710; done 884710 /my-service-15
And we have the offender, it’s
Host network mode
What is different about
my-service-15 is that it runs with
--network host, which means it doesn’t get its own network namespace. Network-wise it acts as a process running directly on the localhost.
At first, I was surprised that the port doesn’t show up in the
docker ps overview, but it makes sense – the list of open ports is not static, so it would be quite hard for docker to track it. Plus there is the standard OS tooling.
The image shows docker containers running with various network modes. To have some fun and research the details of docker networking I’ve created igor-kupczynski/fun-with-docker-networks repo. I encourage you to check it out if you want to learn more!
In the post, we’ve shown how to find a container responsible for a port on the host. We can do that regardless of the network mode of the container:
- If it uses the default
bridgenetwork and simply publishes the port then
docker ps -a | grep <port>is enough.
- If it runs int the host network mode we have to fall back to standard linux tooling to find its PID. Then we can tie the PID back to the docker container.
I’ve also created a igor-kupczynski/fun-with-docker-networks repo to dive deeper into the subject.