Debugging docker containers with gdb and nsenter.
Many a time you feel the need to debug a process running inside the container with gdb (or anything that uses ptrace). It is not as straightforward (at least for now) as attaching gdb to host pid of the container process or to docker client/daemon.
You can go about in following ways:
- If you try to attach gdb to the hostpid (let us call it HPID) of the container process (ie. PID in host namespace), it gives following warning:
- You can install gdb in container beforehand or with docker exec and attach to the process. However, for a non-privileged container or one which doesn’t have CAP_SYS_PTRACE capability this won’t work. You will get ‘Permission denied’ otherwise. More info on capabilities with docker here.
Any of the following will do:
docker run -d -i --cap-add sys_ptrace ---name box centos:centos7 sleep 100000 docker run -d -i --privileged --name box centos:centos7 sleep 100000
Next, you need to:
docker exec -it box gdb -p 1 // assuming 1 is the pid of the process which it usually is
So, essentially, for this, you will need to restart your container (with higher capabilities) to attach or you will need to run all your containers with higher capabilities or as privileged by default. This is not desirable. Also, as mentioned before, gdb also needs to be installed inside container.
- So, lets assume for following discussion that container is started as:
docker run -d -i --name box3 centos:centos7 sleep 100000
- Next, you may want to try nsenter to get inside the namespace of the process. nsenter is a command which can be used to enter namespace of other processes. More on that here.
nsenter -t $HPID -p gdb -p 1
will not work. This can actually attach to PID 1 (systemd) of your host!
nsenter -t $HPID -m -p gdb -p 1
so that it uses both mount and PID namespaces, however, this will also only work if there is gdb installed inside container (Apparently, it seems you can use gdbserver without gdb inside container with nsenter.1).
But there is a bright side to this approach in that your container need not be a privileged one (or with any capabilities)! This means you can use docker exec to install gdb and then debug your container process from host.
(You will need to run nsenter as root since any namespace changes require admin privileges.)
- Now, a more elegant way is desirable. Fortunately, gdb developers have been working on it for a while. The gdb tree is available at https://github.com/gbenson/binutils-gdb.git under namespaces branch. Discussion about it here: https://sourceware.org/bugzilla/show_bug.cgi?id=18368.
After managing to build and install gdb from that branch, you can just do
gdb -p $HPID
and it works!
To conclude, the nsenter with PID and mount namespace strategy is a sound one, the gdb with container support is still in development, so will take a while before it makes into releases. Finally, for complete backtraces, you will need debug symbols installed on guest or you can source them on gdb of the host separately (provided you obtain the debug symbol package on host).
Credits to Gary Benson from Redhat for github tree and discussion in https://sourceware.org/bugzilla/show_bug.cgi?id=18368.
Photo credits: 1, 2.