{"id":195,"date":"2015-05-05T22:32:48","date_gmt":"2015-05-05T17:02:48","guid":{"rendered":"https:\/\/blog.wnohang.net\/?p=195"},"modified":"2020-05-08T21:58:41","modified_gmt":"2020-05-08T16:28:41","slug":"debugging-docker-containers-with-gdb-and-nsenter","status":"publish","type":"post","link":"https:\/\/blog.wnohang.net\/index.php\/2015\/05\/05\/debugging-docker-containers-with-gdb-and-nsenter\/","title":{"rendered":"Debugging docker containers with gdb and nsenter."},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 3<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><p>Many a time you feel the need to debug a process running inside the container with gdb (or anything that uses ptrace). It is not as straightforward (at least for now) as attaching gdb to host pid of the container process or to docker client\/daemon.<\/p>\n<p><a href=\"https:\/\/blog.wnohang.net\/wp-content\/uploads\/2015\/05\/GDBLogo.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/blog.wnohang.net\/wp-content\/uploads\/2015\/05\/GDBLogo.jpg\" alt=\"GDBLogo\" width=\"200\" height=\"125\" class=\"aligncenter size-full wp-image-201\" \/><\/a><br \/>\n<a href=\"https:\/\/blog.wnohang.net\/wp-content\/uploads\/2015\/05\/docker.png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/blog.wnohang.net\/wp-content\/uploads\/2015\/05\/docker-300x232.png\" alt=\"docker\" width=\"300\" height=\"232\" class=\"aligncenter size-medium wp-image-203\" srcset=\"https:\/\/blog.wnohang.net\/wp-content\/uploads\/2015\/05\/docker-300x232.png 300w, https:\/\/blog.wnohang.net\/wp-content\/uploads\/2015\/05\/docker.png 792w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<h2>You can go about in following ways:<\/h2>\n<ul>\n<li>If you try to attach gdb to the hostpid (let us call it HPID) of the container process (ie. PID in host namespace), it gives following warning:<\/li>\n<\/ul>\n<p><script src=\"https:\/\/gist.github.com\/anonymous\/332c1bee50b81a0179f6.js\"><\/script><\/p>\n<ul>\n<li>You can install gdb in container beforehand or with docker exec and attach to the process. However, for a non-privileged container or one which doesn&#8217;t have CAP_SYS_PTRACE capability this won&#8217;t work. You will get &#8216;Permission denied&#8217; otherwise.  More info on capabilities with docker <a href=\"https:\/\/docs.docker.com\/articles\/security\/#linux-kernel-capabilities\">here<\/a>.\n<p>Any of the following will do:<\/li>\n<\/ul>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n    docker run   -d  -i  --cap-add sys_ptrace ---name box centos:centos7 sleep 100000\r\n    docker run   -d  -i  --privileged  --name box centos:centos7 sleep 100000\r\n<\/pre>\n<p><!--more--><\/p>\n<p>Next, you need to:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ndocker exec -it box gdb -p 1  \/\/ assuming 1 is the pid of the process which it usually is\r\n<\/pre>\n<p>So, essentially, for this, you will need to restart your container (with higher capabilities) to attach or you will need to run all your containers with higher capabilities or as privileged by default. This is not desirable. Also, as mentioned before, gdb also needs to be installed inside container.<\/p>\n<ul>\n<li>So, lets assume for following discussion that container is started as:<\/li>\n<\/ul>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ndocker run  -d -i  --name box3 centos:centos7 sleep 100000\r\n<\/pre>\n<ul>\n<li>Next, you may want to try nsenter to get inside the namespace of the process. nsenter is a command which can be used to enter namespace of other processes. More on that <a href=\"http:\/\/man7.org\/linux\/man-pages\/man1\/nsenter.1.html\">here<\/a>.<\/li>\n<\/ul>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n    nsenter -t $HPID -p gdb -p 1  \r\n<\/pre>\n<p>will not work. This can actually attach to PID 1 (systemd) of your host!<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n    nsenter -t $HPID -m -p gdb -p 1  \r\n<\/pre>\n<p>so that it uses both mount and PID namespaces, however, this will also only work if there is gdb installed inside container (Apparently, it seems you can use gdbserver without gdb inside container with nsenter.<sup id=\"fnref-195-1\"><a href=\"#fn-195-1\" rel=\"footnote\">1<\/a><\/sup>).<\/p>\n<p>But there is a bright side to this approach in that your container need not be a privileged one (or with any capabilities)! This means you can use docker exec to install gdb and then debug your container process from host.<br \/>\n(You will need to run nsenter as root since any namespace changes require admin privileges.)<\/p>\n<ul>\n<li>Now, a more elegant way is desirable. Fortunately, gdb developers have been working on it for a while. The gdb tree is available at <a href=\"https:\/\/github.com\/gbenson\/binutils-gdb.git\">https:\/\/github.com\/gbenson\/binutils-gdb.git<\/a> under namespaces branch. Discussion about it here: <a href=\"https:\/\/sourceware.org\/bugzilla\/show_bug.cgi?id=18368\">https:\/\/sourceware.org\/bugzilla\/show_bug.cgi?id=18368<\/a>.<\/li>\n<\/ul>\n<li>\n<p>After managing to build and install gdb from that branch, you can just do<\/p>\n<\/li>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ngdb -p $HPID\r\n<\/pre>\n<p>and it works!<\/p>\n<p><script src=\"https:\/\/gist.github.com\/ronin13\/4fc2f49152ccb3fc2692.js\"><\/script><\/p>\n<p>To conclude, the nsenter with PID and mount namespace strategy is a sound one, the gdb with container support is still in development, so will take a while before it makes into releases. Finally, for complete backtraces, you will need debug symbols installed on guest or you can source them on gdb of the host separately (provided you obtain the debug symbol package on host).<\/p>\n<p>Credits to Gary Benson from Redhat for  github tree and discussion in  <a href=\"https:\/\/sourceware.org\/bugzilla\/show_bug.cgi?id=18368\">https:\/\/sourceware.org\/bugzilla\/show_bug.cgi?id=18368<\/a>.<br \/>\nPhoto credits: <a href=\"http:\/\/upload.wikimedia.org\/wikipedia\/commons\/1\/1e\/GDBLogo.jpg\">1<\/a>, <a href=\"http:\/\/blog.docker.com\/wp-content\/uploads\/2013\/06\/Docker-logo-011.png\">2<\/a>.<\/p>\n<div class=\"footnotes\">\n<hr \/>\n<ol>\n<li id=\"fn-195-1\">\n<a href=\"https:\/\/sourceware.org\/bugzilla\/show_bug.cgi?id=18368#c3\">https:\/\/sourceware.org\/bugzilla\/show_bug.cgi?id=18368#c3<\/a>&#160;<a href=\"#fnref-195-1\" rev=\"footnote\">&#8617;<\/a>\n<\/li>\n<\/ol>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 3<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>Many a time you feel the need to debug a process running inside the container with gdb (or anything that uses ptrace). It is not as straightforward (at least for now) as attaching gdb to host pid of the container process or to docker client\/daemon. You can go about in following ways: If you try &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/blog.wnohang.net\/index.php\/2015\/05\/05\/debugging-docker-containers-with-gdb-and-nsenter\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Debugging docker containers with gdb and nsenter.&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[50],"tags":[51,52,53,57,54,55,56],"class_list":["post-195","post","type-post","status-publish","format-standard","hentry","category-tech","tag-container","tag-docker","tag-gdb","tag-namespace","tag-nsenter","tag-ptrace","tag-trace"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p3AlYV-39","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":59,"url":"https:\/\/blog.wnohang.net\/index.php\/2014\/04\/30\/saving-form-data\/","url_meta":{"origin":195,"position":0},"title":"Saving form data in firefox","author":"Raghavendra","date":"April 30, 2014","format":false,"excerpt":"When commenting on sites, I have sometimes, seen that the commenting system just swallows the comment, or there is a browser crash, or a system one. In these cases it would be great if you can recover it somehow, particularly when you typed quite a bit. There are plugins for\u2026","rel":"","context":"Similar post","block_context":{"text":"Similar post","link":""},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":114,"url":"https:\/\/blog.wnohang.net\/index.php\/2014\/05\/03\/annoying-access-keys-web-pages\/","url_meta":{"origin":195,"position":1},"title":"Annoying access keys on web pages","author":"Raghavendra","date":"May 3, 2014","format":false,"excerpt":"HTML access keys were introduced to improve web accessibility and they still seem to serve that purpose. Wikipedia defines them as \"In a web browser, an access key or accesskey allows a computer user to immediately jump to a specific part of a web page via the keyboard.\" If you\u2026","rel":"","context":"In \"accesskey\"","block_context":{"text":"accesskey","link":"https:\/\/blog.wnohang.net\/index.php\/tag\/accesskey\/"},"img":{"alt_text":"5463604427_8e7998ddcc_b","src":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/5463604427_8e7998ddcc_b.jpg?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/5463604427_8e7998ddcc_b.jpg?resize=350%2C200 1x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/5463604427_8e7998ddcc_b.jpg?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/5463604427_8e7998ddcc_b.jpg?resize=700%2C400 2x"},"classes":[]},{"id":160,"url":"https:\/\/blog.wnohang.net\/index.php\/2015\/04\/29\/feedback-directed-optimization-with-gcc-and-perf\/","url_meta":{"origin":195,"position":2},"title":"Feedback directed optimization with GCC and Perf","author":"Raghavendra","date":"April 29, 2015","format":false,"excerpt":"Gcc 5.0 has added support for FDO which uses perf to generate profile. There is documentation for this in gcc manual, to quote: -fauto-profile=path Enable sampling-based feedback-directed optimizations, and the following optimizations which are generally profitable only with profile feedback available: -fbranch-probabilities, -fvpt, -funroll-loops, -fpeel-loops, -ftracer, -ftree-vectorize, -finline-functions, -fipa-cp, -fipa-cp-clone,\u2026","rel":"","context":"In \"fdo\"","block_context":{"text":"fdo","link":"https:\/\/blog.wnohang.net\/index.php\/tag\/fdo\/"},"img":{"alt_text":"feedback","src":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2015\/04\/feedback.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":126,"url":"https:\/\/blog.wnohang.net\/index.php\/2014\/05\/03\/twitter-setup\/","url_meta":{"origin":195,"position":3},"title":"My twitter setup","author":"Raghavendra","date":"May 3, 2014","format":false,"excerpt":"I have been using Twitter for a while under the handle randomsurfer. I tend to use web interface sometimes, but regularly I use the command-line\/ncurses interface. There are two main clients that I use regularly. One is ttytter which is a nice command-line client which offers advanced functionality such as\u2026","rel":"","context":"In \"bitlbee\"","block_context":{"text":"bitlbee","link":"https:\/\/blog.wnohang.net\/index.php\/tag\/bitlbee\/"},"img":{"alt_text":"2014-05-03-150028_1920x1060_scrot","src":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/2014-05-03-150028_1920x1060_scrot-1024x565.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/2014-05-03-150028_1920x1060_scrot-1024x565.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2014\/05\/2014-05-03-150028_1920x1060_scrot-1024x565.png?resize=525%2C300 1.5x"},"classes":[]},{"id":319,"url":"https:\/\/blog.wnohang.net\/index.php\/2020\/05\/12\/circuit-breakers-stock-markets-and-distributed-systems\/","url_meta":{"origin":195,"position":4},"title":"Circuit Breakers:  Stock Markets and Distributed Systems","author":"Raghavendra","date":"May 12, 2020","format":false,"excerpt":"There are many parallels between the stock markets and the distributed systems in computer science. This post, in particular, is about circuit breakers prevalent in them for better resilience against\u00a0irrational exuberance\u00a0and upstream service errors respectively. In particular, this is about exploring breakers in stock markets from a distributed systems perspective.\u00a0\u2026","rel":"","context":"In &quot;musings&quot;","block_context":{"text":"musings","link":"https:\/\/blog.wnohang.net\/index.php\/category\/musings\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2020\/05\/Circuit_Breaker_115_kV.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2020\/05\/Circuit_Breaker_115_kV.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2020\/05\/Circuit_Breaker_115_kV.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":349,"url":"https:\/\/blog.wnohang.net\/index.php\/2022\/12\/11\/weekend-with-chatgpt\/","url_meta":{"origin":195,"position":5},"title":"Weekend with ChatGPT","author":"Raghavendra","date":"December 11, 2022","format":false,"excerpt":"A few days ago, OpenAI released a chat-based model called\u00a0ChatGPT\u00a0and provided an interface for users to interact with. ChatGPT is a form of conversational AI where you can ask questions or have a conversation with a bot backed by a model. As per the announcement - The dialogue format makes\u2026","rel":"","context":"In \"ai\"","block_context":{"text":"ai","link":"https:\/\/blog.wnohang.net\/index.php\/tag\/ai\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2022\/12\/Screenshot-2022-12-11-at-20.11.10.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2022\/12\/Screenshot-2022-12-11-at-20.11.10.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2022\/12\/Screenshot-2022-12-11-at-20.11.10.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2022\/12\/Screenshot-2022-12-11-at-20.11.10.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2022\/12\/Screenshot-2022-12-11-at-20.11.10.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/blog.wnohang.net\/wp-content\/uploads\/2022\/12\/Screenshot-2022-12-11-at-20.11.10.png?resize=1400%2C800&ssl=1 4x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/posts\/195","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/comments?post=195"}],"version-history":[{"count":16,"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/posts\/195\/revisions"}],"predecessor-version":[{"id":262,"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/posts\/195\/revisions\/262"}],"wp:attachment":[{"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/media?parent=195"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/categories?post=195"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.wnohang.net\/index.php\/wp-json\/wp\/v2\/tags?post=195"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}