TIP: use Strace to debug issues inside Docker

Yesterday my Docker application wasn't working correctly -- the appserver is hanging.  Debugging this is a challenge: there's no crash nor stack trace to point out the issue.  Is the appserver misconfigured, so it's trying to talk to a non-existent database? Is the config okay, but the network is not set up correctly?  Can an external service not see our Docker container correctly?

To debug this I used my good old "strace" command to trace exactly what is happening. It outputs log messages for all system calls the appserver does, including all the network I/O.  Alas it didn't work for me:

strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permitted

This is odd, as the Docker container is running with root permissions, and the parent container is Debian.

My buddy Loren says this is a Docker thing -- the ptrace system call (which strace uses) is disabled by default.  To run a Docker container, re-enabling ptrace, run this:

docker run -i -t --security-opt=seccomp:unconfined --rm debian sh -c 'apt update ; apt install -y strace; strace -e trace=network ping -c1 8.8.8.8'

The above command does some Docker stuff, then sends a single ICMP ping packet to Google's global (and easily-remembered) DNS server.

Output:

socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3

setsockopt(3, SOL_SOCKET, SO_BROADCAST, [-68864700], 4) = 0

PING 8.8.8.8 (8.8.8.8): 56 data bytes

sendto(3, "\10\0\363\330\0008\0\0b\207\23X\0\0\0\0\4~\f\0\0\0\0\0\0\1\2\3\4\5\6\7"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = 64

recvfrom(3, "E\0\0TU\r\0\0%\1\204y\10\10\10\10\254\21\0\2\0\0\373\330\0008\0\0b\207\23X"..., 136, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, [16]) = 84

64 bytes from 8.8.8.8: icmp_seq=0 ttl=37 time=0.339 ms

--- 8.8.8.8 ping statistics ---

1 packets transmitted, 1 packets received, 0% packet loss

round-trip min/avg/max/stddev = 0.339/0.339/0.339/0.000 ms

+++ exited with 0 +++

Note the "sendto" and "recvfrom" system calls.  They show that the Docker container can talk to the internet.  This is verified with the "packets" line.

Strace is so awesome, Julia "b0rk" Evans wrote an entire zine about it! ~ http://jvns.ca/zines/

Thanks Loren and Julia!