TIP: use Strace to debug issues inside Docker

Yesterday my Docker application wasn't working correctly -- the appserver is hanging.  Debugging this is a challenge: there's no crash nor stack trace to point out the issue.  Is the appserver misconfigured, so it's trying to talk to a non-existent database? Is the config okay, but the network is not set up correctly?  Can an external service not see our Docker container correctly?

To debug this I used my good old "strace" command to trace exactly what is happening. It outputs log messages for all system calls the appserver does, including all the network I/O.  Alas it didn't work for me:

strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permitted

This is odd, as the Docker container is running with root permissions, and the parent container is Debian.

My buddy Loren says this is a Docker thing -- the ptrace system call (which strace uses) is disabled by default.  To run a Docker container, re-enabling ptrace, run this:

docker run -i -t --security-opt=seccomp:unconfined --rm debian sh -c 'apt update ; apt install -y strace; strace -e trace=network ping -c1 8.8.8.8'

The above command does some Docker stuff, then sends a single ICMP ping packet to Google's global (and easily-remembered) DNS server.

Output:

socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3

setsockopt(3, SOL_SOCKET, SO_BROADCAST, [-68864700], 4) = 0

PING 8.8.8.8 (8.8.8.8): 56 data bytes

sendto(3, "\10\0\363\330\0008\0\0b\207\23X\0\0\0\0\4~\f\0\0\0\0\0\0\1\2\3\4\5\6\7"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = 64

recvfrom(3, "E\0\0TU\r\0\0%\1\204y\10\10\10\10\254\21\0\2\0\0\373\330\0008\0\0b\207\23X"..., 136, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, [16]) = 84

64 bytes from 8.8.8.8: icmp_seq=0 ttl=37 time=0.339 ms

--- 8.8.8.8 ping statistics ---

1 packets transmitted, 1 packets received, 0% packet loss

round-trip min/avg/max/stddev = 0.339/0.339/0.339/0.000 ms

+++ exited with 0 +++

Note the "sendto" and "recvfrom" system calls.  They show that the Docker container can talk to the internet.  This is verified with the "packets" line.

Strace is so awesome, Julia "b0rk" Evans wrote an entire zine about it! ~ http://jvns.ca/zines/

Thanks Loren and Julia!

Never type a Bash command again with Infinite Shell History!

To turn on infinite history, add this to your ~/.bash_profile file :

# infinite timestamped history
export HISTTIMEFORMAT='%F %T '
export HISTSIZE=''

When next you log in, your terminal history will be saved forever!  I'm up to about 30,000 commands typed in the last two years.

Example

Today I was looking up that command, you know, that one where you can hop in to Docker container running the main appserver, for debugging and testing.  IE: run a Bash shell on a named container.

I knew it was "docker exec..." something:

history | egrep docker.*exec

26790  2016-08-31 16:18:12 docker exec -it e3f bash

Oops, I found the command where I created a Bash shell in a specific Docker container.  I'd have to used "docker ps", found the specific ID, and typed a second command to run a shell.  Lame.

How about running a command that uses docker ps to list containers, and gave it a "filter", so that the same command always works?

history | egrep docker.*exec.*filter

24559  2016-08-05 14:07:37 docker exec -it $(docker ps --filter ancestor=theblacktux_web,status=running --quiet) bash

That's it!

The above command will create a Bash shell in the currently running Docker container, where the container is running a custom version of the "theblacktux_web" appserver image.

TIP: work with Amazon AWS ECS Container Service Docker containers

When running Amazon EC2 Container Service (ECS), have these two commands running in Tmux on your instances:

tail -f $(ls -1t /var/log/ecs/ecs-agent.log.* | head -1)

sudo tail -f /var/log/upstart/docker.log


In another Tmux tab, this command lets you look at the logs from your running Docker container:

docker logs -f $(docker ps --filter name=myname,status=running --quiet)

TIP: fix bizarre Docker problems

docker rm -fv $(docker ps -aq)

Sometimes testing produces strange Docker-related problems.  The above command kills all containers, including the stopped ones.  The "-v" also destroys each container's private storage volumes.  This is great because Docker likes to run differently different times you run it -- because it has local state.  Local state is evil!  Run the above command every now and then so Docker starts from a clean, state-less, state.

In Docker, stopping a container is different from killing it.  The former stops the running container/process, but leaves the private volume.  The latter zaps the volume.

Thanks Sam!

TIP: find specific Docker container without using "cut" or other Shell silliness

People on the Interwebs love doing things in a leaden, verbose, laborious, and dusty manner.  Like piping a command through egrep then cut and sed then awk, when just "awk" will do.

Take Docker. We often want to find out which container is running something specific. Once we find our container, we want to "exec" a shell into it, or we want to kill it.

Here's the best way that doesn't mess with cut/sed/awk:

$ docker ps --filter ancestor=web,status=running --quiet

8fbc117716c3

In the previous command I was looking for the container ID that is descended from my "web" image. I'm not interested in my database nor Redis nor anything else, I want my web container.

Here's how to use it in practice, by killing the web container:

$ docker rm -f $(docker ps --filter ancestor=web,status=running --quiet)

8fbc117716c3

Also, here's how to use the same technique to hop into the web container, and run a shell.  This I use constantly, so I can examine a container to see what's going on, what it's doing, or what it's not doing:

$ docker exec -it $(docker ps --filter ancestor=web,status=running --quiet) bash
root@23b5a26eb252:/#

TIP: realtime logs from Nginx Docker container

docker logs -ft $(docker ps | awk '/nginx/ {print $1}')