When I was first learning how to use Docker, I ran into an interesting problem. I was trying to make a container with all the services needed to test an email client: SMTP and IMAP daemons, any services useful for debugging, plus any dependencies of those. Docker is perfect for this — one command to bring up a testbed environment, one command to shut it down and return it to a pristine state — but most of the Docker docs assume one process per container. I stumbled around a bit trying to figure out the best way to run all the processes I’d need. After I had a solution that seemed to work, as I was playing around inside the container, I would notice zombie processes piling up. Also, I found that running docker stop on the container would hang for a few seconds, and none of the running daemons would log any indication of being shut down before they were killed.

As I came to realize, these anomalies were caused by the process I was using to start my container. Let’s take a look at what I was doing, then I’ll go into more details about the problem and how to fix it. The full source is available on github, but I’ll summarize the relevant parts here for clarity.

Here’s the Dockerfile I was using:

FROM ubuntu
MAINTAINER Charles Lindsay

RUN apt-get update && apt-get upgrade
# ...Other stuff that sets things up so these packages install without error...
RUN apt-get install -y openssh-server dovecot-imapd postfix
# ...Other stuff to configure those packages...

# Set root password so we can log in.
RUN echo "root:root" | chpasswd

ADD start.sh /start.sh

ENTRYPOINT ["/start.sh"]

Basically, we’re installing the dovecot-imapd and postfix packages and openssh-server (which lets us log in to diagnose mail issues), setting up the system so everything will run without error, then telling Docker to run our start.sh script when it runs the container.

Here’s start.sh:

#!/bin/sh

/usr/sbin/sshd
/usr/sbin/dovecot
/usr/sbin/postfix start

exec /bin/sleep 999d

You can probably already tell this is going to be janky. Why do we need to sleep at the end? Docker will shut down the whole container when the process it runs for the container exits, so we need the script to hang until we explicitly kill it. We could instead run sshd or dovecot last and in the foreground (postfix has no option to run in the foreground, so that won’t work), but which one should it be? And what happens if you need to restart that service while the container is still running? You’d kill the whole container.

Because containers run in their own process namespace, whatever command you tell Docker to run becomes PID 1 inside the container. PID 1, more commonly known as init, has certain responsibilities in Unix systems:

• Run processes necessary to boot the system into a usable state.
• Gracefully stop running processes when the system is going down.
• Stay in the background as long as the system is up, reaping any terminated orphan processes (zombie processes).

Init probably has other responsibilities (see man init), but those are the ones we need to know about for containers.

With this in mind, it should be easy to see why my start.sh script approach was problematic: it completely ignores the middle bullet point and half of the last one. There’s probably some bash magic involving trap and wait that could get you closer to what’s needed (and if you look at the repo, you’ll see I started down that road but quickly gave up). Instead, I wanted a more general solution.

The Docker docs have a couple of examples of running multiple processes using Supervisor or CFEngine, but both of those seemed like overkill for just running a couple of daemons. Also, it could be handy to be able to SSH in and stop a server manually without having to work around a process management tool. Around this time, the baseimage-docker scare piece hit the news, so I knew I wasn’t alone in trying to solve this problem. Still, I wanted something that was simpler and gave me more control than baseimage-docker.

Not finding any existing projects that fit my needs, I set out to make my own. I call my solution minit, a minimalistic init implementation. It’s a simple C program that handles the duties of PID 1 inside Docker containers. Here are some simplified highlights of its operation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
static volatile pid_t shutdown_pid = 0;
static volatile int terminate = 0;

// SIGCHLD handler:
static void handle_child(int sig) {
    for(pid_t pid; (pid = waitpid(-1, NULL, WNOHANG)) > 0; ) {
        if(pid == shutdown_pid)
            shutdown_pid = 0;
    }
}

// SIGTERM/SIGINT handler:
static void handle_termination(int sig) {
    terminate = 1;
}

static pid_t run(const char *filename) {
    ...fork & exec, return child PID...
}

int main(int argc, char *argv[]) {
    ...Set up our signal handlers...

    run("/etc/minit/startup");

    while(!terminate)
        ...Suspend until we receive a signal...

    shutdown_pid = run("/etc/minit/shutdown");
    while(shutdown_pid > 0)
        ...Suspend until we receive a signal...

    kill(-1, SIGTERM);
    while(wait(NULL) > 0)
        continue;

    return 0;
}

Following the logic in main: after first setting up the signal handlers, we run the startup program, where container authors can define what services will be started when the container starts. Then, we wait around in the background for the terminate flag to be set by our SIGTERM handler (line 14). While we’re in the background, we’re calling wait on any terminated children (line 6), inherited or otherwise, so zombies don’t pile up. Once we’ve gotten the signal to terminate, we run the shutdown program, where container authors can define any behavior that needs to happen before all processes in the container are killed. We capture the shutdown program’s PID in shutdown_pid and check for its termination in our SIGCHLD handler (line 7) because we need to let it finish on its own before the final step: sending all running processes a SIGTERM and waiting for them all to end before exiting ourselves.

I’ve left out some esoteric signal-handling details and a couple bonus features here, but the full program is still pretty simple, weighing in at fewer than 150 lines including the license and other comments. Most importantly, minit covers all the above bullet points on init’s responsibilities inside containers.

Now that we have a solution to the problems I mentioned above, let’s change the container to use it (though since the original code was for a former place of employment, I won’t update the code on github). We have to add minit now, and move our script to where minit can find it. We’ll also rename the script from start.sh to startup, and simply get rid of the sleep line at the end. Here are the final few lines of the new Dockerfile:

ADD minit /sbin/minit
ADD startup /etc/minit/startup

ENTRYPOINT ["/sbin/minit"]

It’s now a hard requirement that we use the exec form of ENTRYPOINT (i.e. inside []) so Docker executes minit directly instead of starting it under a shell.

Sure enough, when I run this version, no zombie processes pile up and docker stop shuts things down immediately and gracefully. Problems solved!