Persistent storage with Docker Swarm and NFS


I had an interesting project that required persistent storage in a swarm and this is a hot topic now so here is one solution to the problem.

It is tested with Docker for AWS and Amazon EFS , but should work on any host that runs alpine and any nfs4 server (Docker for Mac , Docker for Windows , Docker for AWS all use alpine)

Problem 1 - can’t access the Docker host

First I didn't even think to use a Docker image. Why not just ssh into each node instal the nfs client and mount the folder right ?

Well not so fast, the problem is that ssh takes you inside a container not in the host itself.

I created the vipconsult/moby-nfs-mount image that uses nsenter to escape the container and run commands directly on the host *(installs a nfs client and mounts the folder)

Problem 2 – can’t run docker service create –privileged

It is logical to run the image as a global service so it mounts the folder on every swarm node right ? But another problem - it need to run as privileged and docker service create doesn't support privileged mode so we need another solution.

Lucky enough #Madhu_Venugopal from the docker team integrated the swarm-exec script which does exactly what I need – run any command globally on all nodes – perfect!

swarm-exec \
        docker run -d \
        --privileged --pid=host \
        --restart=unless-stopped \
     -e SERVER=?????:/  -e MOUNT=/host/mount/folder vipconsult/moby-nfs-mount 

--restart=unless-stopped – ensures that it will be mounted again even after a host restart

/host/mount/folder will be mounted on the host and can be used from any another container

after it mounts the folder all it does is to output logs using inotifywait so stopping the container will not make the nfs disappear

Let's test it

add some data

docker run --rm \
    -v /host/mount/folder:/container/folder \
    alpine touch  /container/folder/testFile

and deploy a simple service that will show the folder content on every node

docker service create --mode=global \
    --mount type=bind,src=/host/mount/folder,dst=/container/folder \
    alpine /bin/sh -c  "while true; do ls  /container/folder; sleep 1; done"

Notes:

Amazon EFS is running a nfs4 server so I created this image only for nfs4 Hopefully this image becomes obsolete with the new infinity storage plugin, but until then feel free to use and comment on improvements.

So there you have it - easy persistent storage and Docker

https://github.com/vipconsult/dockerfiles/tree/master/moby-nfs-mount