Inotify in Containers

It is usually necessary to watch for any changes in file systems, both in development and in production modes. For example, in the development mode Webpack can watch files and recompile whenever they change; in the production mode Consul Template can watch runtime configs and invoke specific applications whenever they change.

These are well-known scenarios in traditional pre-container world. How about the container world? Do they behave the same in the new container world?

I’ve occasionally found that something behave differently in the container world. To clarify this I’ve conducted a series of experiments. In this article I’m going to explain the experiments and preliminary findings. All experiment materials are available in the fswatch repo.

Experiment setting

I’ve divided the experiments into 4 groups.

• Group A (experiment 1 to 3) is the traditional native mode: run native apps in their native host operating systems, respectively. This is considered as the control group.

• Group B (experiment 4 to 6) is the Linux container mode: run the same containerized Linux app in 3 different host operating systems.

• Group C (experiment 7) is the Windows container mode: run the containerized Windows app in the Windows operating system.

• Group D (experiment 8) is the Kubernetes mode: run the containerized Linux app in Kubernetes.

Group A: native mode

Let’s start with the native mode as the control group. We’ll see how it works in the traditional world of Linux, Mac, and Windows.

To make life easier, I’m using the fsnotify library to unify a variety of underlying operating system APIs (e.g., inotify in Linux, kqueue in macOS, and ReadDirectoryChangesW in Windows). Statically-linked binaries for the 3 platforms are generated with the Go compiler 1.12.5:

• 2,943,200 (bytes) fswatch-linux-x86_64
• 2,861,360 (bytes) fswatch-mac
• 2,914,304 (bytes) fswatch-x86_64.exe

These experiments are easy to try by yourself. Take experiment 1 “run native Linux app in Linux host OS” for example:

In this group, changes in the file system can be tracked successfully by their native API mechanisms, respectively.

Group B and C: container mode

Containers make things a little bit complicated.

TL;DR: The LCOW version doesn’t work well.

When there’s a mismatch between host OS and container, inotify may not work well.

Take experiment 6 (LCOW) “run containerized Linux app in Windows host OS” for example. The document for Docker Desktop for Windows has a warning for us:

Inotify on shared drives does not work

Currently, inotify does not work on Docker Desktop for Windows. This becomes evident, for example, when an application needs to read/write to a container across a mounted drive. Instead of relying on filesystem inotify, we recommend using polling features for your framework or programming language.

For Windows users, inotify works well in the WCOW mode (experiment 7).

On the other hand, Mac users are luckier. Docker Desktop for Mac doesn’t have much trouble here (experiment 5) thanks to excellent implementation of osxfs. See “File system sharing (osxfs)” and “Performance tuning for volume mounts (shared filesystems)” articles for more information.

File system events

Most inotify events are supported in bind mounts, and likely dnotify and fanotify (though they have not been tested) are also supported. This means that file system events from macOS are sent into containers and trigger any listening processes there.

Group D: K8s mode

How about Kubernetes? Does inotify work well with the ConfigMap?

Below is the demo for experiment 8:

As you can see in this demo, any changes in the ConfigMap will propagate to related pods in a couple of seconds, and inotify will detect this event as well. ¹

Caution about symbolic links

In such situation, however, you should use inotify to watch for directories instead of merely for files. It is because Kubernetes may use symbolic links to point to versioned ConfigMap volumes, and inotify doesn’t work well with such symbolic links. ²

Let’s retry the demo, but this time we’ll focus on the directory layout from the pod’s point of view.

The demo from 0:38 to 1:55 shows the directory layout from the pod’s point of view:

/mnt/site # ls -al
total 12
drwxrwxrwx    3 root  root   4096 Jun 12 06:19 .
drwxr-xr-x    1 root  root   4096 Jun 12 06:19 ..
drwxr-xr-x    2 root  root   4096 Jun 12 06:19 ..2019_06_12_06_19_15.187277003
lrwxrwxrwx    1 root  root     31 Jun 12 06:19 ..data -> ..2019_06_12_06_19_15.187277003
lrwxrwxrwx    1 root  root     15 Jun 12 06:19 main.css -> ..data/main.css

Simply put, things go well if you’re inside the pod and watch for /mnt/site directory, but may not go well if you try to watch for a specific file /mnt/site/main.css since it is internally a symbolic link managed by Kubernetes.

Conclusion

The inotify mechanism works in the container and Kubernetes world, except for the LCOW case.

If such a feature is needed in the LCOW setting, maybe you have to seek another workaround workflow; e.g., put inotify mechanism outside the container, and propagate the event explicitly into the containers if any. This is exactly what Skaffold is doing.

Also note that inotify works better when ConfigMap is mounted as directories, rather than as merely files.