It is usually necessary to watch for any changes in file systems, both in development and in production modes. For example, in the development mode Webpack can watch files and recompile whenever they change; in the production mode Consul Template can watch runtime configs and invoke specific applications whenever they change.
These are well-known scenarios in traditional pre-container world. How about the container world? Do they behave the same in the new container world?
I've occasionally found that something behave differently in the container world. To clarify this I've conducted a series of experiments. In this article I'm going to explain the experiments and preliminary findings. All experiment materials are available in the fswatch repo.
I've divided the experiments into 4 groups.
Group A (experiment 1 to 3) is the traditional native mode: run native apps in their native host operating systems, respectively. This is considered as the control group.
Group B (experiment 4 to 6) is the Linux container mode: run the same containerized Linux app in 3 different host operating systems.
Group C (experiment 7) is the Windows container mode: run the containerized Windows app in the Windows operating system.
Group D (experiment 8) is the Kubernetes mode: run the containerized Linux app in Kubernetes.
|Host OS||App run asnative app||App run asLinux Container||App run asWindows Container||‖||App run inK8s (Linux Container)|
|Windows||3||6 (LCOW)||7 (WCOW)|
Group A: native mode
Let's start with the native mode as the control group. We'll see how it works in the traditional world of Linux, Mac, and Windows.
To make life easier, I'm using the fsnotify library to unify a variety of underlying operating system APIs (e.g.,
inotify in Linux,
kqueue in macOS, and
ReadDirectoryChangesW in Windows). Statically-linked binaries for the 3 platforms are generated with the Go compiler 1.12.5:
- 2,943,200 (bytes)
- 2,861,360 (bytes)
- 2,914,304 (bytes)
These experiments are easy to try by yourself. Take experiment 1 “run native Linux app in Linux host OS” for example:
In this group, changes in the file system can be tracked successfully by their native API mechanisms, respectively.
Group B and C: container mode
Containers make things a little bit complicated.
TL;DR: The LCOW version doesn't work well.
When there's a mismatch between host OS and container,
inotify may not work well.
Inotify on shared drives does not work
inotifydoes not work on Docker Desktop for Windows. This becomes evident, for example, when an application needs to read/write to a container across a mounted drive. Instead of relying on filesystem
inotify, we recommend using polling features for your framework or programming language.
For Windows users,
inotify works well in the WCOW mode (experiment 7).
On the other hand, Mac users are luckier. Docker Desktop for Mac doesn't have much trouble here (experiment 5) thanks to excellent implementation of osxfs. See “File system sharing (osxfs)” and “Performance tuning for volume mounts (shared filesystems)” articles for more information.
File system events
inotifyevents are supported in bind mounts, and likely
fanotify(though they have not been tested) are also supported. This means that file system events from macOS are sent into containers and trigger any listening processes there.
Group D: K8s mode
How about Kubernetes? Does
inotify work well with the ConfigMap?
Below is the demo for experiment 8:
As you can see in this demo, any changes in the ConfigMap will propagate to related pods in a couple of seconds, and
inotify will detect this event as well. 1
Caution about symbolic links
In such situation, however, you should use
inotify to watch for directories instead of merely for files. It is because Kubernetes may use symbolic links to point to versioned ConfigMap volumes, and
inotify doesn't work well with such symbolic links. 2
Let's retry the demo, but this time we'll focus on the directory layout from the pod's point of view.
The demo from 0:38 to 1:55 shows the directory layout from the pod's point of view:
/mnt/site # ls -al total 12 drwxrwxrwx 3 root root 4096 Jun 12 06:19 . drwxr-xr-x 1 root root 4096 Jun 12 06:19 .. drwxr-xr-x 2 root root 4096 Jun 12 06:19 ..2019_06_12_06_19_15.187277003 lrwxrwxrwx 1 root root 31 Jun 12 06:19 ..data -> ..2019_06_12_06_19_15.187277003 lrwxrwxrwx 1 root root 15 Jun 12 06:19 main.css -> ..data/main.css
Simply put, things go well if you're inside the pod and watch for
/mnt/site directory, but may not go well if you try to watch for a specific file
/mnt/site/main.css since it is internally a symbolic link managed by Kubernetes.
inotify mechanism works in the container and Kubernetes world, except for the LCOW case.
If such a feature is needed in the LCOW setting, maybe you have to seek another workaround workflow; e.g., put
inotify mechanism outside the container, and propagate the event explicitly into the containers if any. This is exactly what Skaffold is doing.
Also note that
inotify works better when ConfigMap is mounted as directories, rather than as merely files.
Series of Articles
❶ Inotify in Containers