When developing there is often an edit-compile-test cycle, or an edit-distribute-changes cycle or a similar repetetive task. You could poll changes, for example with cron every minute or similarly, but that is wasteful and slow.
All modern operating systems have mechanisms for processes to subscribe to file or directory changes. In MacOS, we do have the File System Events API since 10.5, in Linux we got three different implementations (as described in LWN): The original dnotify, its replacement inotify and the even more recent fanotify (which got its own LWN article). BSD has kqueue.
The idea is that you subscribe to a directory and get notified for change/create/delete/rename events inside that directory and/or all events recursively beneath that starting point (a ‘root’). You would be interested into the type of change and the name of the file path that changes, and you would probably want to be able to retrieve lists of these changes in batch.
To make that useful, you would need a shell interface to this, and there are quite a few by now.
- The most convenient seems to be entr, because it works most closely with shell programs.
- One of the first programs to use filesystem subscriptions is fswatch, but while highly portable, it is cumbersome to use. Instead of running commands, it just reports filenames to feed into a pipe to handle.
- Ruby seems to have a library called Guard that also comes with an interface to shell, but can also being used as a ruby gem.
- spy is a weird piece of Haskell that produces a small binary that can run commands on file system changes.
Python seems to come with a bunch of modules and interfaces in various states of disrepair, inotify-tools, the very tiny wrapper inotify_simple (the simple here refers to the fact that it is a very thing wrapper around the C library, not simple to use), the more convenient inotify and the high level wrapper watchdog.
A test scenario
As a test scenario I have a
ship-to-kvm command that I want to
run on every file change. It looks like this:
when I save my local file from my local editor so that the tree myproject is made available on my devbox.
With entr, that is rather
simple. The package
entr is available in Homebrew on MacOS
brew install entr) or as a package in Linux (
yum install -y
apt install entr).
You ask entr to watch a list of files or a directory, and when
things change to run a command. You can hit space to force
execution even when nothing changed, or
q to end the command.
Various ways to handle changes are provided:
-r option here will SIGTERM the node instance, wait for it
to complete and then restart it.
To get notification of new and deleted files, you need to watch
directories, which are inferred from a file list. This is done
-d option and in fact the command terminates so you
need to wrap it in a loop:
There are a few other options, but these two should cover the most common use cases.
watchman is the facebook take on things. It consists of a daemon that is automatically started when you are using the frontend command, and a frontend command that actually does not expose all the functionality unless you feed it JSON job files. All command results are also JSON.
watchman has the concept of roots, filesystem subtrees that are being watched, and then triggers that are attached to roots or subtrees of roots, and are being run on change. A simple predicate language and a selection of regex libraries can be used to formulate conditions for triggers.
and the actual job definition is then something like
This may look nicer to developers, but I seem to prefer the entr way of doing things.
The Python library watchdog provides a convenient programmatic interface to inotify and friends by defining an Observer class and scheduling operations to the observer when there are events outstanding.
The example from the manual looks like this:
and when run does things like this for a
touch keks; sleep 1;
rm keks in a secondary shell:
The actual observer selection allows a rich palette of event classes and filters, so dispatching and filering events is easy.