Building a text preprocessor
Let's say we want to process a bunch of files, each in the same way. An easy example is a text preprocessor: we'll look through the text and replace variables of the form %VARNAME% with real content. In our example, we'll use variable %VERSION%, representing a version number, and %DATE%, with today's date.
To play with this code on your own machine, get the redo
source code and look in the
Let's create some input files we can use. The input to our
preprocessor will have file extension
.in. The output can be in various
For fun, let's also make a python test program that calls the newly generated version.py:
Finally, we need to provide values for the variables. Let's put each one
in its own file, named
Now we want to teach redo how to do our substitutions: how to, in general, generate file X from file X.in.
We could write a separate .do file for every output file X. For example, we
version.py.do. But that gets tedious. To
make it easier, if there is no specific
X.do for a target named X, redo
will try using
default.do instead. Let's write a default.do for our text
default.do is asked to build
X, and there exists a file named
sed to do our variable substitutions. In that case,
redo-ifchange to depend on
date. If a file named
X.in does not exist, then we don't know what to do, so we give an error.
On the other hand, if we try to generate a file that already exists, like
test.py, redo does not call default.do at all. redo only tries to create
files that don't exist, or that were previously generated by redo. This
stops redo from accidentally overwriting your work.
$ redo test.txt redo test.txt $ cat test.txt This is the documentation for MyProgram version 1.0. It was generated on 1970-01-01. $ redo chicken redo chicken default.do: Fatal: don't know how to build 'chicken' redo chicken (exit 99) $ redo version.py redo version.py # test.py was created by us, so it's a "source" file. # redo does *not* call default.do to replace it. $ redo test.py redo: test.py: exists and not marked as generated; not redoing. $ python test.py Version '1.0' has build date '1970-01-01'
While we're here, let's make an
all.do so that we don't have to tell redo
exactly which files to rebuild, every single time.
$ redo redo all redo test.txt redo version.py redo include/version.h # input files didn't change, so nothing to rebuild $ redo redo all $ touch test.txt.in $ redo redo all redo test.txt
Auto-generating the version and date (redo-always and redo-stamp)
Of course, in a real project, we won't want to hardcode the version number and date into a file. Ideally, we can get the version number from a version control system, like git, and we can use today's date.
To make that happen, we can replace the static
date files with
default.do already uses redo-ifchange to
date, so redo will create them as needed, and
if they change, redo will rebuild all the targets that depend on them.
date files are special: they depend on the
environment outside redo itself. That is, there's no way to declare a
dependency on the current date. We might generate the
date file once, but
tomorrow, there's no way for redo to know that its value should change.
To handle this situation, redo has the
redo-always command. If we run
redo-always from a .do file, it means every time someone depends on that
target, it will be considered out-of-date and need to be rebuilt. The
result looks like this:
$ redo redo all redo test.txt redo version redo date redo version.py redo include/version.h # version.do and date.do are redo-always, so # everything depending on them needs to rebuild # every time. $ redo redo all redo test.txt redo version redo date redo version.py redo include/version.h
Of course, for many uses, that's overcompensating: the version number and
date don't change that often, so we might end up doing a lot of
unnecessary work on every build. To solve that, there's
redo-stamp does the opposite of redo-always: while redo-always makes things
build more often, redo-stamp makes things build less often.
Specifically, it lets a .do file provide a "stamp value" for its output; if
that stamp value is the same as before, then the target should be considered
unchanged after all.
The most common stamp value is just the content itself. Since in redo, we write the content to $3, we can also read it back from $3:
And the final result is what we want. Although
generated every time, the targets which depend on them are not:
$ redo clean redo clean # version and date are generated just once per run, # the first time they are used. $ redo redo all redo test.txt redo version redo date redo version.py redo include/version.h # Here, (test.txt) means redo is considering building # test.txt, but can't decide yet. In order to decide, # it needs to first build date and version. After # that, it decides not to build test.txt after all. $ redo redo all redo (test.txt) redo date redo version
Sometimes you want to override a file even if it is a target (ie. it has previously been built by redo and has a valid .do file associated with it). In our example, maybe you want to hardcode the version number because you're building a release. This is easy: redo notices whenever you overwrite a file from outside redo, and will avoid replacing that file until you subsequently delete it:
$ echo "1.0" >version $ redo redo all redo (test.txt) redo date redo: version - you modified it; skipping redo test.txt redo version.py redo include/version.h $ redo redo all redo (test.txt) redo date redo: version - you modified it; skipping $ rm version $ redo redo all redo (test.txt) redo date redo version redo test.txt redo version.py redo include/version.h
default.do, subdirectories, and redo-whichdo
There's one more thing we should mention, which is the interaction of
default.do with files in subdirectories. Notice that we are building
include/version.h in our example:
$ redo include/version.h redo include/version.h redo version redo date $ cat include/version.h // C/C++ header file identifying the current version #ifndef __VERSION_H #define __VERSION_H #define VERSION "redo-0.31-3-g974eb9f" #define DATE "2018-11-26" #endif // __VERSION_H
redo works differently from the
make command when you ask it to build
files in subdirectories. In make's case, it always looks for a
in the current directory, and uses that for all build instructions. So
make include/version.h and
cd include && make version.h are two
different things; the first uses
Makefile, and the second uses
include/Makefile (or crashes if the latter does not exist).
redo, on the other hand, always uses the same formula to find a .do file for a particular target. For a file named X, that formula is as follows:
- first, try X.do
- then try default.do
- then try ../default.do
- then try ../../default.do
- ...and so on...
(Note: for targets with an extension, like X.o, redo actually tries even
more .do files, like
../default.o.do. For precise
details, read the redo man page.)
You can see which .do files redo considers for a given target by using the
redo-whichdo command. If redo-whichdo returns successfully, the last name
in the list is the .do file it finally decided to use.
$ redo-whichdo include/version.h include/version.h.do include/default.h.do include/default.do default.h.do default.do
Redo always runs in the .do file's directory
To ensure consistency, redo always changes the current directory to
the directory containing the selected .do file (not the directory
containing the target, if they are different). As a result,
redo include/version.h and
cd include && redo version.h always have
exactly the same effect:
$ redo include/version.h redo include/version.h redo version redo date $ (cd include && redo version.h) redo version.h redo ../version redo ../date
(redo's display is slightly different between the two: it always shows the files it's building relative to the $PWD at the time you started redo.)
This feature is critical to redo's recursive nature; it's the reason that essays like Recursive Make Considered Harmful don't apply to redo. Any redo target, anywhere in your source tree, can use redo-ifchange to depend on any of your other targets, and the dependency will work right.
Why does redo change to the directory containing the .do file, instead of
the directory containing the target? Because usually, the .do file needs to
refer to other dependencies, and it's easier to always express those
dependencies without adjusting any paths. In our text preprocessor example,
redo-ifchange version date; this wouldn't work properly
if it were running from the
include/ directory, because there are no files
date in there.
Similarly, when compiling C programs, there are
usually compiler options like
-I../mylib/include. If we're compiling
mydir/bar.o, we would like
-I../mylib/include to have the
same meaning in both cases.