Text substitution (default.do, redo-always, redo-stamp)

Building a text preprocessor

Let's say we want to process a bunch of files, each in the same way. An easy example is a text preprocessor: we'll look through the text and replace variables of the form %VARNAME% with real content. In our example, we'll use variable %VERSION%, representing a version number, and %DATE%, with today's date.

To play with this code on your own machine, get the redo source code and look in the docs/cookbook/defaults/ directory.

Input files

Let's create some input files we can use. The input to our preprocessor will have file extension .in. The output can be in various formats.

For fun, let's also make a python test program that calls the newly generated version.py:

Finally, we need to provide values for the variables. Let's put each one in its own file, named version and date, respectively.

version1.0

date1970-01-01

default.do files

Now we want to teach redo how to do our substitutions: how to, in general, generate file X from file X.in.

We could write a separate .do file for every output file X. For example, we might make test.txt.do and version.py.do. But that gets tedious. To make it easier, if there is no specific X.do for a target named X, redo will try using default.do instead. Let's write a default.do for our text preprocessor.

If default.do is asked to build X, and there exists a file named X.in, we use sed to do our variable substitutions. In that case, default.do uses redo-ifchange to depend on X.in, version, and date. If a file named X.in does not exist, then we don't know what to do, so we give an error.

On the other hand, if we try to generate a file that already exists, like test.py, redo does not call default.do at all. redo only tries to create files that don't exist, or that were previously generated by redo. This stops redo from accidentally overwriting your work.

$ redo test.txt
redo  test.txt

$ cat test.txt
This is the documentation for MyProgram version
1.0.  It was generated on 1970-01-01.

$ redo chicken
redo  chicken
default.do: Fatal: don't know how to build 'chicken'
redo  chicken (exit 99)

$ redo version.py
redo  version.py

# test.py was created by us, so it's a "source" file.
# redo does *not* call default.do to replace it.
$ redo test.py
redo: test.py: exists and not marked as generated; not redoing.

$ python test.py
Version '1.0' has build date '1970-01-01'

Nice!

While we're here, let's make an all.do so that we don't have to tell redo exactly which files to rebuild, every single time.

Results:

$ redo
redo  all
redo    test.txt
redo    version.py
redo    include/version.h

# input files didn't change, so nothing to rebuild
$ redo
redo  all

$ touch test.txt.in

$ redo
redo  all
redo    test.txt

Auto-generating the version and date (redo-always and redo-stamp)

Of course, in a real project, we won't want to hardcode the version number and date into a file. Ideally, we can get the version number from a version control system, like git, and we can use today's date.

To make that happen, we can replace the static version and date files with version.do and date.do. default.do already uses redo-ifchange to depend on version and date, so redo will create them as needed, and if they change, redo will rebuild all the targets that depend on them.

However, the version and date files are special: they depend on the environment outside redo itself. That is, there's no way to declare a dependency on the current date. We might generate the date file once, but tomorrow, there's no way for redo to know that its value should change.

To handle this situation, redo has the redo-always command. If we run redo-always from a .do file, it means every time someone depends on that target, it will be considered out-of-date and need to be rebuilt. The result looks like this:

$ redo 
redo  all
redo    test.txt
redo      version
redo      date
redo    version.py
redo    include/version.h

# version.do and date.do are redo-always, so
# everything depending on them needs to rebuild
# every time.
$ redo
redo  all
redo    test.txt
redo      version
redo      date
redo    version.py
redo    include/version.h

Of course, for many uses, that's overcompensating: the version number and date don't change that often, so we might end up doing a lot of unnecessary work on every build. To solve that, there's redo-stamp. redo-stamp does the opposite of redo-always: while redo-always makes things build more often, redo-stamp makes things build less often. Specifically, it lets a .do file provide a "stamp value" for its output; if that stamp value is the same as before, then the target should be considered unchanged after all.

The most common stamp value is just the content itself. Since in redo, we write the content to $3, we can also read it back from $3:

And the final result is what we want. Although version and date are generated every time, the targets which depend on them are not:

$ redo clean
redo  clean

# version and date are generated just once per run,
# the first time they are used.
$ redo
redo  all
redo    test.txt
redo      version
redo      date
redo    version.py
redo    include/version.h

# Here, (test.txt) means redo is considering building
# test.txt, but can't decide yet. In order to decide,
# it needs to first build date and version.  After
# that, it decides not to build test.txt after all.
$ redo
redo  all
redo    (test.txt)
redo    date
redo    version

Temporary overrides

Sometimes you want to override a file even if it is a target (ie. it has previously been built by redo and has a valid .do file associated with it). In our example, maybe you want to hardcode the version number because you're building a release. This is easy: redo notices whenever you overwrite a file from outside redo, and will avoid replacing that file until you subsequently delete it:

$ echo "1.0" >version

$ redo
redo  all
redo    (test.txt)
redo    date
redo: version - you modified it; skipping
redo    test.txt
redo    version.py
redo    include/version.h

$ redo
redo  all
redo    (test.txt)
redo    date
redo: version - you modified it; skipping

$ rm version

$ redo
redo  all
redo    (test.txt)
redo    date
redo    version
redo    test.txt
redo    version.py
redo    include/version.h

default.do, subdirectories, and redo-whichdo

There's one more thing we should mention, which is the interaction of default.do with files in subdirectories. Notice that we are building include/version.h in our example:

$ redo include/version.h
redo  include/version.h
redo    version
redo    date

$ cat include/version.h
// C/C++ header file identifying the current version
#ifndef __VERSION_H
#define __VERSION_H

#define VERSION "redo-0.31-3-g974eb9f"
#define DATE "2018-11-26"

#endif // __VERSION_H

redo works differently from the make command when you ask it to build files in subdirectories. In make's case, it always looks for a Makefile in the current directory, and uses that for all build instructions. So make include/version.h and cd include && make version.h are two different things; the first uses Makefile, and the second uses include/Makefile (or crashes if the latter does not exist).

redo, on the other hand, always uses the same formula to find a .do file for a particular target. For a file named X, that formula is as follows:

first, try X.do
then try default.do
then try ../default.do
then try ../../default.do
...and so on...

(Note: for targets with an extension, like X.o, redo actually tries even more .do files, like default.o.do and ../default.o.do. For precise details, read the redo man page.)

You can see which .do files redo considers for a given target by using the redo-whichdo command. If redo-whichdo returns successfully, the last name in the list is the .do file it finally decided to use.

$ redo-whichdo include/version.h
include/version.h.do
include/default.h.do
include/default.do
default.h.do
default.do

Redo always runs in the .do file's directory

To ensure consistency, redo always changes the current directory to the directory containing the selected .do file (not the directory containing the target, if they are different). As a result, redo include/version.h and cd include && redo version.h always have exactly the same effect:

$ redo include/version.h
redo  include/version.h
redo    version
redo    date

$ (cd include && redo version.h)
redo  version.h
redo    ../version
redo    ../date

(redo's display is slightly different between the two: it always shows the files it's building relative to the $PWD at the time you started redo.)

This feature is critical to redo's recursive nature; it's the reason that essays like Recursive Make Considered Harmful don't apply to redo. Any redo target, anywhere in your source tree, can use redo-ifchange to depend on any of your other targets, and the dependency will work right.

Why does redo change to the directory containing the .do file, instead of the directory containing the target? Because usually, the .do file needs to refer to other dependencies, and it's easier to always express those dependencies without adjusting any paths. In our text preprocessor example, default.do does redo-ifchange version date; this wouldn't work properly if it were running from the include/ directory, because there are no files named version and date in there.

Similarly, when compiling C programs, there are usually compiler options like -I../mylib/include. If we're compiling foo.o and mydir/bar.o, we would like -I../mylib/include to have the same meaning in both cases.