A LaTeX typesetting example
LaTeX is a typesetting system that's especially popular in academia. Among other things, it lets you produce postscript and pdf files from a set of (mostly text) input files.
LaTeX documents often include images and charts. In our example, we'll show how to auto-generate a chart for inclusion using an R script with ggplot2.
To play with this code on your own machine, get the redo
source code and look in the
Generating a plot from an R script
First, let's tell redo how to generate our chart. We'll use the R language, and ask it to plot some of its sample data (the mpg, "miles per gallon" data set) and save it to an eps (encapsulated postscript) file. eps files are usually a good format for LaTeX embedded images, because they scale to any printer or display resolution.
First, let's make an R script that generates a plot:
And then a .do file to tie that into redo:
We can build and view the image:
$ redo mpg.eps redo mpg.eps # View the file on Linux $ evince mpg.eps # View the file on MacOS $ open mpg.eps
Running the LaTeX processor
Here's the first draft of our very important scientific paper:
Notice how it refers to the chart from above,
mpg.eps, and a text file,
discovery.txt. Let's create the latter as a static file.
With all the parts of our document in places, we can now compile it directly
$ pdflatex paper.latex This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016/Debian) (preloaded format=pdflatex) restricted \write18 enabled. entering extended mode ...[a lot of unnecessary diagnostic messages]... Output written on paper.pdf (2 pages, 68257 bytes). Transcript written on paper.log.
But this has a few problems. First of all, it doesn't understand
mpg.R changes, it won't know to rebuild
Secondly, the TeX/LaTeX toolchain has an idiosyncracy that means you might
have to rebuild your document more than once. In our example, we generate a
table of contents, but it ends up getting generated before processing the
rest of the content in the document, so it's initially blank. As it
continues, LaTeX produces a file called
paper.aux with a list of the
references needed by the table of contents, and their page numbers. If we
run LaTeX over again, it'll use that to build a proper of table of contents.
Of course, life is not necessarily so easy. Once the table of contents
isn't blank, it might start to push content onto the next page. This will
change all the page numbers! So we'd have to do it one more time. And that
might lead to even more subtle problems, like a reference to page 99
changing to page 100, which pushes a word onto the next page, which changes
some other page number, and so on. Thus, we need a script that will keep
looping, re-running LaTeX until
The whole script we'll use is below. Instead of running
directly, we'll use the regular
latex command, which produces a .dvi
(DeVice Independent) intermediate file which we can later turn into a pdf or
LaTeX produces a bunch of clutter files (like
paper.aux) that can be used
in future runs, but which also make its execution nondeterministic. To
avoid that problem, we tell it to use a temporary
we delete and recreate before each build (although we might need to run
latex multiple times in one build, to get
paper.aux to converge).
Virtual targets, side effects, and multiple outputs
Why did we call our script
default.runtex.do? Why not
default.dvi.do, depending what kind of file we ask LaTeX to produce?
The problem is that the
latex command actually produces several
files in that temporary directory, and we might want to keep them around.
If we name our .do file after only one of those outputs, things get messy.
The biggest problem is that redo requires a .do file to write its output to
$3 (or stdout), so that it can guarantee the output gets replaced
atomically. When there is more than one output, at most one file can
be sent to $3; how do you choose which one? Even worse, some programs don't
even have the ability to choose the output filename; for an input of
latex command just writes a bunch of files named
paper.* directly. You can't ask it to put just one of them in $3.
The easiest way to handle this situation in redo is to use a "virtual
target", which is a target name that doesn't actually get created has a file,
and has only side effects. You've seen these before: when we use
clean.do, we don't expect to produce a file named
expect redo to run a collection of other commands. In
make, these are
sometimes called ".PHONY rules" because of the way they are declared in a
Makefile. But the rules aren't phony, they really are executed; they just
don't produce output. So in redo we call them "virtual."
redo paper.runtex, it builds our virtual target. There is no
paper.runtex file or directory generated. But as a side effect, a
paper.tmp is created.
(Side note: it's tempting to name the directory the same as the target. So
we could have a
paper.runtex directory instead of
paper.tmp. This is
not inherently a bad idea, but currently redo behaviour is undefined if you
redo-ifchange a directory. Directories are weird. If one file in that
directory disappears, does that mean you "modified" the output by hand?
What if two redo targets modify the same directory? Should we require
scripts to only atomically replace an entire output directory via $3? And
so on. We might carefully define this behaviour eventually, but for now,
it's better to use a separate directory name and avoid the undefined
Depending on side effects produced by virtual targets
Next, we want to produce .pdf and .ps files from the collection of files
produced by the
latex command, particularly
paper.tmp/paper.dvi. To do
that, we have to bring our files back from the "virtual target" world into
the real world.
Depending on virtual targets is easy; we'll just
redo-ifchange paper.runtex. Then we want to materialize
the temporary files in
paper.tmp/paper.dvi, which we can do with an
efficient hardlink (rather than
making an unnecessary copy), like this:
Notice that we don't do
redo-ifchange paper.tmp/paper.dvi. That's
because redo has no knowledge of that file. If you ask redo to build that
file for you, it doesn't know how to do it. You have to ask for
paper.runtex, which you know - but redo doesn't know - will produce the
input file you want. Then you can safely use it.
Once we have a .do file that produces the "real" (non-virtual,
paper.dvi file, however, it's safe to depend directly on
it. Let's use that to produce our .ps and .pdf outputs:
(As above, we include
exec >&2 lines because LaTeX tools incorrectly write
their log messages to stdout. We need to redirect it all to stderr. That
way redo-log can handle all the log output appropriately.)
We've made a generalized script,
default.runtex.do, that can compile any
.latex file and produce a .tmp directory with its output. But that's not
quite enough: different .latex files might have extra dependencies that need
to exist before the compilation can continue. In our case, we need the
mpg.eps that we discussed above.
To make that work,
default.runtex.do looks for a .deps file with the same
name as the .latex file being processed. It contains just a list of extra
dependencies that need to be built. Here's ours:
You can use this same ".deps" technique in various different places in redo.
For example, you could have a default.do that can link a C program from any
set of .o files. To specify the right set of .o files for target
default.do might look in an
X.list file. If you later want to
get even fancier, you could make an
X.deps.do that programmatically
generates the list of dependencies; for example, it might include one set of
files on win32 platforms and a different set on unix platforms.
paper.latex file actually includes two files:
mpg.eps, which we
explicitly depended upon above, and
discovery.txt, which we didn't. The
latter is a static source file, so we can let redo discover it
automatically, based on the set of files that LaTeX opens while it runs.
latex command has a
--record option to do this; it produces a file
paper.tmp/paper.fls (.fls is short for "File LiSt").
One of redo's best features is that you can declare dependencies after
you've done your build steps, when you have the best knowledge of which
files were actually needed. That's why in
default.runtex.do, we parse the
.fls file and then redo-ifchange on its contents right at the end.
(This brings up a rather subtle point about how redo works. When you run redo-ifchange, redo adds to the list of files which, if they change, mean your target needs to be rebuilt. But unlike make, redo will not actually rebuild those files merely because they're listed as a dependency; it just knows to rebuild your target, which means to run your .do file, which will run redo-ifchange again if it still needs those input files to be fresh.
This avoids an annoying problem in
make where you can teach it about
which .h files your C program depended on last time, but if you change
A.c to no longer include X.h, and then delete X.h, make might complain
that X.h is missing, because A.c depended on it last time. redo will
simply notice that since X.h is missing, A.c needs to be recompiled, and let
your compilation .do script report an error, or not.)
Anyway, this feature catches not just our
discovery.txt dependency, but
also the implicit dependencies on various LaTeX template and font files, and
so on. If any of those change, our LaTeX file needs to be rebuilt.
$ redo --no-detail paper.pdf redo paper.pdf redo paper.dvi redo paper.runtex redo mpg.eps $ redo --no-detail paper.pdf redo paper.pdf $ touch discovery.txt $ redo --no-detail paper.pdf redo paper.pdf redo paper.dvi redo paper.runtex $ redo --no-detail paper.pdf redo paper.pdf
As usual, to polish up our project, let's create an
Because this project is included in the redo source and we don't want redo
to fail to build just because you don't have LaTeX or R installed, we'll
all.do quit politely if the necessary tools are missing.