GNU Make and the foreach() function

Make is a nice tool for building stuff. Software, of course, but not only, I use it also for building pdf files from Latex sources. However, its usage is not obvious at all, and many pitfalls lie in front of the newcomer. And the manual is not a great help when it comes to learning new features. Joyfully, there are bunches of tutorials out there.

However, some advanced tricks are rarely covered. Here is one trick that can be useful in some situations. Beware, it assumes you already know the basics about Make (GNU flavor), in case not, please go read some tutorial and come back in a while...

Lets say you have a folder containing some source files of type ".in", say foo.in and boo.in. These files need to be processed with some tool (call it "some_tool" for now) to build the output files. This (awesome) thing has a rather classical syntax:

some_tool in_file out_file [option]

Say you need to build these with two different configurations to produce two output files for each input files, say one of type "X" and one of type "Y". Here, from the two input files, you need to build:
foo_X.out
foo_Y.out
boo_X.out
boo_Y.out

How can Make help you so you only build the ones that are needed ? For the example, say that each build process takes several hours, so you really don't want to build them if it's not needed.

First create a variable holding all the input files:

infiles := $(wildcard *.in) 

Then, add a generic rule telling how to build a ".out" file:

%.out: %.in 
    @some_tool $< $@ 

Yes, remember these special variables: $< is the first dependency, and $@ is the target (%.out here), so that will generate the correct command-line. Oh, and for the different output flavors (say, by adding some option), just dupe the generic rule:

%_X.out: %.in
 @some_tool $< $@ X
%_Y.out: %.in
 @some_tool $< $@ Y

Finally, you will try to build the main (all) rule. Its dependencies are all the input files (thats easy, its $(infiles) ) and the output files, so that each of them gets produced only in case it is not present. Ah. Here comes some trouble. How do you create a variable holding all the expected targets ? Well, first, lets get the files names without extension. Easy:

filenames :=$(basename $(infiles))

Okay, and now, how do I automatically generate a string holding all the output files names ? This is a job for the foreach() function. Its description in the manual is (IMHO) mostly unreadable, so lets see this through a simpler example:

what = $(foreach a, ham cheese salad, john likes $(a) )
all:
    @echo "what=$(what)"

produces:

what= john likes ham  john likes cheese  john likes salad

Usually, this is used using some variables, and it would be written:

types= ham cheese salad
what = $(foreach a, $(types), john likes $(a) )

Mmmh, I'm sure you're starting to see the point. The 'types' will be our different output flavors, say, something like:

types= X Y
outfile := $(foreach a, $(types), boo_$(a).out )
     @echo "outfile=$(outfile)"

that will produce:

outfile= boo_X.out boo_Y.out

But this has to be done for each input files, and the following fails:

types= X Y
outfiles := $(foreach a, $(types), $(filenames)_$(a).out )

It gets expanded to:

boo foo_X.out boo foo_Y.out

So the solution is to use nested foreach() loops, just as you would do with some regular procedural programming language.

types:=X Y
outfiles := $(foreach a,$(filenames), \
     $(foreach b,$(types), \
         $(a)_$(b).out ) ) 

Then, the main rule can just get written as follows:

all: $(infiles) $(outfiles)
    @echo "That's all, folks !"

For further reading, check this series of articles on advanced topics on Make.