Writing portable makefiles

Edit 2016-10-21: I notice this post comes on first page of Google "portable makefile" request, so I thought I'd might add some context. This post was written when I was struggling with this kind of stuff, and should be taken as a "proof of concept" post. For me, this is definitely over, as I (almost) completely quit using Windows, being for several years now a happy GNU/Linux user. Readers must be aware that although some tricks are given here, it is certainly not the best approach for setting up a portable build system. If you are in that situation, the best way to go is probably CMake, as it is today the de facto standard tool.

This note is about GNU Make makefiles syntax, and how to write them to keep them OS-independent as much as possible.

1- Introduction: computers and file systems

When it comes to computers and their associated filesystem, there are two worlds on earth.
One that considers that a path to a file spells this way:
path/to/the/file
and the other world that considers that the correct syntax is:
path\to\the\file

This may sound silly (and it is) but it can lead to some complications. Not only because these two worlds use a different symbol, but mostly because they both have a special meaning for the other symbol.

To put it clearly, on a Linux machine (that uses the '/' path separator), the backslash has a special meaning in some situations (shell scripts, makefiles, ...) meaning "I have no more room on this line, lets keep on and continue the current command on next line" (and that trick is very valuable for readability). And it follows what is a convention in C and C++ source files.
 
In the other world (MS Windows), the default shell (cmd.exe) interprets the slash character as the option separator. For example: del /F path\to\file.txt

And, no, at least in XP, the Windows shell DOES NOT accept both path/to/file and path\to\file, as it is frequently said in many places. Try to do something like del path/to/file to check. Maybe this has changed with Windows 7, 8, 11 or 42, I'm not really interested, but with Windows XP's shell, it does-not-work.The cause of that misunderstanding is probably that system calls (that is, the functions you call from inside a program), DO accept forward slashes or backslashes in paths).

Anyway, the two shells (Linux/bash and Windows/cmd.exe) are sooo different, only insane people would consider trying to write a "compatible" script, running equally on both systems (1).

However, there is one situation where a same command semantic must be executed equally in those two different environments: makefiles

Basically, a makefile is a set of commands that are executed by the shell.
Say for one target, we want to erase some file, even "read-only" ones. On one environment, this must be done with the following command:
rm -f path/to/file
while on the other, it will be:
del /F path\to\file

The question is: how can a write that command in my makefile so that it expands in these two different syntaxes at runtime? And more generally, how do I write portable makefiles ?

2 - Handling command names

First, lets manage the different command names (and their options). That's the easiest. Just define a variable holding the name of the command, that will hold different values depending on platform. The easiest way to detect the platform is to check for the existence of a Windows-only environment variable, say ComSpec (but some sources relie on SystemRoot that can be used too).

ifdef ComSpec
    RM=del /F /Q
else
    RM=rm -f
endif

This will be in the upper part of the makefile, before any recipes. Then, in the recipes, just use $(RM) in place of the command.

3 - Handling paths

Secondly, you need to handle the path separator. Two situations need to be handled:
- paths to explicit files (the example above),
- automatic paths, build using make's wildcards and substitution functions.

Remember that you need to care for this only for system calls. Whatever the platform, GNU Make, gcc or other "regular" development tools handle very well paths with forward slashes, whatever the platform. To make it clear, say we have these lines that follow the classical target-prerequisite-command scheme:
mytarget: path/to/file
   $(SOMECOMMAND) path/to/file

The first line will do fine, but the second line will generate an error on Windows if SOMECOMMAND expands to a built-in shell command.

3.1 - Processing explicit paths

First, for the explicit paths, we can proceed with the same trick: define a variable holding the required separator ('\' or '/'), then use this variable in the commands.

ifdef ComSpec
    PATHSEP2=\
else
    PATHSEP2=/
endif

Ha. Unfortunatly, this does not work, because the backslash is interpreted by make as the "keep on same line!" request, and not as a character. Ok, so we need to escape that backslash, in order to fool make:

ifdef ComSpec
    PATHSEP2=\\
else
    PATHSEP2=/
endif

Funilly, this works half ways: the definition is accepted, but the variable holds the two backslashes! Fortunatly, the Windows shell accepts paths that looks like path\\to\\file (don't ask me why...)

Almost done. This still does not work: the above definition adds a trailing space at the end of the variable, i.e. its usage in:
path$(PATHSEP2)file
will expand into:
path/ file  (or  path\ file  on Windows)
and that will not be ok, for sure!

So finally, we need to add the following definition and function call, that removes that ugly trailing space:
PATHSEP=$(strip $(PATHSEP2))

That way, an explicit erasing command in a makefile (for example) can be portably written as:
$(RM) path$(PATHSEP)to$(PATHSEP)file

Ok, now how about paths that are automatically build.

3.2 - Processing generated paths

For example, you usually define a variable holding all the object files, that is build from all the source files. If these are in a folder named src, and the object files are in a folder named obj, then you can define the list of the source files with:
SRC_FILES=$(wildcard src/*.cpp)

and the list of corresponding object files with (2):
OBJ_FILES=$(patsubst src/%.cpp,obj/%.o,$(SRC_FILES))

But heres comes the problem, trying to erase all the object files with:
$(RM) $(OBJ_FILES)
will expand on Windows as something like:
del /F obj/file1.o obj/file2.o obj/file3.o
and that will throw an error, because the shell will consider that what is behind the slash as some option.

Two solutions can be used:

• either use PATHSEP in the "patsubst" function call above:

OBJ_FILES=$(patsubst src/%.cpp,obj$(PATHSEP)%.o,$(SRC_FILES))

• either use the "subst function", that replaces some pattern in a string with another:

OBJ_FILES_CORRECT=$(subst \,/,$(OBJ_FILES))

But this latter solution implies the creation of another variable, which can be error-prone in dense makefiles.

4 - Command separator

Another problem that needs to be handled is the command separator. In many make tutorials, you see commands written this way:
cd MyFolder; SomeCommand Some Arguments
which means: "get down into folder MyFolder, and execute SomeCommand with Some Arguments"
This is an invalid syntax on Windows where the command separator is &.
So, again, the variable trick:

ifndef ComSpec
    CMDSEP=;
else
    CMDSEP=&
endif


and the above command will be written:
cd MyFolder $(CMDSEP) SomeCommand Some Arguments

5 - Debugging makefiles

Portable makefiles are also tougher to debug that ordinary makefile. You will run into countless issues, and each one might require some special treatment.

As you may know, you can prefix each make command with the special symbol '@', that will suppress the default echo to terminal. This is useful for the makefile user, that doesn't want to see all the steps of the build process: he just wants to get the job done, as quickly as possible.
But while writing the makefile (an debugging it), you will need all this information, so you don't use that prefix, of course...
   ... until you're done ! Then you want to deliver a nice experience to the makefile user, and you carefully edit your code to prefix each command with '@'
And then... Ah. Something got broken in the makefile when this new build step was added, but where exactly ? So you start again removing all these stupid '@' characters that you spend so much time to add!

Don't. Instead use again the good'ol variable trick. Instead of using the special character '@', prefix right away all of your commands with $(L). For example:
obj/%.o : src/%.cpp
    $(L)$(CXX) -o $@ -c $< $(CFLAGS)


And add the following lines in the first part of the makefile:
ifeq "$(LOG)" ""
    LOG=no
endif

ifeq "$(LOG)" "no"
    L=@
endif


This way, launching make with no special option will run silently, and in case of trouble, just tell your mate to run:
make <target-name> LOG=yes
and all the commands that are launched will (magically) appear on screen (3).

6 - Conclusion

These are some hints that can help you design more portable makefiles. I'll finish with one remark. For "big" projects, maybe you should rely on "makefile generators", that is, programs that do all these low-level tasks (and much more). The most known are CMake and the GNU set of tools, but others can be used.

Finally, one quote from "Managing projects with GNU Make": "... there is no such thing as perfect portability, so it is our job to balance effort versus portability."


(1) Unless you use a non-native modern script langage such as Python of course.
(2) In real life, the folders name would also be stored in variables, i.e. :
OBJ_FILES=$(patsubst $(SRC_DIR)/%.cpp,$(OBJ_DIR)/%.o,$(SRC_FILES))
(3) Or in a text file if you redirect it, and that is usually a good idea when output starts to get large.

GNU Make and the foreach() function

Make is a nice tool for building stuff. Software, of course, but not only, I use it also for building pdf files from Latex sources. However, its usage is not obvious at all, and many pitfalls lie in front of the newcomer. And the manual is not a great help when it comes to learning new features. Joyfully, there are bunches of tutorials out there.

However, some advanced tricks are rarely covered. Here is one trick that can be useful in some situations. Beware, it assumes you already know the basics about Make (GNU flavor), in case not, please go read some tutorial and come back in a while...

Lets say you have a folder containing some source files of type ".in", say foo.in and boo.in. These files need to be processed with some tool (call it "some_tool" for now) to build the output files. This (awesome) thing has a rather classical syntax:

some_tool in_file out_file [option]

Say you need to build these with two different configurations to produce two output files for each input files, say one of type "X" and one of type "Y". Here, from the two input files, you need to build:
foo_X.out
foo_Y.out
boo_X.out
boo_Y.out

How can Make help you so you only build the ones that are needed ? For the example, say that each build process takes several hours, so you really don't want to build them if it's not needed.

First create a variable holding all the input files:

infiles := $(wildcard *.in) 

Then, add a generic rule telling how to build a ".out" file:

%.out: %.in 
    @some_tool $< $@ 

Yes, remember these special variables: $< is the first dependency, and $@ is the target (%.out here), so that will generate the correct command-line. Oh, and for the different output flavors (say, by adding some option), just dupe the generic rule:

%_X.out: %.in
 @some_tool $< $@ X
%_Y.out: %.in
 @some_tool $< $@ Y

Finally, you will try to build the main (all) rule. Its dependencies are all the input files (thats easy, its $(infiles) ) and the output files, so that each of them gets produced only in case it is not present. Ah. Here comes some trouble. How do you create a variable holding all the expected targets ? Well, first, lets get the files names without extension. Easy:

filenames :=$(basename $(infiles))

Okay, and now, how do I automatically generate a string holding all the output files names ? This is a job for the foreach() function. Its description in the manual is (IMHO) mostly unreadable, so lets see this through a simpler example:

what = $(foreach a, ham cheese salad, john likes $(a) )
all:
    @echo "what=$(what)"

produces:

what= john likes ham  john likes cheese  john likes salad

Usually, this is used using some variables, and it would be written:

types= ham cheese salad
what = $(foreach a, $(types), john likes $(a) )

Mmmh, I'm sure you're starting to see the point. The 'types' will be our different output flavors, say, something like:

types= X Y
outfile := $(foreach a, $(types), boo_$(a).out )
     @echo "outfile=$(outfile)"

that will produce:

outfile= boo_X.out boo_Y.out

But this has to be done for each input files, and the following fails:

types= X Y
outfiles := $(foreach a, $(types), $(filenames)_$(a).out )

It gets expanded to:

boo foo_X.out boo foo_Y.out

So the solution is to use nested foreach() loops, just as you would do with some regular procedural programming language.

types:=X Y
outfiles := $(foreach a,$(filenames), \
     $(foreach b,$(types), \
         $(a)_$(b).out ) ) 

Then, the main rule can just get written as follows:

all: $(infiles) $(outfiles)
    @echo "That's all, folks !"

For further reading, check this series of articles on advanced topics on Make.