practical use of GNU Makefile for testing

Recently I reviewed a bunch of code from students. The projects are in Java. To test each one, we:

1. compile Java source file into class file
2. run the class file
3. examine output

Over time I'll be getting more code, thus I want a simple workflow that will automatically adapt to having more source in the same directory. I can't just list all the source files then write out a Bash script to compile + run everything, for example. As new code comes in, I want it to be included in the overall test run.

How would you accomplish this?

GNU Make is the bomb

I used my good old friend, GNU Make! This simple tool runs programs to turn files into other files. It understands dependencies, and won't do work unless it needs to. For example, it'll compile C files into objects, then at the end will link all the objects into a single executable. The input language has rules like "to make X into Y, run program Z". You say "make Y", the system will automatically go find file X, run program Z on it, giving you the result.  If the file Y already exists, it won't re-make it, and will exit.

Make understands dependencies. For example if it knows Y depends on X, and X has been edited after Y was created, Make understands that it has to re-build Y given the updated source X.

I often use Make for holding lots of tiny 1-2 line scripts, example make backup will copy all my important files into a compressed, timestamped tarball for archival purposes. I tell Make that source code is important, but object and executable files are not -- thus my archive is small but contains all my valuable work.

Simple example: use Make to build a Java class file

Create this file, name it "Makefile" in the current directory. NOTE: the second line has a tab character -- don't use spaces!  Make is ornery in this regard.

%.class: %.java
javac $<

The % character means "anything", and $< means "dependent file".  The above stanza says "to make a class file from a java file, run the "javac" command on the dependent Java file".

To create a class file from Java source, type make myfile.class (if your source is in myfile.java)

Sample run:

$ make JenExercise2.class

javac JenExercise2.java

Make automatically outputs each command as it's running it, thus you can see example what Make is doing.  For example, after the class file is built, Make doesn't need to, err, make it again. If we re-run the above command, the class file will not be rebuilt.

$ make JenExercise2.class

make: 'JenExercise2.class' is up to date.

Use Make to compile and test a single program

In my case, I want to type one command and have all my programs compiled and run so I can see everyone's output all at once. For this to happen we'll make all the class files like above. Then, for each class file we'll run it to get the output.

How do we do this?

We use a pseudo suffix. The java files and class files exist on disk, and can be manipulated directly. However to say "run test on file X", the output just goes to the screen, there's nothing stored.  To resolve this I tell Make: "hey, when I ask for X.test, build the X.class file if needed, then run it with the java command".  Make is happy and runs my commands. Since the "X.test" file isn't created, when I ask Make to do it again, it'll just re-run the commands. When we make X.class files, they live on disk, so Make won't rebuild them.  Test files don't exist -- ".test" is a pseudo-suffix -- thus Make will always run my testing commands, which is what we want.

Here's the Makefile which allows compiling and testing of Java programs:

%.class: %.java
javac $<

%.test: %.class
java $(subst .class,,$<)

The "$(subst...)" bit says "run the Makefile function subst to do string substitution". In this case, it uses the dependent file ("$<"), searches it for the text ".class", then substitutes it with the empty string.  That is, it converts "beer.class" into "beer". We use this so that when we specify "make beer.test", it'll first build the class file beer.class, then run it, stripping the ".class" to run the command "java beer". Our test runs.

And a sample run:

$ make JenExercise2.test
javac JenExercise2.java
java JenExercise2

<a href=github.com>Github</a>

The bit after "java JenExercise2" is the program's output -- we can now type "make x.test" to automatically compile and run our program!

I no longer specify "make the class file", Make is smart enough to build it if necessary.  As you use Make more, it gets smarter, so it automatically does what you want!

Real example: use Make to test all Java programs

Most Makefiles are not much more complex that the example above.  However I'm a wiseass and I want a single command to run all the tests. I don't want to manually know the file names.  So, I'll define a Make variable to figure all the test names for me! To do this we'll use lots of Makefile functions.

Here's the first:

zoot:

echo $(wildcard *.java)

$ make zoot

echo JenExercise2.java

JenExercise2.java

Make runs the function "wildcard" to find all Java files in the current directory.  It then runs the "zoot" command (which we're using for testing the Makefile), running a shell command to echo the function result to the screen. This is the easiest (and silliest) way to test Makefile functions.

Okay, we have a list of the Java files. We want the list of tests. That is, given "beer.java", we want "beer.test".  Here's our second testing Makefile:

zoot:

echo $(patsubst %.java,%.test,$(wildcard *.java))

$ make zoot

echo JenExercise2.test

JenExercise2.test

It worked!

Now, we want to tell Make that when we type "make" with no arguments, it'll go find out which tests should be run, then run them.  To do that, we'll define a simple named verb in the top of our Makefile:

test: $(patsubst %.java,%.test,$(wildcard *.java))

This says "generate a list of tests from the list of Java files.  The "test" command depends on running all the tests". Thus when we type "make test", or just "make", it'll compile all our programs, and run them!

Whole Makefile:

test: $(patsubst %.java,%.test,$(wildcard *.java))

%.class: %.java

javac $<

%.test: %.class

java $(subst .class,,$<)

clean:

-$(RM) *.class

Output:

$ make test

javac JenExercise2.java

java JenExercise2

<a href=github.com>Github</a>

We're done!

( The "clean" verb is an example of my little 1-2 line helper scripts.  It zaps all class files, thus cleaning my directory up. )