Friday, June 30, 2017

workshop: Zero to Webapp in Python

(This page is

Slides: Google Slides

Workflowy outline

GitHub source code

Web development is amazingly complicated. Can we get away with not learning that much?  John will do an interactive workshop on how to start with nothing and wind up with a working web app!


Students will learn how to write a tiny webapp without their brains exploding. NON-GOAL: we'll simplify the stack to exclude many things required for real projects. We won't cover classes, many types, modules, the debugger, nor testing. We might get to packages, virtual environments, and 3rd party modules like Flask. The intent of this workshop is to get something working with a minimum of magic.


* Python!
* HTML+CSS (a tiny amount)
* Bash shell (a little)

The goal is to deliver a simple working webapp, then refine it with small changes, each time understanding _why_. Students  become much more comfortable and confident with Python, the console, and web technologies in general.

Monday, June 26, 2017

John at work

I'm updating a class for Saturday, "Zero to Webapp in Three Hours", for non- or intro programmers. I just realized that all of the students have already learned most of the material!  So here's me updating the class:

re: hyperproductive development and debugging microservices

Story time.
At one company I've worked with, two guys knew the code completely cold. The other 5 devs only worked on bits and pieces, so it was difficult for them to have substantial discussions let alone valuable changes to the codebase. Visibility was further limited by there being no "end to end, production-like" stack. Each dev would run one component, developing and testing it always in isolation from the other components.
When I Am King(tm), all devs (and qa) will work on independent, production-like environments. They can beat the shit out of the entire system and get high-quality feedback on changes before it lands in production, where visibility is more awkward, and there are consequences.
For those people working with services, I mean microservices: check out Linkerd from the Cloud Native Computing Foundation (they host Kubernetes, Prometheus, and other valuable software). Linkerd allows services to be assembled from other services, with additional requirements. For example in production let's say service A calls service B. In environment "Staging", it would be configured to use (production) service A, but instead of A calling B, it would call the new version, B'. This allows "services" to be stitched up together easily. Another advantage is that B can be wrapped by another service C, allowing C to emit before/after messages and enabling further tracing. This allows service components to be safely debugged, even in production! This "remapping" feature can even be done on a per-request basis, allowing you to route and test services on the fly! Check it out.

Monday, June 12, 2017

practical use of GNU Makefile for testing

Recently I reviewed a bunch of code from students. The projects are in Java. To test each one, we:

  1. compile Java source file into class file
  2. run the class file
  3. examine output

Over time I'll be getting more code, thus I want a simple workflow that will automatically adapt to having more source in the same directory. I can't just list all the source files then write out a Bash script to compile + run everything, for example. As new code comes in, I want it to be included in the overall test run.

How would you accomplish this?

GNU Make is the bomb

I used my good old friend, GNU Make! This simple tool runs programs to turn files into other files. It understands dependencies, and won't do work unless it needs to. For example, it'll compile C files into objects, then at the end will link all the objects into a single executable. The input language has rules like "to make X into Y, run program Z". You say "make Y", the system will automatically go find file X, run program Z on it, giving you the result.  If the file Y already exists, it won't re-make it, and will exit.

Make understands dependencies. For example if it knows Y depends on X, and X has been edited after Y was created, Make understands that it has to re-build Y given the updated source X.

I often use Make for holding lots of tiny 1-2 line scripts, example make backup will copy all my important files into a compressed, timestamped tarball for archival purposes. I tell Make that source code is important, but object and executable files are not -- thus my archive is small but contains all my valuable work.

Simple example: use Make to build a Java class file

Create this file, name it "Makefile" in the current directory. NOTE: the second line has a tab character -- don't use spaces!  Make is ornery in this regard.

javac $<

The % character means "anything", and $< means "dependent file".  The above stanza says "to make a class file from a java file, run the "javac" command on the dependent Java file".

To create a class file from Java source, type make myfile.class (if your source is in

Sample run:

$ make JenExercise2.class

Make automatically outputs each command as it's running it, thus you can see example what Make is doing.  For example, after the class file is built, Make doesn't need to, err, make it again. If we re-run the above command, the class file will not be rebuilt.

$ make JenExercise2.class
make: 'JenExercise2.class' is up to date.

Use Make to compile and test a single program

In my case, I want to type one command and have all my programs compiled and run so I can see everyone's output all at once. For this to happen we'll make all the class files like above. Then, for each class file we'll run it to get the output.

How do we do this?

We use a pseudo suffix. The java files and class files exist on disk, and can be manipulated directly. However to say "run test on file X", the output just goes to the screen, there's nothing stored.  To resolve this I tell Make: "hey, when I ask for X.test, build the X.class file if needed, then run it with the java command".  Make is happy and runs my commands. Since the "X.test" file isn't created, when I ask Make to do it again, it'll just re-run the commands. When we make X.class files, they live on disk, so Make won't rebuild them.  Test files don't exist -- ".test" is a pseudo-suffix -- thus Make will always run my testing commands, which is what we want.

Here's the Makefile which allows compiling and testing of Java programs:

javac $<

%.test: %.class
java $(subst .class,,$<)

The "$(subst...)" bit says "run the Makefile function subst to do string substitution". In this case, it uses the dependent file ("$<"), searches it for the text ".class", then substitutes it with the empty string.  That is, it converts "beer.class" into "beer". We use this so that when we specify "make beer.test", it'll first build the class file beer.class, then run it, stripping the ".class" to run the command "java beer". Our test runs.

And a sample run:

$ make JenExercise2.test
java JenExercise2


The bit after "java JenExercise2" is the program's output -- we can now type "make x.test" to automatically compile and run our program!

I no longer specify "make the class file", Make is smart enough to build it if necessary.  As you use Make more, it gets smarter, so it automatically does what you want!

Real example: use Make to test all Java programs

Most Makefiles are not much more complex that the example above.  However I'm a wiseass and I want a single command to run all the tests. I don't want to manually know the file names.  So, I'll define a Make variable to figure all the test names for me! To do this we'll use lots of Makefile functions.

Here's the first:

echo $(wildcard *.java)

$ make zoot

Make runs the function "wildcard" to find all Java files in the current directory.  It then runs the "zoot" command (which we're using for testing the Makefile), running a shell command to echo the function result to the screen. This is the easiest (and silliest) way to test Makefile functions.

Okay, we have a list of the Java files. We want the list of tests. That is, given "", we want "beer.test".  Here's our second testing Makefile:

echo $(patsubst,%.test,$(wildcard *.java))

$ make zoot
echo JenExercise2.test

It worked!

Now, we want to tell Make that when we type "make" with no arguments, it'll go find out which tests should be run, then run them.  To do that, we'll define a simple named verb in the top of our Makefile:

test: $(patsubst,%.test,$(wildcard *.java))

This says "generate a list of tests from the list of Java files.  The "test" command depends on running all the tests". Thus when we type "make test", or just "make", it'll compile all our programs, and run them!

Whole Makefile:

test: $(patsubst,%.test,$(wildcard *.java))

javac $<

%.test: %.class
java $(subst .class,,$<)

-$(RM) *.class


$ make test
java JenExercise2

We're done!

( The "clean" verb is an example of my little 1-2 line helper scripts.  It zaps all class files, thus cleaning my directory up. )