Message Queuing slides and source code!

I gave a talk at the local Python user group, SoCal Piggies. We discussed message queues: they're useful for synchronizing processes, increasing reliability and scalability. The entire source tree, with slides and extra goodies, is linked below. Enjoy!



https://bitbucket.org/shavenwarthog/amqp-ex/src/

(diagram credit: RedHat)

Superdevelopers at Facebook

Facebook gets unprecedented traffic. They don't want things to go down so they... release changes constantly, a few times a day! They use peer review and their immense audience to see how well their solutions work, then iterate. The other teams *support* engineering, rather than being gatekeepers.

I actually did this at a medium-sized site many years ago. Push change to live, capture logs for five minutes while people beat it, then undo the change. It worked well.

ntop and RRD

"ntop" conveniently displays web pages with charts and plots of your local network traffic. To record historical traffic, it uses Round Robin Database (rrd). The default ntop package enables the RRD plugin, but doesn't give itself permission to write out its data. Here's how to get the two to work:

sudo apt-get install ntop
sudo install -d /var/lib/ntop
sudo install -o nobody -d /var/lib/ntop/interfaces
sudo install -o nobody -d /var/lib/ntop/rrd

To test, start "sudo ntop" (probably in "screen"), then open the web page http://localhost:3000

To activate the RRD plugin, select Plugins / Round Robin / Describe, then click the red "no" to become a blue "yes." You'll see messages in the terminal where you started ntop.

After a few moments, you can view RRD graphs. One is in Summary / Network Load.

Fast Parallel Downloading (for apt-get and aptitude) #3

"aptitude" has more features, but is as low-hassle and scriptable as "apt-get". For instance, you can say "this package was automatically installed", which means if you don't need it any longer, it'll be automatically removed. Another perk is a direct way of asking "why was/wasn't this package installed?", to help sort out sometimes confusing package dependencies.

Alas, it doesn't have the "print-uris" trick which I relied upon in doing quick upgrades. I've found a simple workaround: use both aptitude and apt-get. The following script updates the list of available packages. It uses apt-get to download -- but not install -- all updates. Lastly, it uses aptitude to upgrade the packages, using the freshly-downloaded files. A "safe" upgrade won't delete a package to let another package be installed correctly, it just does installs.

For a one-line version, see morch's a previous comment.

cd /var/cache/apt/archives/
aptitude update
apt-get -y --print-uris -u upgrade | grep -o -e "http://[^\']+" \
| xargs -r -l3 -P5 wget -nv
aptitude -y safe-upgrade

Bits of Evidence

Slideshow adding Science to "Computer Science". Includes references to a lot of myths like "best programmers are 45x better than the worst ones."

Bits of Evidence

View more presentations from Greg Wilson.

turning batch MapReduce into infinitely streaming data wrangling

Pipelining and Real-time Analytics with MapReduce Online

Comparison of Scrum, XP, Kanban, and other Agile Methodologies

State of Agile