Friday, December 12, 2014

awful, but useful, Python

Have you ever wanted to open and decompress all log files, streaming each line to the caller?

In Python, this is:

import bz2, sys
from itertools import chain

# set 'lines' to all lines of all files after they've been decompressed
lines = chain(*(bz2.BZ2File(path) for path in sys.argv[1:]))

# as a check, print the first 100 lines, then crash
lines = islice(lines, 100)
print ''.join(lines)

Monday, November 24, 2014

speeding up tests with Postgres

Dear Lazyweb,

I'm developing a site using Django and Postgres. How can I get my tests to run a lot faster?

Answer: tell Postgres to use a tablespace in RAM. Selects, updates, and inserts will run at fast RAM speed vs slow disk speed.

Caveat: this doesn't work for writes. Postgres is designed to keep data reliable at all times. Any INSERT/UPDATE/DELETE gets written to a "write-ahead log" (WAL), so that if the database crashed it'll restore the data.  Since this is on a disk no matter what the tablespace, the above trick doesn't work without defeating the WAL.

Answer2: use UNLOGGED tables, TBD.


Unlogged table performance in postgresql 9.1: (2011) "unlogged tables have shown an increase of output by 13~17%"; includes performance-oriented Postgres settings.

WAITING FOR 9.1 – UNLOGGED TABLES (2011): "that's really fast"

wal_buffers performance by Robert Haas (2012): pretty graphs

Monday, November 3, 2014

talk Thursday: Networking Tricks with the Linux Kernel

If you're in LA, come see me Thursday at the Unix Users meeting!

Everyone knows about networking. The Linux kernel provides a wide range of networking and other services, most of which are not generally known. In fact, a lot of what "everyone knows" is wrong or is lacking in detail. In this talk I'll give a technical survey of what's available and how to use it to accomplish magic tricks. With the right knowledge,our software can do great things without much code!

Monday, October 27, 2014

Programming vs Programmer

Although I have many years of professional programming experience, it's still startling to realize the distance between writing a program for "Hello World", vs working as a day-to-day programmer. The amount of stuff you have to know, and use quickly in ever-changing combinations, is substantial.

A student decried that his two-month class in Ruby didn't really prepare him for being a big-P Programmer, it just taught him little-p programming.

My view is that the material in that class, as well as Computer Science in general, is meaningless. What you learn is not useful, it's all about learning how to learn.

The following article shows how to do professional-level deployment of your Python webapp onto a server.  It uses ten different technologies, only one of which is "Python"!  This is all after your webapp is programmed and complete.  The pieces are: 1) version control with Git, 2) Python, 3) separate environments with Virtualenv, 4) installing packages with Pip, 5) using Flask for the webapp itself, 6) running webapps with Gunicorn, 7) handling web clients with Nginx, 8) keeping services running with Supervisor, deploying with 9) Fabric (TLDR: don't use this), and 10) off by one errors.

Most of the above technologies are complex, and have multiple books written about them. Using them together requires a firm grasp not on each tech itself, but in which parts to ignore, and which to use, in combination with which other technologies.

This is a hard skill to learn, but this characteristic separates little-p programmers from professional Programmers.

Kickstarting Flask on Ubuntu - setup and deployment - Real Python

Wednesday, October 22, 2014

Why I prefer Python to Java, in two code samples.

Why I prefer Python to Java, in two code samples.


Vagrant/Ansible trick: autoreload

When testing Ansible scripts, it's nice to have your Vagrant VM automatically re-provision itself when you change the configuration.  In this way just saving a file lets you see the VM reinstall packages and so forth, with errors appearing immediately.

The following runs a tiny reprovisioning script each time your Ansible YAML files change:

pip install watchmedo

watchmedo shell-command --drop --pattern='*.yml' --command='clear; vagrant provision'