Monday, November 24, 2014

speeding up tests with Postgres

Dear Lazyweb,

I'm developing a site using Django and Postgres. How can I get my tests to run a lot faster?

Answer: tell Postgres to use a tablespace in RAM. Selects, updates, and inserts will run at fast RAM speed vs slow disk speed.

Caveat: this doesn't work for writes. Postgres is designed to keep data reliable at all times. Any INSERT/UPDATE/DELETE gets written to a "write-ahead log" (WAL), so that if the database crashed it'll restore the data.  Since this is on a disk no matter what the tablespace, the above trick doesn't work without defeating the WAL.

Answer2: use UNLOGGED tables, TBD.

Reference:

Unlogged table performance in postgresql 9.1: (2011) "unlogged tables have shown an increase of output by 13~17%"; includes performance-oriented Postgres settings.

WAITING FOR 9.1 – UNLOGGED TABLES (2011): "that's really fast"

wal_buffers performance by Robert Haas (2012): pretty graphs

Monday, November 3, 2014

talk Thursday: Networking Tricks with the Linux Kernel

If you're in LA, come see me Thursday at the Unix Users meeting!

Everyone knows about networking. The Linux kernel provides a wide range of networking and other services, most of which are not generally known. In fact, a lot of what "everyone knows" is wrong or is lacking in detail. In this talk I'll give a technical survey of what's available and how to use it to accomplish magic tricks. With the right knowledge,our software can do great things without much code!


Monday, October 27, 2014

Programming vs Programmer

Although I have many years of professional programming experience, it's still startling to realize the distance between writing a program for "Hello World", vs working as a day-to-day programmer. The amount of stuff you have to know, and use quickly in ever-changing combinations, is substantial.



A student decried that his two-month class in Ruby didn't really prepare him for being a big-P Programmer, it just taught him little-p programming.



My view is that the material in that class, as well as Computer Science in general, is meaningless. What you learn is not useful, it's all about learning how to learn.



The following article shows how to do professional-level deployment of your Python webapp onto a server.  It uses ten different technologies, only one of which is "Python"!  This is all after your webapp is programmed and complete.  The pieces are: 1) version control with Git, 2) Python, 3) separate environments with Virtualenv, 4) installing packages with Pip, 5) using Flask for the webapp itself, 6) running webapps with Gunicorn, 7) handling web clients with Nginx, 8) keeping services running with Supervisor, deploying with 9) Fabric (TLDR: don't use this), and 10) off by one errors.



Most of the above technologies are complex, and have multiple books written about them. Using them together requires a firm grasp not on each tech itself, but in which parts to ignore, and which to use, in combination with which other technologies.



This is a hard skill to learn, but this characteristic separates little-p programmers from professional Programmers.



Kickstarting Flask on Ubuntu - setup and deployment - Real Python

Wednesday, October 22, 2014

Why I prefer Python to Java, in two code samples.

Why I prefer Python to Java, in two code samples.



heh

Vagrant/Ansible trick: autoreload

When testing Ansible scripts, it's nice to have your Vagrant VM automatically re-provision itself when you change the configuration.  In this way just saving a file lets you see the VM reinstall packages and so forth, with errors appearing immediately.

The following runs a tiny reprovisioning script each time your Ansible YAML files change:

pip install watchmedo

watchmedo shell-command --drop --pattern='*.yml' --command='clear; vagrant provision'

Hidden features of Python - Stack Overflow

Hidden features of Python - Stack Overflow -- lots of great, semi-obscure features!

Saturday, October 11, 2014

How to write dumber tests

Tests are hard.  The goal is to write a little bit of stuff that finds bugs in your code, you don't want to write unit tests so complex that they have bugs.  Charles (Luke) Lee breaks down some of the challenges, and gives solutions. Video from PyTexas 2014.

I have an entire rant about mocks, but that will wait for a later blog post.

How to write dumber tests