Django: speed up Sqlite 1000x!

I'm working on a project analyzing large code bases. For just messing around, I'm using Sqlite. A strange thing happened when updating ~10,000 rows: they were really slow! Sqlite was updating about 10 records a second.

It turns out Sqlite correctly cares about your data, so it makes a transaction around the UPDATE statement, verifying that each and every bit of your data hits disk before returning. For me, if disaster strikes and I lose my tiny database I can recreated in a few seconds, so I decided to live life dangerously, and tell Sqlite to go ahead and try to update data, but don't wait around for it to hit disk.

Here's the magic. Put this before your bulk INSERT/UPDATE statements. Note that it affects the entire session, so you don't want to do this before your valuable data-manipulation commands.

from django.db import connection
if connection.vendor == 'sqlite':

    connection.cursor().execute('PRAGMA synchronous=OFF')

Three commands sped up my program a zillion percent. The entire 10K rows updated in a fraction of a second, vs minutes. Yay!

talk: Practical Python Testing

I'm speaking at this month's SoCal Python Meetup!  I'll post slides and notes here soon.

http://www.meetup.com/socalpython/events/224586741/

How and why we use DevOps checklists - Server Density Blog

In the health care and airline industries, simple checklists save thousands of lives.  Here are several clear examples how the same technique is used in DevOps:



How and why we use DevOps checklists - Server Density Blog

tip: easily run Postgres administration commands

TIP: on Linux it helps to be the Postgres user to do administration stuff with the database.  Either do some configuration twiddling, or run commands as the postgres user:

sudo su -c 'dropdb mydb' postgres

In Ansible, this is:

 - name: database -- zap database

      command: dropdb mydb 

      sudo: yes

      sudo_user: postgres

talk: Better Browser Tests with Selenium

Nicer Browser Tests with Selenium presentation

slides as Google doc

me presenting at the Python Meetup on 7/23:

Youtube videos of Python3 and Nicer Browser Tests

Thanks to Esther for putting everything together, Hulu for sponsoring, Philip for helping, and Carl Mullins for the video!

TIP: tweak number of Gunicorn workers

Count number of Gunicorn workers

$ pgrep gunicorn.*work | wc -l
5

Add another worker by signaling the Gunicorn master

$ sudo pkill -TTIN  gunicorn.*mast

Verify another worker has appeared

$ pgrep gunicorn.*work | wc -l
6

Decrease the number of workers, verify its effect

$ sudo pkill -TTOU  gunicorn.*mast
$ pgrep gunicorn.*work | wc -l
5

DevOps in Southern California

If you're a networking/DevOps person in Southern California, here is a meetup I recommend:

UUASC - on Meetup

The UNIX Users Association of Southern California is mostly about DevOps and system administration.  The last talk I attended was on Google's Kubernetes. Kubernetes is a lightweight system for deploying, scaling, and maintaining containerized applications. You can run on top of nearly anything: bare metal, Vagrant, AWS, Google GKE, OpenStack, Mesos... the list is enormous.

We meet monthly, normally at Q in Howard Hughes Center. Jordan and Carolyn are open and sweet people.  The audience is generally ~30, and can be quite lively. I gave a "Tricking Out the Linux Kernel Networking Stack" talk last year, and it was helpful to have people let me know which slides were too simplistic, and which material was controversial. Unlike many meetups, the group is is very experienced, and can help put "new hotness" technologies in contrast to other solutions. It's really important to know when and why to use technology, and how it can work in concert with other techniques.