code: sync files using FTP

My webhost doesn't support rsync, only FTP.  For grins I wrote a script to scan through interesting local files and push them up to my webhost.  Alas the progress wasn't as trivial as I'd hoped, so I abandoned the effort and used lftp the "mirror" command.

The code shows a number of best practices:

• don't store passwords/keys, pass them in via environment variables
• make an iterator of files, add to it using chain()
• Python3 subprocess.check_output() is a little odd
• logging, like voting, should be done early and often
• code ensures FTP connection always closed cleanly

https://github.com/shavenwarthog/johntellsall/blob/master/class/django-queryset/ftpsync.py

talk: Functional Programming and Django QuerySets

slides -- source on GitHub

My talk yesterday went well!  I warned a few people that their brains would explode, but in a good way.  It seemed everyone understood the material, and they had three different types of beer for jogging the little grey cells. Success!

For further reference, check the last slide for great stuff to read, or use these links:

Functional Programming HOWTO by Andy Kuchling -- clearly written, with tons of answers to "what?" and "why?", along with lots of Python examples

Can Your Programming Language Do This? by Joel Spolsky -- not Python specific

Wikipedia: Functional Programming -- clear, general use cases

Using Django querysets effectively by Dave Hall -- great, very useful for Django

Thanks to all who came out to learn new stuff. My hour talk was only a brief sketch, but it was a lot of fun to hang out and share knowledge. As always I learn from every single question people have. Thank you so much.

code: use Bash to control workflow of processes

On Stack Overflow, someone asked how to control a workflow of processes.  The issue is that sometimes the procs will jam up, and should be restarted.  The following code does this.

One technique to control an errant pipeline is to count the number of pipeline processes after a while. If they haven't finished, then zap them and restart.

In this example code we use ask Bash for its process id (the $$ variable), then use pgrep to find all sub-processes of the script. That is, pgrep tells us all the workflow process IDs. If there are any, then we use the corresponding pkill command to squash them before a restart.

In addition, we use the date command to output a pretty log file, showing what the system is doing at what time.

full post on SO: How to send SIGKILLs to entire pipeline?

source: https://github.com/shavenwarthog/johntellsall/tree/master/karma/kill-pipeline

code: Multiprocessing producer/consumer with logging

Wordclouds (and multiple processes) are fun

Let's say you're writing a browser toy, to display what people say in their important #beer tweets. On the server you want to scan Twitter for #beer, and store the tweets. On occasion, a browser will fetch the list of words, then display most recent related words in those tweets as a word cloud.

To reduce complexity you don't want to add any extra packages, which might be untested and/or sketchy. What do you do with a standard "batteries included" Python? You use multiprocessing!

The multiprocessing module lets you write programs as a system of a connected processes. In this case, one is a producer: it does work then pushes information to list of tweets, shared in the system. Another process is a consumer: it waits for data from producer, then processes it for display on a browser in a pretty word cloud.

Server programming: log early and often

Without writing asynchronous code it's hard to do a lot of I/O in a single Python process. By splitting up your project into multiple tasks, each with its own process, each task can run on a separate CPU in parallel. The multiprocessing module helps us start and stop proceses, and communicate data back and forth.

In app programming, debugging is interactive; "print" statements are one way of testing the code.  On a server, this doesn't work as well.  It's best to have logging statements and lots of them to make sure the system works, and to diagnose errors.  Each logging entry has a timestamp, a severity, and a message.  By reviewing ERROR and WARNING entries one can verify the system works, and can diagnose it if it's behaving strangely, even if issues are rare.

It's better to have too much logging than not enough. Your Operations people don't understand your code.  If they see overall system problems it's easier for them to sift out the irrelevant logging messages, rather than add more logging into a complex system.

Log early, log often -- you and your operations people will love you for it.


This post was inspired by Playing with REALTIME data, Python and D3 by Brett Dangerfield. His code actually scans Twitter and does the word cloud display.

If you're even curious about Python, run don't walk to get Python Cookbook by David Beazley and Brian K. Jones.  I've been programming in Python for 15 years and learn new tools and techniques from every chapter!

In modern Python3, take a look at the more graceful concurrent.futures solution.

Code: mptest_proxy

#!/usr/bin/env python

'''
mptest_proxy.py -- producer adds to fixed-sized list; scanner uses them

OPTIONS:
-v  verbose multiprocessing output
'''

import logging, multiprocessing, sys, time


def producer(objlist):
    '''
    add an item to list every 2 sec; ensure fixed size list
    '''
    logger = multiprocessing.get_logger()
    logger.info('start')
    while True:
        try:
            time.sleep(1)
        except KeyboardInterrupt:
            return
        msg = 'ding: {:04d}'.format(int(time.time()) % 10000)
        logger.info('put: %s', msg)
        del objlist[0]
        objlist.append( msg )


def scanner(objlist):
    '''
    every now and then, run calculation on objlist
    '''
    logger = multiprocessing.get_logger()
    logger.info('start')
    while True:
        try:
            time.sleep(5)
        except KeyboardInterrupt:
            return
        logger.info('items: %s', list(objlist))
            

def main():
    opt_verbose = '-v' in sys.argv[1:] 
    logger = multiprocessing.log_to_stderr(
            level=logging.DEBUG if opt_verbose else logging.INFO,
    )
    logger.info('setup')

    # create fixed-length list, shared between producer & consumer
    manager = multiprocessing.Manager()
    my_objlist = manager.list( # pylint: disable=E1101
        [None] * 10
    )

    multiprocessing.Process(
        target=producer,
        args=(my_objlist,),
        name='producer',
    ).start()

    multiprocessing.Process(
        target=scanner,
        args=(my_objlist,),
        name='scanner',
        ).start()

    logger.info('running forever')
    try:
        manager.join() # wait until both workers die
    except KeyboardInterrupt:
        pass
    logger.info('done')
    

if __name__=='__main__':
    main()

tool: IPython: Math Processor

Using IPython we can use a full programming language connected to a sophisticated rendering engine.  It will draw pretty equations, 2D and 3D plots of your data, in an interactive shell!



* IPython: Math Processor

reference: Real Time Web sans Node

Real Time Web sans Node by Michael Irani

presented at LA Django meetup in March, 2014

slides - source

Class Topics 2Q2014

I'm now available for teaching classes.  I have 15+ years experience with Python and the Web, 5 with Django, and ... many with Linux and Devops.  To see one of these talks, or to customize one for your group, please ping me at LinkedIn or visit the LA Django, SoCal AngularJS, or LA DevOps meetups.

Upcoming talks:

- Intermediate Django Class-based Views

Includes CBVs were a Mistake

- Modern Django Stack with WebSocket support

Ideas:

- Something Devops: high level, aka "Why All the Things?"

Dev and Operations can be seen as opposites.  From a high level, we have several "knobs" to twiddle in a company -- Devops is one such arrangement.  How can we make concrete decisions to move our project forward?  How can we be creative enough to see the entire range of options? Most importantly: how can we work the least, to have the awesomest project, and enjoy the most amount of quality beer-drinking time?

- Something Devops: low level.  Devops as advanced programming, and vice versa

- Something Workflow with Knobs

aka "why not run production on my laptop?"

- Startup "handwaving with code" vs Durable Systems

- Lies, Damn Lies, and Benchmarking

- Old School Stuff means Less Work

aka "xargs ftw"

Historical Talks/Classes

- Twisted -- high performance brain-melting asynchronous server

- Intro to Programming with Python

- Python for Visual Effects Artists