Tuesday, September 2, 2014

preview: Python Library class

I'm developing a course on Python, and another on Professional Software development. Enclosed is a piece, showing the main basic modules in Python and why to use them.

Saturday, August 23, 2014

best practices with Python's multiprocessing module

Thanks to dano and others on StackOverflow, I have more best practices for Python's multiprocessing module:

1) put everything in an if __name__=='__main__' block, vs having top-level code.  This is a Windows thing, but is good practice in general. Main-level code will be executed even if the module is imported, as opposed to being executed directly.

2) I'd previously been using signal.pause() to wait for subprocesses to exit and/or die.  Alas this has two challenges: a) this isn't available in Windows, and b) if a subproc dies, the parent will still hang around until explicitly killed/interrupted.

Here's an example of the right and wrong way to wait for a subproc.  If the signal.pause() path is enabled, then the program will run forever, ignoring the child proc that exits with an error.  The other path is more verbose, but correctly runs on Windows, and handles children correctly.

source

    import multiprocessing, signal, sys, time
    
    def wait_crash():
        time.sleep(1)
        sys.exit('crash')
    
    if __name__=='__main__':
        procs = [ multiprocessing.Process(target=wait_crash) ]
        for proc in procs:
            proc.start()
        if 0:
            signal.pause()              # XXX: not on Windows
        else:
            for proc in procs:
                proc.join()

output

    crash

Wednesday, August 20, 2014

talk: "Tricking out Linux kernel networking" tomorrow night

I'll be giving an excerpt of this talk tomorrow at Santa Monica Coloft -- please drop by!

Summary: "We are all familiar with classical TCP sockets. The Linux kernel gives us lots of other sockets, and internal services, to accomplish cool tricks with minimal effort."

The full thing will wind up to be around two hours, so this 20-30 min version will be less code and more jazz hands, which will make it fun!

http://www.meetup.com/LAHackers/events/198602002/

Tuesday, August 19, 2014

Monday, August 18, 2014

talk tomorrow: Automatic Parameter Optimization

good news: I'll be presenting for the Professional Python group tomorrow in Santa Monica, on "Automatic Parameter Optimization". I'll also highlight upcoming classes and lead a discussion. Meetup: Professional Python Users Group

The gist: let's say you have 1000 files to process. Do you want to process them all, and get all the results at once, a long time later? Do you want to work on one at a time? Do you care about the overall time it takes, or being efficient with the CPU or storage? I'll demonstrate the Hyperopt library which is easily adapted to automatically figure out different types of parameters for you.

In related news, Facebook spins servers up and down to efficiently manage power consumption -- Making Facebook’s software infrastructure more energy efficient with Autoscale