best practices with Python's multiprocessing module

Thanks to dano and others on StackOverflow, I have more best practices for Python's multiprocessing module:

1) put everything in an if __name__=='__main__' block, vs having top-level code.  This is a Windows thing, but is good practice in general. Main-level code will be executed even if the module is imported, as opposed to being executed directly.

2) I'd previously been using signal.pause() to wait for subprocesses to exit and/or die.  Alas this has two challenges: a) this isn't available in Windows, and b) if a subproc dies, the parent will still hang around until explicitly killed/interrupted.

Here's an example of the right and wrong way to wait for a subproc.  If the signal.pause() path is enabled, then the program will run forever, ignoring the child proc that exits with an error.  The other path is more verbose, but correctly runs on Windows, and handles children correctly.

source

    import multiprocessing, signal, sys, time
    
    def wait_crash():
        time.sleep(1)
        sys.exit('crash')
    
    if __name__=='__main__':
        procs = [ multiprocessing.Process(target=wait_crash) ]
        for proc in procs:
            proc.start()
        if 0:
            signal.pause()              # XXX: not on Windows
        else:
            for proc in procs:
                proc.join()

output

    crash

slides: Tricky Linux Networking

slides: Tricky Linux Networking

talk: "Tricking out Linux kernel networking" tomorrow night

I'll be giving an excerpt of this talk tomorrow at Santa Monica Coloft -- please drop by!

Summary: "We are all familiar with classical TCP sockets. The Linux kernel gives us lots of other sockets, and internal services, to accomplish cool tricks with minimal effort."

The full thing will wind up to be around two hours, so this 20-30 min version will be less code and more jazz hands, which will make it fun!

http://www.meetup.com/LAHackers/events/198602002/

talk: Automatic Parameter Optimization

slides: http://johntellsall.com/class/auto-optimize/slides/index.html

talk tomorrow: Automatic Parameter Optimization

good news: I'll be presenting for the Professional Python group tomorrow in Santa Monica, on "Automatic Parameter Optimization". I'll also highlight upcoming classes and lead a discussion. Meetup: Professional Python Users Group

The gist: let's say you have 1000 files to process. Do you want to process them all, and get all the results at once, a long time later? Do you want to work on one at a time? Do you care about the overall time it takes, or being efficient with the CPU or storage? I'll demonstrate the Hyperopt library which is easily adapted to automatically figure out different types of parameters for you.

In related news, Facebook spins servers up and down to efficiently manage power consumption -- Making Facebook’s software infrastructure more energy efficient with Autoscale

(upcoming talk) Tricking out your Linux Kernel Networking

teaser (1-2 sentences)

The Linux kernel can be taught to perform amazing tricks. In this talk we’ll show cool stuff to do using nonstandard networking, interprocess communication, and  services like filesystems and network routing.
---
Tired of normal TCP sockets? Linux can do so much more!

ideas

The kernel provides control and connection with external systems, kernel services, kernel itself, and other processes. Full knowledge allows application magic, like reconnecting streaming sockets to unconnected processes, or getting info from 3rd party systems by watching indirect effects.

outline

In the Wild-- Faint of Heart need Not Apply
* Standard packages use tricks to achieve high performance and lots of features
Nginx
Apache
Gunicorn
uwsgi
Socket/File similarities (-> Netlink)
* Sockets and files are similar to the kernel; Netlink sockets give bulk info from kernel services
Socket
TCP
UDP
Unixdom + Pipes
cheat: unixdom as server lock  (“Abstract Socket Namespace”)
* By using Unix domain sockets we can produce reliable “server once once” semantics.  To my knowledge no one has discovered this before.
cheat: rebind socket
* Programmers think of TCP sockets as point-to-point.  That’s true, but you can rebind the endpoints!
cheat: rebind socket inside container?
* Docker and LXC allow higher security by having separate user spaces utilizing the same kernel. It may be possible to rebind a socket from the parent container into another, giving very high security and durability.
Files (kernel service)
inotify
cheat: inotify + MySQL
* By using the inotify kernel service we can transparently add a feature to MySQL -- we can detect when a table is modified.
filesystem
cheat: autoconvert videos for smart TV w/ DLNA
* We can write a virtual filesystem which shows video files automatically transcoded into the correct format for a TV or Roku box.
Netlink (kernel service)
Socket/File/Netlink comparison
wifi up/down
* Easy to run scripts when network connects to certain wifi hotspots; no polling needed.

another talk

Strace/Ftrace
very powerful
high overhead

Wizardry, Trickery, Jedi master mind control
http://troydhanson.github.io/misc/Unix_domain_sockets.html
Direct vs Indirect: ask socket for info, vs watch side effects in the system