TIP: more info in Python's IPDB debugger

In ipdb the “dunder exception” variable has more information after your program crashes:

pdb> pp __exception__

(,
 ClientError...)

TIP: auto-fix your Python code!

The script autopep8 will automatically fix and reindent your Python code!

The following shows a "diff" of what PEP8 changes should be made, and then we apply the changes:

$ autopep8 -d collect.py
@@ -29,5 +29,5 @@

-if __name__=='__main__':
+if __name__ == '__main__':
     main()

$ autopep8 -i collect.py

talk: Functional Programming and Django QuerySets

thanks everyone for coming, for Media Temple for hosting, and for Esther for coordinating! I thought Justin's talk (his first!) on Swagger was fun and extremely useful. I hope everyone had a good time.

Here are the slides and notes for "Functional Programming and Django QuerySets" (2016 edition) -- have fun! http://johntellsall.github.io/johntellsall.com/class/django-queryset3/_build/

talk: Functional Programming and Django QuerySets soon!

I'll be speaking at the next SoCal Python meetup! http://www.meetup.com/socalpython/events/230329648/ If you want to go, sign up soon, there are only a few spots left.

This talk is mostly about Functional Programming in the Python world.  FP is great for testing, but it can be a bit mysterious at times. I highlight the awesomeness of FP, the dangerous bit, and how it is related to Django QuerySets.

I've given this talk a few times, and people always get excited about it. FP is not magic!  It's a clean and graceful way to write code.

Here's a link with my previous version's slides, notes, and other references: http://johntellsall.blogspot.com/2014/05/talk-functional-programming-and-django.html

Each time I give a talk I change it up a little or a lot. This time I'll probably add more cat photos.

UPDATED: quickly download lots of Python packages

This trick downloads Python packages up to 9x faster than normal:

egrep -o '^([A-Za-z].*==[^ ]+)' requirements.txt | xargs -n1 -P9 pip download

After things are downloaded, actually build and install the packages:

pip install -r requirements.txt

EDIT: original code gave `egrep: Invalid range end` -- fixed. Also added "-n1" to xargs so it'll download in parallel, vs sequentially.

Learning Python Programming

I give a number of talks on Python and other subjects. The two universal questions I get are:

- how can I... learn Python? get better at Python? learn programming?

- which web framework should I use?

There's a number of resources for learning programming with Python. Here are my favorites:

- Girl Develop It https://www.girldevelopit.com/

This (USA) national organization helps train women (and men) in learning web and software development. In the Los Angeles area they have classes nearly every weekend. My girlfriend took the "CSS" and "Javascript" classes and found them rewarding.

As an experienced dev who wants teaching experience, the organization itself is pretty easy to get in to. You can be a TA (teacher assistant) or work up to being a Teacher.  You can be paid or donate your salary to the national nonprofit.

I've taught a number of people, and found working with beginners to be humbling. They always ask insightful questions. As a teacher, to be clear and patient is a wonderful skill.

- MakerSquare http://www.makersquare.com/

This organization has an intensive "boot camp" in which you work, full time, for three months on training up to become a professional Javascript programmer. They also have "keep your day job" night class sessions.  Reportedly 96% of graduates get a professional Programming job in three months! Locations in Austin, San Francisco, and Los Angeles.

- Hack Night at Carbon Five http://www.meetup.com/Hack-Night-at-Carbon-Five-LA/

It's great fun to bring your laptop and projects to meet other geeks. This event is super chill and it's easy to talk to people and learn new things!

- Learn to Code with Us http://www.learntocodewithus.com/

I've been to the Culver City location (at Crash Space, a great hacker space), and it was a lot of fun. Lots of people work on projects. Michelle is very quiet and open, doing one-on-ones with people to help install Python or other software, or she'll help with general programming questions.

slides: Practical Python Testing

The talk at The Black Tux last night went really well!  Here are the slides:

- Practical Python Testing (google docs)

and me:

talk: Better Browser Tests with Selenium

Nicer Browser Tests with Selenium presentation

slides as Google doc

me presenting at the Python Meetup on 7/23:

Youtube videos of Python3 and Nicer Browser Tests

Thanks to Esther for putting everything together, Hulu for sponsoring, Philip for helping, and Carl Mullins for the video!

talk: Functional Programming and Django QuerySets

slides -- source on GitHub

My talk yesterday went well!  I warned a few people that their brains would explode, but in a good way.  It seemed everyone understood the material, and they had three different types of beer for jogging the little grey cells. Success!

For further reference, check the last slide for great stuff to read, or use these links:

Functional Programming HOWTO by Andy Kuchling -- clearly written, with tons of answers to "what?" and "why?", along with lots of Python examples

Can Your Programming Language Do This? by Joel Spolsky -- not Python specific

Wikipedia: Functional Programming -- clear, general use cases

Using Django querysets effectively by Dave Hall -- great, very useful for Django

Thanks to all who came out to learn new stuff. My hour talk was only a brief sketch, but it was a lot of fun to hang out and share knowledge. As always I learn from every single question people have. Thank you so much.

code: Multiprocessing producer/consumer with logging

Wordclouds (and multiple processes) are fun

Let's say you're writing a browser toy, to display what people say in their important #beer tweets. On the server you want to scan Twitter for #beer, and store the tweets. On occasion, a browser will fetch the list of words, then display most recent related words in those tweets as a word cloud.

To reduce complexity you don't want to add any extra packages, which might be untested and/or sketchy. What do you do with a standard "batteries included" Python? You use multiprocessing!

The multiprocessing module lets you write programs as a system of a connected processes. In this case, one is a producer: it does work then pushes information to list of tweets, shared in the system. Another process is a consumer: it waits for data from producer, then processes it for display on a browser in a pretty word cloud.

Server programming: log early and often

Without writing asynchronous code it's hard to do a lot of I/O in a single Python process. By splitting up your project into multiple tasks, each with its own process, each task can run on a separate CPU in parallel. The multiprocessing module helps us start and stop proceses, and communicate data back and forth.

In app programming, debugging is interactive; "print" statements are one way of testing the code.  On a server, this doesn't work as well.  It's best to have logging statements and lots of them to make sure the system works, and to diagnose errors.  Each logging entry has a timestamp, a severity, and a message.  By reviewing ERROR and WARNING entries one can verify the system works, and can diagnose it if it's behaving strangely, even if issues are rare.

It's better to have too much logging than not enough. Your Operations people don't understand your code.  If they see overall system problems it's easier for them to sift out the irrelevant logging messages, rather than add more logging into a complex system.

Log early, log often -- you and your operations people will love you for it.


This post was inspired by Playing with REALTIME data, Python and D3 by Brett Dangerfield. His code actually scans Twitter and does the word cloud display.

If you're even curious about Python, run don't walk to get Python Cookbook by David Beazley and Brian K. Jones.  I've been programming in Python for 15 years and learn new tools and techniques from every chapter!

In modern Python3, take a look at the more graceful concurrent.futures solution.

Code: mptest_proxy

#!/usr/bin/env python

'''
mptest_proxy.py -- producer adds to fixed-sized list; scanner uses them

OPTIONS:
-v  verbose multiprocessing output
'''

import logging, multiprocessing, sys, time


def producer(objlist):
    '''
    add an item to list every 2 sec; ensure fixed size list
    '''
    logger = multiprocessing.get_logger()
    logger.info('start')
    while True:
        try:
            time.sleep(1)
        except KeyboardInterrupt:
            return
        msg = 'ding: {:04d}'.format(int(time.time()) % 10000)
        logger.info('put: %s', msg)
        del objlist[0]
        objlist.append( msg )


def scanner(objlist):
    '''
    every now and then, run calculation on objlist
    '''
    logger = multiprocessing.get_logger()
    logger.info('start')
    while True:
        try:
            time.sleep(5)
        except KeyboardInterrupt:
            return
        logger.info('items: %s', list(objlist))
            

def main():
    opt_verbose = '-v' in sys.argv[1:] 
    logger = multiprocessing.log_to_stderr(
            level=logging.DEBUG if opt_verbose else logging.INFO,
    )
    logger.info('setup')

    # create fixed-length list, shared between producer & consumer
    manager = multiprocessing.Manager()
    my_objlist = manager.list( # pylint: disable=E1101
        [None] * 10
    )

    multiprocessing.Process(
        target=producer,
        args=(my_objlist,),
        name='producer',
    ).start()

    multiprocessing.Process(
        target=scanner,
        args=(my_objlist,),
        name='scanner',
        ).start()

    logger.info('running forever')
    try:
        manager.join() # wait until both workers die
    except KeyboardInterrupt:
        pass
    logger.info('done')
    

if __name__=='__main__':
    main()

automating web tests with Ghost

Developing reliable software for the web is a challenging proposition. A site's code organization is complex, with libraries interacting with each other in an exponentially expanding number of possibilities.  It's quite easy to make a minor change in one part of the site that breaks another part.

Manual testing isn't an option. If you have a rarely-used but critical part of the site's workflow, like "email me my forgotten password", it would affect users if the functionality breaks. With manual testing you might never know your users are getting a poor experience.

One technique for making reliable web software is using automated tests.  In addition to the lower-level unit tests of code, it's valuable to have a scriptable "browser" that is able to click through your site and check various features.

Install and verify Ghost

Here's how to install and use Ghost, a Python library that emulates a Webkit browser. The browser is comparable to Apple Safari or Google Chrome.  Instructions are for Ubuntu.

sudo apt-get install python-pyside
sudo pip install Ghost.py

Verify that things were installed correctly:

python -c 'from ghost import Ghost'

Example headless web scripting

Here's a sample program.  It loads the Duck Duck Go search page, then types in a search query. It then clicks the submit button, waits for the search results to load, and prints out the HTML of the first result. Finally it captures a snapshot of the page!

#!/usr/bin/python
from ghost import Ghost

ghost = Ghost()
ghost.open('http://duckduckgo.com/') 

ghost.wait_for_selector('input[name=q]')
ghost.fill("#search_form_homepage", {'q': 'beer'})
ghost.fire_on("#search_form_homepage",
              "submit",
              expect_loading=True) 

ghost.wait_for_selector('#r1-0')
result, _resources = ghost.evaluate(
    "document.getElementById('r1-0').innerHTML;")
print result

ghost.capture_to('beer.png')

Here's the above in a Gist for your use: Ghost example with Duck Duck Go

Resources:

• Steven Fruchter's Ghost tutorial
• Ghost.py home page

trivial Jenkins configuration

Knowing when your software breaks is useful. If you catch subtle errors soon after the change, it's *much* easier to figure out which bit of code "optimization" broke things for the users.

Another benefit is fixing "works for me" syndrome. Even if you're building software just for yourself, it's nice to have a 3rd party to verify that you didn't do silly things. Forgetting to check in South database changes is quite easy, but that simple omission will break everything.

Jenkins is a cranky to set up, because it's a collection of moving parts. The best setup would be having Jenkins rebuild your tests every time GitHub detects a source code change.

The following setup is not as geeky, but it's much, much easier: tell Jenkins copy local source files and test, every now and then.

1. go to Jenkins on your local computer: http://localhost:8080/
2. New Job ("free-style software project")
3. You're now on the Configure page. click Add Build Step.
4. Execute Shell
5. adjust Makefile to your liking
6. run the test every few minutes

For #4, in the Command area, type this little script:


rsync -av --delete --exclude='.git' MYSOURCEDIR $WORKSPACE
make -C $WORKSPACE test

Replace MYSOURCEDIR with your source directory -- the full path. For me it's /home/johnm/work/yed.   The above copies source from your normal development area to the job-specific Jenkins area.  It strips files that have been deleted in the source tree

In step #5, put a Makefile with a "test" verb in the top of your source code. I'm running Django, so to test the project I want to run the Django test suite on my main app.


test:
./manage.py test -v1

Remember to use the TAB key instead of spaces on the 2nd line.

At this stage, test your Jenkins job.  On the Job screen click Configure on the left to return to the Configuration page.  Edit the job, but don't press Save at the bottom of the page -- press Apply.  Open your job in a second browser window.  Click Build Now to start the job, wait for a few seconds, then click the new link which appears in the Build History section of the page.

If it's not perfect, go back to the Configuration page, make changes, press Apply; switch to the Project browser window, click Build Now again.  This workflow lets you rapidly make changes to your Jenkins project without having lots of windows open.


Step 6: Once #1-5 above is working to your satisfaction, you can automate the test.  On the Job Configure screen, in the Build Triggers section, enable "Build periodically".  In the text area, type this schedule:

H/15 * * * *

This means run your test every 15 minutes, no matter if anything has changed or not.  The "H" is to not overwhelm your system if lots of tests are running at once.



As an alternate for #5, just to make sure things are running, you can use the following Makefile instead:

test:
echo 'out of beer!' ; false

When you run the Jenkins job on this, the test will fail because the "false" command returns a status of 1. This proves that Jenkins sees your code, is copying to the right place, and is running your Makefile test command.

excellent introduction to idiomatic Python

http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

Profiling Django

Install django-extensions

http://pythonhosted.org/django-extensions/runprofileserver.html


1) sudo pip install django-extensions

2) add 'django_extensions' to your app's INSTALLED_APPS list.

Run server in profiling mode

python  ./manage.py runprofileserver --prof-path=/tmp 8001

Do a query

time curl -i http://localhost:8001/account/eventboard/update/

Write little reporting module

prof.py -- given later

Run report

/tmp/account.eventboard.update.017442ms.1363891644.prof :
         202963 function calls (197801 primitive calls) in 17.443 seconds

   Ordered by: internal time, call count
   List reduced from 2280 to 5 due to restriction <5>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    26201   14.956    0.001   14.956    0.001 /usr/lib/python2.7/ssl.py:154(read)
        2    0.725    0.363    0.725    0.363 /usr/lib/python2.7/ssl.py:301(do_handshake)
      937    0.470    0.001   15.069    0.016 /usr/lib/python2.7/socket.py:406(readline)
       10    0.318    0.032    0.318    0.032 /usr/lib/python2.7/socket.py:223(meth)
        2    0.238    0.119    1.291    0.646 /usr/local/lib/python2.7/dist-packages/httplib2/__init__.py:982(connect)


In this case, most of the page delay was transferring SSL data -- probably from Google Analytics. Huzzah!




#!/usr/bin/env python
# prof.py

import glob, logging, hotshot.stats, pstats, sys

logging.basicConfig(stream=sys.stderr)
LOG = logging.getLogger(__name__)

for path in sys.argv[1:] or glob.glob('/tmp/*.prof'):
    if 'static.' in path:
        continue
    try:
        stats = hotshot.stats.load(path)
    except ValueError:
        stats = pstats.Stats(path)

    stats.sort_stats('time', 'calls') # sort the output based on time spent
    print path,':'
    stats.print_stats(5) # print the top culprits