Wednesday, September 28, 2016

TIP: work with Amazon AWS ECS Container Service Docker containers

When running Amazon EC2 Container Service (ECS), have these two commands running in Tmux on your instances:

tail -f $(ls -1t /var/log/ecs/ecs-agent.log.* | head -1)

sudo tail -f /var/log/upstart/docker.log


In another Tmux tab, this command lets you look at the logs from your running Docker container:

docker logs -f $(docker ps --filter name=myname,status=running --quiet)

Friday, September 9, 2016

TIP: fix bizarre Docker problems

docker rm -fv $(docker ps -aq)

Sometimes testing produces strange Docker-related problems.  The above command kills all containers, including the stopped ones.  The "-v" also destroys each container's private storage volumes.  This is great because Docker likes to run differently different times you run it -- because it has local state.  Local state is evil!  Run the above command every now and then so Docker starts from a clean, state-less, state.

In Docker, stopping a container is different from killing it.  The former stops the running container/process, but leaves the private volume.  The latter zaps the volume.

Thanks Sam!

Monday, September 5, 2016

talk: Platonic Solids of Quality

(This page is bit.ly/jta-platonic )

Slides: John Tells All: Platonic Solids of Quality

This page will be updated when the video appears.  Thanks to Don Westland for the A/V, Marcel for organizing and the picture, DataScience Inc for the hosting, and Keith B. for the presentation and the smooooth scotch.




Friday, September 2, 2016

TIP: auto-fix your Python code!

The script autopep8 will automatically fix and reindent your Python code!

The following shows a "diff" of what PEP8 changes should be made, and then we apply the changes:

$ autopep8 -d collect.py
@@ -29,5 +29,5 @@

-if __name__=='__main__':
+if __name__ == '__main__':
     main()

$ autopep8 -i collect.py

reach your next-level DevOps Career

I often get questions about how to get into development or how to progress in a dev/webdev/devops career.  Here's my most recent "From Dev to DevOps" ideas:

- become very familiar with doing container-centric development.  At the moment "Docker Compose" will get you comfortable with the terminology and mechanics of containers.  It's extremely useful for local dev work. It's also very similar to Docker Cloud, and leads pretty directly to Kubernetes and Amazon EC2 Container Service (ECS).

- learn one or more orchestration softwares.  Chef is the biggest, Puppet is cool, Salt Stack is the new hotness.  My favorite is Ansible: it's human-readable, flexible, fast, and obvious.  The other day I wrote a "create or update a distributed S3 bucket, and verify it has the correct permissions" in 7 lines of code!

- Amazon AWS is a pain in the ass to learn. However, there are tons of really excellent talks, so you can teach yourself quite a bit. You don't have to do Amazon, but it's what 95% of the startups use.

- Invest in O'reilly Safari Online ($40/month?) to read all their books and watch all their videos.  I'll post my favorites in another article. Start with "Agile Testing" and "Continuous Delivery."

- DevOps can really accelerate development by creating and maintaining a strong Continuous Integration system and Testing culture in general.  Run don't walk to read "Agile Testing" by Lisa Crispin. I steal her stuff for about half of my talks. (If you saw my "Python Testing" talk last year, there was a bit in there borrowed from the book).

- tell your current employer that your career is moving into DevOps (/ Amazon). They're getting a good deal in you if you stay there and contribute, so they'll start investing in you by sending you to conferences and/or training you up.

- Run don't walk to go to all Meetups you're even vaguely interested in.  I do Unix Users Group (UUASC, heavy servers) and LA DevOps (these are listed on the sidebar.) My favorite is the Socal Python group. It's super friendly and wide-ranging.

- enjoy the ride!

Sunday, August 28, 2016

Speeding up UI Browser tests with Robot Framework

My current AWS demo project is a Cat Voting Booth.  It uses SocketIO/WebSockets, so in addition to server-side unit tests I've built a number of browser-level tests, using Robot Framework.

One feature people don't realize is that Robot can actually be quite speedy. Opening and closing a browser window is very slow, on the order of 1-2 seconds. Depending on the application, this can be done once, saving 1-2 seconds for *every* test!  For my Very Important Cat Voting Booth, this is the case.

Here's the magic bit:

*** Settings ***
Resource          resource.robot
Test Setup        Reset Votes
Suite Setup       Open Browser To Voting Page
Suite Teardown    Close All Browsers

*** Test Cases ***
Valid Page
    Votes Not Available

Register Up Vote
    Vote Up
    Element Text Should Be    vote-count-up    1
    Element Text Should Be    vote-count-down    0

Normally, Robot will open/close a browser on each test.  The above stanza says "Test Setup: Reset Votes", that is, it'll reset my database and *not* open a browser.  The suite-level setup is "Suite Setup: Open Browser", which means Robot will open a browser once and not per test.  Similarly the suite-level teardown closes all the (Robot-created) browsers.

I can run four (admittedly modest) browser-level tests, in a full real browser, in four seconds! This includes going back and forth to my Flask-SocketIO server, interacting with Redis, and other things.  I'm quite happy with this result.

As a heavy Test Driven Development (TDD) guy, with the above speed I can do full browser-level tests as TDD!  I don't have to wait until I absolutely *have* to run UI tests, they can be fast enough for running many times an hour, to give me very fast development feedback.

Full source code here: https://github.com/johntellsall/aws-realtime-metrics

Tuesday, August 23, 2016