reach your next-level DevOps Career

I often get questions about how to get into development or how to progress in a dev/webdev/devops career.  Here's my most recent "From Dev to DevOps" ideas:

- become very familiar with doing container-centric development.  At the moment "Docker Compose" will get you comfortable with the terminology and mechanics of containers.  It's extremely useful for local dev work. It's also very similar to Docker Cloud, and leads pretty directly to Kubernetes and Amazon EC2 Container Service (ECS).

- learn one or more orchestration softwares.  Chef is the biggest, Puppet is cool, Salt Stack is the new hotness.  My favorite is Ansible: it's human-readable, flexible, fast, and obvious.  The other day I wrote a "create or update a distributed S3 bucket, and verify it has the correct permissions" in 7 lines of code!

- Amazon AWS is a pain in the ass to learn. However, there are tons of really excellent talks, so you can teach yourself quite a bit. You don't have to do Amazon, but it's what 95% of the startups use.

- Invest in O'reilly Safari Online ($40/month?) to read all their books and watch all their videos.  I'll post my favorites in another article. Start with "Agile Testing" and "Continuous Delivery."

- DevOps can really accelerate development by creating and maintaining a strong Continuous Integration system and Testing culture in general.  Run don't walk to read "Agile Testing" by Lisa Crispin. I steal her stuff for about half of my talks. (If you saw my "Python Testing" talk last year, there was a bit in there borrowed from the book).

- tell your current employer that your career is moving into DevOps (/ Amazon). They're getting a good deal in you if you stay there and contribute, so they'll start investing in you by sending you to conferences and/or training you up.

- Run don't walk to go to all Meetups you're even vaguely interested in.  I do Unix Users Group (UUASC, heavy servers) and LA DevOps (these are listed on the sidebar.) My favorite is the Socal Python group. It's super friendly and wide-ranging.

- enjoy the ride!

Quality DevOps: installing and verifying Network Time Protocol (NTP)

I lurve Ansible. It lets me install or update software on one or 100 instances, easily. The entire system becomes a set of scripts to run and run and run again until I get things exactly the way I want them.

In today's devops ecosystem, where "infrastructure is code", how do we test our infrastructure?

Ansible gives us one way to do this.  When we install or update a service, run a service-specific command to make doubly sure that things are working as expected. If something's not quite right, Ansible will abort and we can figure out what went kablooey.

Save the following into "ntp.yml" and run with ansible-playbook -vvi myhost ntp.yml

Thanks to phillipuniverse !

# ntp.yml -- install NTP time sync daemon
# Adapted from https://gist.github.com/phillipuniverse/7721288#file-ntp_playbook-yml
#
# USAGE: ansible-playbook -vvi myhost ntp.yml
#
---

- hosts: all
  become: yes
  gather_facts: no

  tasks:

    - name: Install NTP
      apt: package=ntp state=present update_cache=yes
      tags: ntp

    - name: Make sure NTP is started up
      service: name=ntp state=started enabled=yes
      tags: ntp

    - name: verify NTP synchronized
      command: timedatectl status
      register: ntp_result
      failed_when: "'synchronized: yes' not in ntp_result.stdout"
      tags: ntp


  handlers:
    - name: restart ntp
      service: name=ntp state=restarted

DevOps, Availability, and Risk

excellent points from an episode of Arrested DevOps, entitled "Who Owns Your Availability?" (TLDR: you do!) https://www.arresteddevops.com/availability/

My thoughts:

- technical risk can produce business risk, as in "your hundred employees can't do anything for an hour", up to "your database is gone therefore the company is gone" kinds of risk.  Or, "feature X doesn't work for user class Y" kinds of risk. Do you as a business prioritize consumers paying you, or you delivering their stuff, or your admins/phone people delighting your customers, or your developers fixing bugs?

From the show (Charity Majors, Pete Cheslock, ADO crew). (Quotes are my foggy recollections, not quotes.):

- cache ("vendor") your dependencies

If you can't deploy to production because GitHub or a 3rd party package server in China is down, things are not good.  Likewise, if your server is connecting to China and all your packages are local, perhaps it's time for a security check. (If you don't know what servers your server is talking to, that's another risk.)

- what is your Risk Profile? What is considered acceptable risk?

As your company starts it's probably fine to rely on the internet being always available all the time. Not being able to deploy for an hour/day might be okay. Spending resources on growing your company might be a good tradeoff vs security and availability.

- your dependencies are cached. What about deps of deps of deps?

- "Packerize the base"

If your system has a baked, reliable base, with a little bit of changes on top, then it's easier to track down and fix things that break.

One mechanism is "baking" all your random dependencies to a Docker layer.  Or, network volume -- Amazon S3 for example (deb-s3).  It can still go down, but if it's up you get everything in one place. It'll be there for you even if the original host is not happy for whatever reason. One person mentioned she had more problems with GitHub's reliability than her own.

Another failure mode: known-good version is broken. Your business depends on the "beer-1.0" package. It's been working fine for months.  Developer gets drunk and uploads a broken package, but uses the same version number -- "beer-1.0" is now broken.  You can no longer make changes to your business!  Since you own your availability, it's your problem.

- "if you treat your devs like children, they'll act like children. They'll become subject matter experts on doing things the wrong way. We as devops can be spirit guides, career counselors for your leveling up skills." Developers own the code, the availability. Give them pagers and wake them up when the site has problems.

- site should have "circuit breakers" - if the site is in "continuous partial failure", that's better than just being down for everyone, full stop.


I dig the Arrested DevOps podcast, and listen to it often. Thanks!

tip: easily run Postgres administration commands

TIP: on Linux it helps to be the Postgres user to do administration stuff with the database.  Either do some configuration twiddling, or run commands as the postgres user:

sudo su -c 'dropdb mydb' postgres

In Ansible, this is:

 - name: database -- zap database

      command: dropdb mydb 

      sudo: yes

      sudo_user: postgres

TIP: tweak number of Gunicorn workers

Count number of Gunicorn workers

$ pgrep gunicorn.*work | wc -l
5

Add another worker by signaling the Gunicorn master

$ sudo pkill -TTIN  gunicorn.*mast

Verify another worker has appeared

$ pgrep gunicorn.*work | wc -l
6

Decrease the number of workers, verify its effect

$ sudo pkill -TTOU  gunicorn.*mast
$ pgrep gunicorn.*work | wc -l
5

DevOps in Southern California

If you're a networking/DevOps person in Southern California, here is a meetup I recommend:

UUASC - on Meetup

The UNIX Users Association of Southern California is mostly about DevOps and system administration.  The last talk I attended was on Google's Kubernetes. Kubernetes is a lightweight system for deploying, scaling, and maintaining containerized applications. You can run on top of nearly anything: bare metal, Vagrant, AWS, Google GKE, OpenStack, Mesos... the list is enormous.

We meet monthly, normally at Q in Howard Hughes Center. Jordan and Carolyn are open and sweet people.  The audience is generally ~30, and can be quite lively. I gave a "Tricking Out the Linux Kernel Networking Stack" talk last year, and it was helpful to have people let me know which slides were too simplistic, and which material was controversial. Unlike many meetups, the group is is very experienced, and can help put "new hotness" technologies in contrast to other solutions. It's really important to know when and why to use technology, and how it can work in concert with other techniques.

devops: per-process swap usage

awk '/Name/{name=$2}; /VmSwap/{printf "%20s\t%s %s\n",name,$2,$3}END{ print ""}' /proc/*/status

inspired from: adnans

Output:

     chromium-browse 19664 kB

     chromium-browse 6128 kB

        avahi-daemon 320 kB

        avahi-daemon 220 kB

              colord 644 kB

                smbd 840 kB