John Tells All

Ssh tunnels are great

2012-01-13T11:09:00.000-08:00

At work I'm connecting to multiple Munin statistics servers. On my lappytop I'm only running one server, so how do I get multiple sets of results? Answer: create a tunnel to another server!

The Munin protocol is extremely easy, send "fetch X" to get statistics on X. In my example df=Disk File usage. Here's how to get local information, via Munin running locally on port 4949.

$ echo 'fetch df' | nc -q1 localhost 4949

# munin node at freebeer

_dev_sda1.value 5.29318865322941

_dev.value 0.0339391645617528

_dev_shm.value 0.378827479751794

_var_run.value 0.00922469512382616

_var_lock.value 0

.

Here's how to make a remote machine's Munin (on port 4949) show up on localhost (port 4950). This means we can scan multiple local ports to get information on many different machines.

ssh -fNL localport/localhost/remoteport remotehost

Option "-f" means drop into the background after asking for a password. Next option "-N" is so the ssh connection doens't try to run anything remotely. The next bit actually creates the tunnel. It reads (L)ocal port 4950 maps to remotehost 4949. The "localhost" in slashes is in respect to the connected session -- from our perspective it's remotehost.

Here it is in context. Establish a tunnel, use it to get Munin Disk Filesystem information for the remote host.

$ ssh -fNL 4950/localhost/4949 myremotehost

$ echo 'fetch df' | nc -q1 localhost 4950
# munin node at myremotehost
_dev_xvda2.value 3.15090884943856
_dev.value 0.022631566185207
_dev_shm.value 0
_var_run.value 10.3138711271508
_var_lock.value 0
_lib_init_rw.value 0
.

http://www.engadget.com/2006/03/21/how-to-ssh-tunnels-for-secure-network-access/

backing up an Android device

2011-12-11T11:55:00.001-08:00

My beloved Cheese, a Samsung Galaxy S Android phone, has lots of photos on it. I want copies on my main Ubuntu computer. Also Cheese is complaining about running out of room, even though I'm down to 130 apps -- I want to examine the phone and delete hidden files taking up precious space. To do all this I'm installing a file server on the phone, then editing and copying things at my leisure from the main computer vs using the fiddly little phone screen.

1. Install Software Data Cable on Android. This is an FTP server for your phone, so you can easily see files on the phone and copy to/from your computers.
Another choice is the excellent ASTRO File Manager, which has a SMB (Windows network file system) module.

2.Start Software Data Cable. Press "Start Service"

3. In Ubuntu, open the URL given in step #2, like ftp://192.168.11.148:8888/ If all is well you'll see a listing of Android application folders, like "camera360".

4. To sync files, drag and drop using the above file manager, Nautilus. Or use "wget" which understands FTP. The following says "make a copy on my PC of all JPG images stored on the SD card on the phone. Keep all directories the same."

wget --mirror -nv --accept jpg ftp://192.168.11.148:8888/ .

If things don't look right try the above command and leave off "-nv" to make things more verbose.

Note that the above only gives you access to the "external SD card" parts of the Android phone. The other, "internal" flash are generally hidden. To access this area anyway you can enable it in Data Cable's settings area -- but this is sketchy and not recommended.

Using tmux to share a terminal | Richard WM Jones

2011-11-24T12:48:00.000-08:00

Using tmux to share a terminal | Richard WM Jones

Linux is awsome

2011-10-24T15:01:00.000-07:00

At work we process many millions of emails a day, so we get a lot of bounces. The following code helps me make the system smarter. It translates to "for the most recent five users who's received a bounce message, find all of their bounces in the last 24 hours and show me the Subject lines, with user and date information."

find `ls -1t | head -5` -type f -mtime -1 -print0 | xargs -0 egrep Subject /dev/null | less

Webex with Ubuntu Oneiric Ocelot

2011-10-19T14:25:00.000-07:00

Read elsewhere how to install all the needed dependencies. Here's a magic bit:

sudo add-apt-repository "deb http://archive.canonical.com/ natty partner"
sudo apt-get update
sudo apt-get install sun-java6-plugin

A final trick: I use Firefox for work stuff, and Chrome for personal tasks -- thus I don't have to deal with multiple profiles, nor does sketchiness in one browser affect the other.

I love Python (espeak speech synthesizer demo)

2011-09-30T12:21:00.000-07:00

#!/usr/bin/env python

'''
espeak.py -- demo each of the English espeak voices
'''

import os, re, time
from itertools import ifilter, imap

# Pty Language Age/Gender VoiceName       File        Other Langs
#  5  en             M  default           default     

pat = re.compile('\s* \d+ \s+ (en\S*) \s+ \S \s+ (\S+)',re.VERBOSE)
for m in ifilter(None, imap(pat.match, os.popen('espeak --voices=en'))):
lang,voice = m.groups()
if '-' in lang:
continue
print voice
os.system('espeak -v {0} "{0}... beer is good food" 2>/dev/null >/dev/null'.format(
voice))
time.sleep(3)

https://github.com/shavenwarthog/setup/blob/master/misc/espeak.py

Change Screen Brightness with Fn key in Ubuntu 11.04/10.10

2011-09-22T11:17:00.000-07:00

Change Screen Brightness with Fn key in Ubuntu 11.04/10.10

5 tips for packaging your Python projects « Fetchez le Python

2011-08-20T12:42:00.000-07:00

5 tips for packaging your Python projects « Fetchez le Python

link: Some Python Design Patterns

2011-07-29T11:08:00.000-07:00

Some Python Design Patterns « Christopher's Blog

adventures with the Nook Color

2011-07-18T12:11:00.000-07:00

I just got a Nook Color, and it's awesome. I wanted something to 1) read recipes in the kitchen, 2) have some fun with, and 3) use for Caustic and/or other music production.

I rooted it.

More on that later, but I thought I'd add some points for other prospective cheap tablet people:

1) many apps act a little awkward on the tablet. Specifically "Go Launcher" is not really usable. "Frogger" and "SPC Music Sketchpad" look strange but are okay. The Cynaogen Mod7-provided ADW.Launcher works very well.

2) it feels faster than my Samsung Galaxy S, but laggier. That is, sometimes touches aren't recognized when I'd like. It's unpredictable, a minor annoyance.

3) from Amazon Appstore: Autodesk Sketchbook Mobile, Fieldrunners HD, and iCookbook.

When you get a tablet, run to download a bunch of sexy cool apps:

- Google Earth (and Maps)

- Substrate live wallpaper

- Chumby. This gives you access to a *lot* of silly or beautiful useful clocks. This is much better than the Android-ish stash of clocks.

Overall for consumption of media (Gmail, maps, music) the Nook Color is tremendous fun. The lack of a camera, microphone, and 3G make the device feel a lot "quieter" than a cell phone. The battery life is also a bit better. Picking up the small Galaxy S now feels like I'm using a small powerful Swiss army knife, whereas using the Nook makes me feel more relaxed and comfortable.

A lot of fun!

The proper way to install pip, virtualenv, and distribute for Python

2011-06-30T12:52:00.000-07:00

The key bit to realize is that "virtualenv" installs "pip" and "distribute", which are the latest and bestest packaging tools for Python.

(optional) add these to ~/.bashrc:
PIP_REQUIRE_VIRTUALENV=true
PIP_RESPECT_VIRTUALENV=true

1. install system-level virtualenv

sudo aptitude install virtualenv

2. use it to create a clean "distribution" centered around a private Python executable and your own packaged scripts

virtualenv --no-site-packages --distribute ENV

3. activate it for this shell

. ENV/bin/activate

This modifies the local PATH, so now "python" is your private one. Add/remove packages to your heart's content without messing with the system-global one.

which pip
 #  /home/johnm/work/ex-package/ENV/bin/pip

4. verify by installing a package from the internet

pip install coverage
python -c 'import coverage'

5. verify by installing a local package

pip -v install Mypackage-0.1dev.tar.gz
python -c 'import mypackage'

Getting started with Python « Endlessly Curious

2011-06-14T11:00:00.000-07:00

excellent pointers to get started programming in Python: Getting started with Python « Endlessly Curious

Study Hacks » Blog Archive » Anatomy of an A+: A Look Inside the Process of One of the World’s Most Efficient Studiers

2011-05-19T11:30:00.000-07:00

learn quickly by connecting ideas, not memorizing facts

Anatomy of an A+: A Look Inside the Process of One of the World’s Most Efficient Studiers

I salute Fabrice Bellard

2011-05-18T12:11:00.000-07:00

Recently the amazing hack of "boot Linux in a web page" has been circulating around. Bellard wrote a x86 emulator in JavaScript, then took the straight Linux binaries and was able to boot it. In a web page. At reasonable speeds. His distribution even includes a C compiler. This is amazing, but that's not all.

Bellard wrote the emulator QEMU, which lets you run one set of software "inside" another. It's mostly used for virtual machines, so you can run multiple operating systems inside a "host" OS. The guests can run Linux or Windows or whatever. The guest virtual machines can even be of different CPU types: ARM on a x86! This can be incredibly useful. But that's not all.

A large majority of video software uses the "ffmpeg" library. It's quite low level, but lets you do conversion of different screen sizes, bit rates, and codecs. Also written by Bellard, and now extended by a team for the last few years.

Among his other hacks are TCC, a tiny C compiler, which lets you use C as a scripting language. It's fast and complete enough to be used as a boot loader. That is, you can "boot" TCC, which then compiles and runs the Linux kernel. The entire thing. It recompiles on the fly.

Lastly he has some software to turn your computer into a TV transmitter. This is impossible, but he's found a way to do it by tweaking VGA video registers and timings and so forth.

I salute you, Fabrice Bellard.

Fabrice Bellard's Home Page

Dell Vostro V13 review | Linux User

2011-05-15T12:00:00.000-07:00

I just bought one of these -- it's pretty good, an expandable netbook.

Dell Vostro V13 review | Linux User

Hidden Treasures of the Python Standard Library

2011-05-13T12:25:00.000-07:00

Hidden Treasures of the Python Standard Library

Python Workshop slides now available!

2011-05-05T16:53:00.000-07:00

http://173.231.53.155/2011PythonWorkshop/

X keyboard binding

2011-04-30T14:26:00.000-07:00

I have a tiny IOGear wireless keyboard. It's great: about 12" wide, 6" deep, with a little trackball. I can sit eight feet away from my big TV and work. When it's not needed, I can easily tuck it out of the way.

One big drawback: no native Home nor Page Up or Down keys. This is rather awkward: to scroll down a web page I always have to look carefully at the little blue "Fn" key on the left side of the keyboard, then carefully find the down arrow on the right side of the keyboard, then press and release them both. I'd rather not have to actually look while reading very important web pages, so I found a solution.

My window manager "wmii" uses the Super key to switch workspaces and move windows around. The Alt key is used by some apps. I'd like to have both Fn-Down and Super-Down map to PageDown, so it's not as awkward to scroll web pages.

To solve this, there are a lot of twisty little utilities, all slightly similar and different. Xev shows raw X events, and can tell you if your keys (like multimedia keys: Volume Up and so forth) are recognized by the raw X server. Xmodmap lets you remap events, like swapping CapsLock and Alt. Xbindkeys permits commands to run on key sequences, like pressing Alt-T to run a terminal. Xmacro lets you record and play back sequences of raw X events, including things like PageDown.

Lastly there's "xdotool", which sends key, mouse, or window manager events. For example, to scroll inside the current terminal, type:

xdotool key Shift+Page_Up

First version of ~/.xbindkeysrc:

"xdotool key Page_Up &"
  Mod4 + Up
"xdotool key Page_Down &"
  Mod4 + Down

This sort of works, but often fails: the window will flicker but not scroll. Second version searches for the topmost Chrome window, then sends the code to it. That doesn't work either...

To be continued.

Minimum Font (Chrome Extension) - Home - Open wiki - Gitorious

2011-04-06T11:07:00.000-07:00

This extension makes website fonts large enough to read -- very useful when using a big TV as a monitor.

Minimum Font (Chrome Extension) - Home - Open wiki - Gitorious

network management

2011-03-24T12:59:00.000-07:00

I'm firmly in the "NetworkManager is the devil" camp, because it messes up my network configuration without triggering a loud error in syslog or somewhere. Grr.

I'm now using wicd to manage my wireless/ethernet networks.

Note, to use wicd in a non-systray window manager (wmii), type "wicd-client -n &"

Lastly, "ifconfig" is deprecated(!). Use "ip" instead -- http://en.wikipedia.org/wiki/Iproute2

"ifconfig" => "ip a"
"route" => "ip r"

http://andys.org.uk/bits/2010/02/24/iproute2-life-after-ifconfig/

EasyTether internet tethering for Android on Debian

2011-03-04T13:52:00.000-08:00

My local cable provider likes to mess with me, so on occasion I get internet via my cellphone. In practice this works rather well. Install EasyTether on the phone and on your Debian machine. Connect via USB, then run some commands on the Debian side to route packets via your cellphone.The software is designed to be run standalone, not as part of the normal Linux system, which is a little weird. The following stanza helps with this. If you add the following to /etc/network/interfaces, you can turn on/off the tethering using the normal ifup/down commands:

iface easytether0 inet dhcp
      pre-up easytether connect &
      pre-up sleep 1
      pre-down killall easytether

That is, after the above is added, do "sudo ifup easytether0" to route packets through the USB-connected Android device. To turn it off, do "sudo ifdown easytether0".This solution is rather buggy. Disconnect and reconnect the USB cable to solve some problems. Also, by default EasyTether doesn't route ICMP packets, which means you can't use "ping -c1 4.2.2.2" to verify your internet connection is up. "curl google.com" is a good check instead.Note that EasyTether Lite doesn't route https, which means Gmail and Facebook doesn't work. To get around the latter, use the mobile site: http://m.facebook.com The full software is $10 and has been a fantastic value for me -- highly recommended.http://www.appbrain.com/app/easytether-lite/com.mstream.easytether_beta

video: installing drivers and tweaking DPI

2011-02-16T14:22:00.000-08:00

With the new kernel, the nVidia drivers can be installed. I didn'thave luck with the free nouveau drivers, so went with the proprietarynVidia setup. Download it, then:

aptitude install gcc make binutils linux-headers-2.6.37-trunk-686
sh ./NVIDIA-Linux-x86-260.19.36.run

It worked! But... fonts are way too large. At the login page the"user" and "password" fields have letters so big they overflow thespace. Terminal windows are unusable. It looks terrible.

I'd already seen this and had found a fix. The Samsung TV lies like arug to the computer about how big it is. So, on the Linux side justignore the TV's boasts and set the DPI manually.

File /etc/X11/xorg.conf, add text in bold:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GT 430"
# JM:
    Option         "UseEdidDpi" "FALSE"
    Option         "DPI" "100 x 100"
EndSection

Restart X to see the changes. For my system, this was:

service gdm stop ; service gdm start

video: nVidia GT 430 on Debian, upgrading the kernel

2011-02-16T14:18:00.000-08:00

For my new media computer, I chose a nVidia GT 430. It's cheap at $80,small, and doesn't require big power connectors used by my older gamercard. It's also another nVidia card, which has some level of supportfor Linux.

Debian Squeeze runs Linux 2.6.32, but the 430 is new enough that itrequires newer drivers to work well. I wasn't looking forward todicking with the kernel and/or X settings just to be able to use mycomputer. It's easy to spend hours or days tweaking things that mighthelp, or might not.

For additional challenge, the video card takes over the audio. ThenVidia card can transmit both video and audio over the same HDMI cableto the TV. This is cool because I can control computer audio via theTV remote. However this means I'd switch from my nice Mackie studiomonitors to the little speakers inside the TV.

So, under normal Debian with the new GT 430, graphics are old VESAstyle: low resolution and sluggish. And, no audio at all -- notthrough the HDMI connector, nor through the motherboard's audio port.But the video is usable.

To verify Debian with new kernel would work with the 430, I found aderivative: PureOS. Installing to a USB flash drive didn't workvery well, but it gave me enough confidence to redo the kernel in mymain Debian hard drive. http://pureos.org/index.php?lang=english

Upgrading the kernel turned out to be rather easy and not risky. It'sin two steps:

1a) "pin" the system so it sees both stable and unstablepackages. It'll use stable most of the time, but the newer files areavailable if needed.

File /etc/apt/preferences, add the text in bold:

Package: *
Pin: release a=statler
Pin-Priority: 1001

Package: *
Pin: release a=squeeze
Pin-Priority: 100

# JM:
Package: *
Pin: release a=experimental
Pin-Priority: 69

1b) tell the system where unstable packages are located

File /etc/apt/sources.list, add text to the end:

# JM:
deb http://ftp.debian.org/debian/ experimental main contrib non-free
deb-src http://ftp.debian.org/debian/ experimental main contrib non-free

2) install the new kernel. It automatically adjusts the boot managerso that you can use either the new or old kernel when the computerboots.

aptitude update
aptitude install linux-image-2.6.37-trunk-686
reboot

With our system running Debian with an upgraded kernel, we're now ready to install the nVidia drivers, thus making our video card happy

new media computer

2011-02-16T14:10:00.000-08:00

My old computer is a bit of a mess. Over three years it's accumulated a few hard drives, with cables strewn about. A new computer isn't in my budget, but I'm tired of looking at the full-size computer case in my living room.

Thus, my media PC project. I bought a case small enough to fit in the media cabinet under my 46" tv/monitor, which will contain the guts of the old computer. Then the old case will vanish, and my living room will look great! No more cables multiplying like rabits, languidly lazing about, collecting dust and tripping my Roomba.

This project is in several steps:

1) find the right case. This was rather harder than I'd expected.

2) replace video card. My old gamer video card is too big and
power-hungry.

3) rearrange hard drives. New case only has room for two.

4) move motherboard from old case to new, without a hammer.

5) what the hell, let's change operating systems.

"Danger: Emacs"

2010-03-21T14:11:00.001-07:00

http://www.flickr.com/photos/hackaday/3841760416/

heh

HOW TO SET UP REMOTE CONTROL FOR LINUX (lirc)

2010-03-21T14:03:00.000-07:00

Using a infrared (IR) remote control for Linux consists of getting
several pieces to be configured correctly, and to talk to each other
correctly.

STEP 1 -- install Lirc

apt-get install lirc

STEP 2 -- configure your remote

In this case, I have a Microsoft Media Center remote. Therein the problems start. It
has a Vista logo, but is marked "HP" on the bottom, and is recognized
as a "Philips" USB device:

$ lsusb
Bus 005 Device 002: ID 0471:060c Philips

To tell Lirc which remote you have, type:

$ sudo dpkg-reconfigure lirc

and follow the prompts. This changes settings in /etc/lirc/lircd.conf
and probably other files.

To verify this worked, use the "show cooked codes" command and press
keys on your remote. To quit, press control-C.

$ irw
000000037ff07bed 00 ChanUp mceusb
000000037ff07bed 01 ChanUp mceusb
000000037ff07bf2 00 Home mceusb
000000037ff07bf2 01 Home mceusb

At this point, your remote works, and your computer is correctly
translating the bitstream into keys.

STEP 3 -- translate keys into actions

Add this to your ~/.lircrc file:

begin
   prog = irexec
   button = Home
   config = echo "Hello world!"
end

To test this, run the "how does program X translate key Y" command,
ircat. In this case, we're intercepting the irexec command:

$ ircat irexec

echo "Hello world!"

Once the translation step works correctly, run it for real:

$ irexec

Hello world!

STEP 4 -- translate keys into "smart" actions (VLC media player)

Change your ~/.lircrc file into this:

begin
   prog = irexec
   button = Home
   config = echo "Hello world!"
end
begin
   prog = vlc
   button = Home
   config = key-stop
end

Start irexec again, and also VLC on a video:

$ killall irexec
$ irexec &
$ vlc mymovie.avi

Pressing "Home" should stop the video, but it doesn't work -- VLC has
to know to subscribe to Lirc events:

$ vlc --control lirc sample.avi

When Home is pressed on the remote, Lirc sends the event to *all*
applications that are interested. In this case, pressing Home prints
"Hello world!" in the terminal, and the movie stops.

The Lirc service, lircd, grabs infrared data from the USB device and
translates it to button names. In this case, it knows I
have a Media Center remote "mce", so the IR squiggles it got when I
pushed the Home button, it understands as "the user pressed the Home
button".

Each application subscribes to Lirc events. The lircd service
translates specific buttons for specific applications. In this case,
the "vlc" media player received the "key-stop" message, and it stopped
the video. The "irevent" application also received a message, in this
case the "echo hello world" string, so it printed a message to the
terminal.

If something goes wrong, look at each part of the setup:

- IR signals to buttons with lircd

"irw" displays buttons

- buttons to actions

   "ircat vlc" shows actions for a specific program, in this
case vlc

"irexec" does things directly, like printing messages

Different applications understand different messages. Many programs
don't understand remote buttons at all, so an intermediate programs is
used to do the translation.

STEP 5 -- translate buttons into "dumb" mouse/keyboard events (Google Chrome)

Add this to ~/.lircrc:

begin
prog = irxevent
button = Home
config = Key CTRL-t CurrentWindow
end

Test the mapping:

$ ircat irxevent

Key CTRL-t CurrentWindow

Now run irxevent directly. It intercepts the Home button, then sends
a keypress to the current window. In this case, Control-T on a Google
Chrome window opens up a new tab.

$ killall irxevent ; irxevent

Hope this helps!

- john

REFERENCES:

- lircd.conf, mapping buttons to actions:

http://www.lirc.org/html/configure.html#lircd.conf

- keys for Google Chrome:

http://www.google.com/support/chrome/bin/answer.py?hl=en&answer=95743

- VLC

http://en.gentoo-wiki.com/wiki/VLC/LIRC

- my setup:

http://github.com/shavenwarthog/setup/blob/master/lircrc

Message Queuing slides and source code!

2009-11-20T10:03:00.000-08:00

I gave a talk at the local Python user group, SoCal Piggies. We discussed message queues: they're useful for synchronizing processes, increasing reliability and scalability. The entire source tree, with slides and extra goodies, is linked below. Enjoy!

https://bitbucket.org/shavenwarthog/amqp-ex/src/

(diagram credit: RedHat)

Superdevelopers at Facebook

2009-11-16T09:14:00.000-08:00

Facebook gets unprecedented traffic. They don't want things to go down so they... release changes constantly, a few times a day! They use peer review and their immense audience to see how well their solutions work, then iterate. The other teams *support* engineering, rather than being gatekeepers.

I actually did this at a medium-sized site many years ago. Push change to live, capture logs for five minutes while people beat it, then undo the change. It worked well.

ntop and RRD

2009-11-10T10:49:00.000-08:00

"ntop" conveniently displays web pages with charts and plots of your local network traffic. To record historical traffic, it uses Round Robin Database (rrd). The default ntop package enables the RRD plugin, but doesn't give itself permission to write out its data. Here's how to get the two to work:

sudo apt-get install ntop
sudo install -d /var/lib/ntop
sudo install -o nobody -d /var/lib/ntop/interfaces
sudo install -o nobody -d /var/lib/ntop/rrd

To test, start "sudo ntop" (probably in "screen"), then open the web page http://localhost:3000

To activate the RRD plugin, select Plugins / Round Robin / Describe, then click the red "no" to become a blue "yes." You'll see messages in the terminal where you started ntop.

After a few moments, you can view RRD graphs. One is in Summary / Network Load.

Fast Parallel Downloading (for apt-get and aptitude) #3

2009-10-26T11:58:00.000-07:00

"aptitude" has more features, but is as low-hassle and scriptable as "apt-get". For instance, you can say "this package was automatically installed", which means if you don't need it any longer, it'll be automatically removed. Another perk is a direct way of asking "why was/wasn't this package installed?", to help sort out sometimes confusing package dependencies.

Alas, it doesn't have the "print-uris" trick which I relied upon in doing quick upgrades. I've found a simple workaround: use both aptitude and apt-get. The following script updates the list of available packages. It uses apt-get to download -- but not install -- all updates. Lastly, it uses aptitude to upgrade the packages, using the freshly-downloaded files. A "safe" upgrade won't delete a package to let another package be installed correctly, it just does installs.

For a one-line version, see morch's a previous comment.

cd /var/cache/apt/archives/
aptitude update
apt-get -y --print-uris -u upgrade | grep -o -e "http://[^\']+" \
| xargs -r -l3 -P5 wget -nv
aptitude -y safe-upgrade

Bits of Evidence

2009-10-25T17:54:00.001-07:00

Slideshow adding Science to "Computer Science". Includes references to a lot of myths like "best programmers are 45x better than the worst ones."

Bits of Evidence

View more presentations from Greg Wilson.

turning batch MapReduce into infinitely streaming data wrangling

2009-10-20T14:45:00.001-07:00

Pipelining and Real-time Analytics with MapReduce Online

Comparison of Scrum, XP, Kanban, and other Agile Methodologies

2009-10-11T13:49:00.000-07:00

State of Agile

Empirical Data behind Software Development Techniques

2009-10-09T10:40:00.000-07:00

Our friends at Microsoft Research have published four papers testing--and measuring--development ideas. One big surprise was that a company's organization "can predict software failure-proneness with a precision and recall of 85 percent. " It's more important than using assertions or increasing test coverage!

Other results: test coverage isn't a direct measure of software quality, and assertions are useful.

I was glad to see some numbers behind what I've experienced, that Test Driven Development increases quality. "What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer." Hmm.

Exploding Software-Engineering Myths

Unit testing in Coders at Work

2009-10-06T12:20:00.000-07:00

Very interesting essay on how master programmers (including Jamie Zawinski) write. Includes a TDD and traditional Sodoku solvers, with completely different implementations.

Unit testing in Coders at Work

Fast Parallel Downloading (for apt-get) #2

2009-10-06T12:14:00.000-07:00

This is how to upgrade your Ubuntu system, downloading several packages in parallel. See previous article for details:


cd /var/cache/apt/archives/
apt-get -y --print-uris -u dselect-upgrade >debs.list
egrep -o -e "http://[^\']+" debs.list | xargs -l3 -P5 wget -nv
apt-get -u dselect-upgrade

Programs matter more on programmers, than the language

2009-09-18T10:32:00.000-07:00

A study in 2000 showed a very wide range of performance for different programmers, more so than the language used. C/C++ tended to be fastest, but not by much. C/C++ and Java had a wide range of variability -- a programmer's skill was more important here. With Perl, for some reason, skill didn't matter as much.

Never mind the language, the programmer is what matters | Psychology of Programming

Ultimate Quality Development System

2009-09-14T09:28:00.000-07:00

Our friends at Divmod have come up with an extremely clear method of rapidly delivering high quality software. The basic idea is you write a bug report, create a code branch, then check in the branch with a "review" tag. When someone reviews and okays your code, you merge into the trunk. This method has lots of benefits, including taking the drama out of peer review, and making it very easy to safely test on multiple platforms.

http://divmod.org/trac/wiki/UltimateQualityDevelopmentSystem

(via Twisted)

Using a HDTV as a Monitor

2009-09-10T17:33:00.000-07:00

I sit across the room from my 46" TV and use a wireless keyboard. It's great! Alas, some Gnome and other Linuxy programs get confused by the TV's size and dimensions, so they display HUGE FONTS. On some programs, but not others. The last straw was when Emacs23 came out and I couldn't use it. But, now, it's all fixed!

Here's the magic jibberish from /etc/X11/xorg.conf:

Section "Monitor"
       Identifier      "Samsung" # LNA650

# from Samsung manual, for 1920x1080p:
        HorizSync    67.500
        VertRefresh  60.000
# http://www.mythtv.org/wiki/Specifying_DPI_for_NVIDIA_Cards
 Option    "UseEdidDpi"  "FALSE"
        Option "DPI" "100x100"
EndSection

This stanza gives you the old-school behavior so you can quickly test the X configuration. Hit Control-Alt-Backspace to restart the X server:

Section "ServerFlags"
# re-enable Control-Alt-Backspace to restart X server:
       Option          "DontZap"               "false"
EndSection

Thanks, Aun! http://ubuntuforums.org/showthread.php?t=1181631

EDIT: added "UseEdidDpi" line, to turn off the buggy DPI sensing, and use DPI=100 instead.

Python easy_install, really

2009-06-13T12:45:00.000-07:00

Python's easy_install, at least under Ubuntu, is neither easy, nor does it install stuff if you're not root.

Here's the shortcut:
- make your local packages directory. This means you can easily zap all your "I'm playing around with it" libraries and apps without damaging the system.
- tell Python (and easy_install) to search it
- also, tell easy_install to install packages into it

$ mkdir -p $HOME/local/lib/python2.6/site-packages
$ export PYTHONPATH=$HOME/local/lib/python2.6/site-packages
$ easy_install-2.6 -d $HOME/local/lib/python2.6/site-packages pyevolve

If the automatic network search doesn't work, then manually download the Egg and try again:

$ easy_install-2.6 -d $HOME/local/lib/python2.6/site-packages Desktop/Pyevolve-0.5-py2.6.egg

Then test it:

$ python -c 'import pyevolve; print pyevolve.__version__'
0.5

Note you'll have to add the PYTHONPATH into your ~/.bashrc, else you'll get errors like this:

$  (env -i !!)
(env -i python -c 'import pyevolve')
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named pyevolve

Kanban Development Oversimplified

2009-06-05T14:05:00.001-07:00

I loved this article It's clearly written, with a brief descriptions of the large numbers of tradeoffs within the Agile ecosystem. Specifically, choosing a different size length of time to develop each feature makes a huge difference. I've been using very small, one-day timeboxes. The article notes that larger boxes, four weeks, can often help the client make meaningful, business-oriented decisions vs going over each bit of minutia.

taxonomy of trees/sets/lists? automatic parallelization?

2009-06-04T07:43:00.000-07:00

On slide 59, Guy Steele shows how to automatically translate serial into parallel code!:

http://groups.csail.mit.edu/mac/users/gjs/6.945/readings/MITApril2009Steele.pdf

Download and extract a tarball in one step

2009-05-28T14:23:00.000-07:00

wget -O - (URL) | tar xvzf -

Cheap Network Measurement

2009-05-18T20:19:00.000-07:00

You can't control what you can't measure. I want a fast local wireless network, counting only from my workstation to the router. If I change settings or drivers, how can I tell if it's an improvement or not? How can I gauge my network speed, including driver and network settings?

There are jillions of tools available*, but they mostly 1) require a specific client on the server, or 2) don't actually send traffic, they passively analyze what's there.

Here's a solution: transmit a largish file to the server, where it will ignore the data. Measure the network bandwidth locally using 'cpipe' or another tool.

Setup: enable SSH on the router, install "cpipe" locally.

cat /boot/vmlinuz-* | cpipe -vt | ssh router 'cat >/dev/null'

I get lines like this, with statistics on each of many chunks of data:

thru: 280.736ms at  455.9kB/s ( 288.7kB/s avg)  768.0kB
thru: 330.929ms at  386.8kB/s ( 299.6kB/s avg)  896.0kB
thru: 315.523ms at  405.7kB/s ( 309.7kB/s avg) 1024.0kB

This version buffers 10,000KB into a single chunk before sending it over the network, giving a slightly more accurate bandwidth number:

cat /boot/vmlinuz-* | cpipe -b 10000 -vt | ssh router 'cat >/dev/null'
thru: 13908.225ms at  411.6kB/s ( 411.6kB/s avg)    5.6MB

*: http://www.ubuntugeek.com/bandwidth-monitoring-tools-for-ubuntu-users.html

Hassle-free Adobe Flash on Ubuntu 64-bit

2009-05-14T12:37:00.000-07:00

sudo apt-get install flashplugin-nonfree

Of course, install Firefox Flashblock so movies don't autoplay: http://flashblock.mozdev.org/

network monitor?

2009-04-22T15:46:00.001-07:00

Dear Lazyweb, I want a web-based display of everything that's happening on my network. This is easy, and there are dozens of apps for this, however I also want:

- easy to install. "apt-get install beer && firefox http://localhost/beer" should do it.

- default install doesn't ask me for a MySQL root password, or have me edit a something configuration something plugin something, I just want to see roughly what it looks like and does.

- default install should show me something about the local machine (and hopefully, the local network) w/o editing a zillion configuration files.

- hopefully web-based, but a X11 application is good too.

Ideas? Here are some that fail the test:
- Nagios. Lots of stuff, but fails the "show me now" test.
- Munin. A little lighter than Nagios, but similarly uncommunicative.
- Cacti. Well, maybe I'll give this one another five minutes to find the URL.
- Monit. Awesome *very* fast and light web-based tool, alas lacks pretty graphs. It's a distributed process-control tool.
- Tomato. Awesome, very pretty, lots of interactive graphs. Only monitors itself, the router.
- Gkrellm. Fast graphs, not ugly. Only monitors one machine at a time.
- Conky. Similar to gkrellm(?)

Suggestions?

Date and Time in Ubuntu

2009-04-22T10:43:00.000-07:00

This lets you interactively set your time zone:

dpkg-reconfigure tzdata

This sets the clock:

ntpdate ntp.ubuntu.com

Server names in Dnsmasq

2009-04-22T09:06:00.001-07:00

My domain name service (DNS) is running on a router using the wonderful Tomato firmware. When a computer boots up, it sends its hostname (i.e. "sally") to the router, which records the hostname and returns the corresponding IP ("192.168.7.10").

This is all grand. But, what if different services require a computer to have multiple names? That is, when the Puppet service fires up on my lappytop, it looks for the hostname "puppet" by default. Given the above scheme, a computer has one and only one name; DNS doesn't know that "sally" is also known as "puppet".

Here's how to fix that. The router is running Tomato, which uses Dnsmasq as the DNS service. It automatically figures out the normal hostname/IP mapping, for instance "sally" in the above example. We can tell it to point other names, like "puppet", to individual IP addresses.

Open up the Tomato administration page. Click Advanced / DHCP-DNS, and paste this into the Custom Configuration block after enabling all three checkboxes:

bogus-priv
expand-hosts

domain=lan

# service names, these are in addition to /etc/hostname
# for each computer:
address=/puppet.lan/192.168.7.13

To test:

$ ping -c1 puppet
PING puppet.lan (192.168.7.13) 56(84) bytes of data.
64 bytes from spootoxin.lan (192.168.7.13): icmp_seq=1 ttl=64 time=0.077 ms

--- puppet.lan ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.077/0.077/0.077/0.000 ms

Notice the system translates "puppet" to "puppet.lan" -- these are different names! The results for "dig puppet" and "ping puppet" can be different.

Check /etc/resolv.conf :

domain lan
search lan
nameserver 192.168.7.1

Thanks to http://ponderer.org/tomato_firmware for the tip!

Fast Parallel Downloading (for apt-get)

2009-04-18T21:14:00.000-07:00

I'm rebuilding a Ubuntu server. Normally apt-get downloads one file at a time, which can get dull when you're installing 598 files. I found the tool "apt-fast" which downloads one or two files quickly, by downloading with multiple streams per file. This is somewhat sketchy, as it requires installation of additional software, assumes the file gets spliced together correctly, and doesn't gracefully handle network problems.

I have a solution: xargs

Xargs walks on water. It is incredibly useful. In a nutshell, it runs a single command on a list of files. I'll post a lot more later, but here's how to speed up apt-get:


cd /var/cache/apt/archives/
apt-get -y --print-uris install ubuntu-desktop^ > debs.list
egrep -o -e "http://[^\']+" debs.list | xargs -l3 -P5 wget -nv
apt-get -y install ubuntu-desktop^

Replace "ubuntu-desktop^" with whichever task or package you want. Since ubuntu-desktop is a task, a huge collection of packages, the "^" on the end is required (and magic).

The options say to take three packages into a batch (-l3), and download five batches at a time in parallel (-P5). These settings are arbitrary, but provide a nice speedup while also not hammering the Ubuntu repository servers too hard.

Makefile for Python testing

2009-04-08T13:49:00.002-07:00

This version supports Emacs's awesome Flymake syntax checker:

RECENT_PY := $(shell ls -1t *.py | awk '!/flymake/ {print; exit}')

all test test-fast:
 ./$(RECENT_PY)

Agile Software Development

2009-04-07T12:58:00.000-07:00

I suggest everyone run, not walk, to read Alistair Cockburn's "Agile Software Development (2nd edition)." He talks about how people learn, and how Agile turns software development into a "cooperative game", like rock climbing. That is, developers have different roles, and each role supports the others so the team can progress farther, producing more software features to deliver to the client.

The author's notes on how people learn helped me immediately in my teaching practice. My classes are both more accessible (can help more people) and more focused (presenting more sophisticated knowledge.)

(Note the 2nd ed is a bit different from the 1st, and so recommended.)

modest makefile for rapid Java testing

2009-04-02T13:48:00.000-07:00

This makefile is designed to help test several Java programs, each one in its own self-contained source file, for example "hello.java" If you type "make hello.test", the program will be compiled and executed. To test everything in the directory, type "make test-all". To rebuild everything then re-test, do "make clean test-all".

To test a new program, just start writing it. When "make test-fast" is executed, the class file will be built, then run. That is, you never have to update the Makefile -- it picks up the most recent Java source file automatically.

I bind "make test-fast" to the big keypad Enter key, but that's for another blog post.

RECENT_JAVA := $(shell ls -1t *.java | head -1)

all test-fast: $(basename $(RECENT_JAVA)).test

test-all: $(patsubst %.java,%.test,$(wildcard *.java))

%.class: %.java
 javac $<
%.test: %.class
 env CLASSPATH=.:$(CLASSPATH) java $(basename $<)

check-syntax:
 javac -Xlint $(CHK_SOURCES)

clean:
 rm *.class

don't let network/disk hogs take over

2009-04-02T13:12:00.000-07:00

Long-running or big I/O programs -- disk copies or network downloaders -- can make your system sluggish. To remedy this, tell the Linux kernel that those processes should only hit the disk if the system is idle, that is in scheduling class "3".

NOTE: this command, like most examples, is safe. It only prints out what would be done. To actually run this and other shell examples, remove the "echo" statement.

pgrep bash | xargs -i --no-run-if-empty echo ionice -c3 {}

moving old directories out of the way

2009-04-02T12:52:00.000-07:00

This oneliner moves all non-2009 directories into a subdirectory:

mkdir OLD
echo mv `ls -1tl | awk '!/2009-/ {print $NF}'` OLD/

show Java problems in realtime in Emacs

2009-04-02T12:44:00.000-07:00

after installing Flymake, create this Makefile:

all:

check-syntax:
 javac -Xlint $(CHK_SOURCES)

art for William S Burroughs!

2009-04-02T12:34:00.000-07:00

Track 16 Gallery

Overview

2009-03-15T13:20:00.001-07:00

This blog is for technical and personal posts. Over different jobs over the years I've written hundreds of articles, and now it's time to share with everyone!

Los Angeles is filled with arty wonderfulness, and I'll share them too.