Friday, December 12, 2014

awful, but useful, Python

Have you ever wanted to open and decompress all log files, streaming each line to the caller?

In Python, this is:

import bz2, sys
from itertools import chain

# set 'lines' to all lines of all files after they've been decompressed
lines = chain(*(bz2.BZ2File(path) for path in sys.argv[1:]))

# as a check, print the first 100 lines, then crash
lines = islice(lines, 100)
print ''.join(lines)