Big Data/Analytics Zone is brought to you in partnership with:

Alex Holmes is a senior software engineer with over 15 years of experience developing large scale distributed Java systems. For the last four years he has gained expertise in Hadoop solving Big Data problems across a number of projects. He is the author of "Hadoop in Practice", a book published by Manning Publications. He has presented at JavaOne and Jazoon and is currently a technical lead at VeriSign. Alex is a DZone MVB and is not an employee of DZone and has posted 23 posts at DZone. You can read more from them at their website. View Full User Profile

LZOP Decompression - Revenge of the Useless Cat

  • submit to reddit

For me LZOP is the ubiquitous compression codec with working with large text files in HDFS due to its MapReduce data locality advantages. As a result when I want to peek at LZOP-compressed files in HDFS I use a command such as:

shell$ hadoop fs -get /some/file.lzo | lzop -dc | head

With this command the output of a LZOP-compressed file in HDFS is piped to the lzop utility, where the -dcflags tell lzop to decompress the stream and write the uncompressed data to standard out, and the final headwill show the first 10 lines of the data. I may substitute head with other utilities such as awk or sed, but I always follow this general pattern of piping the output lzop output to another utility.

Imagine my surprise the other day when I tried the same command on a smaller file (hence not needing to use the head command), only to see this error:

shell$ hadoop fs -get /some/file.lzo | lzop -dc
lzop: <stdout>: uncompressed data not written to a terminal

What just happened - why would the first command work, but not the second? My guess is that this is likely the authors of the lzop utility safeguarding us accidentally flooding standard output with uncompressed data. Which is frustrating, because as you can see from the following example this is a different route than that which the authors of gunzip took:

shell$ echo "the cat" | gzip -c | gunzip -c
the cat

If we run the same command with lzop we see the same result as was saw earlier:

shell$ echo "the cat" | lzop -c | lzop -dc
lzop: <stdout>: uncompressed data not written to a terminal

A ghetto approach to solving this problem is to pipe the lzop output to cat (which is a necessary violation of theuseless cat pattern):

shell$ hadoop fs -get /some/file.lzo | lzop -dc | cat

Luckily lzop has a -f option which removes the need for the cat:

shell$ hadoop fs -get /some/file.lzo | lzop -dcf

It turns out that man page on lzop is instructive with regards to the -f option, indicates various scenarios where it can be helpful:

shell$ man lzop
-f, --force
   Force lzop to

    - overwrite existing files
    - (de-)compress from stdin even if it seems a terminal
    - (de-)compress to stdout even if it seems a terminal
    - allow option -c in combination with -U

   Using -f two or more times forces things like

    - compress files that already have a .lzo suffix
    - try to decompress files that do not have a valid suffix
    - try to handle compressed files with unknown header flags

   Use with care.

Published at DZone with permission of Alex Holmes, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)