High-Level Architecture for HDFS Slurper V2

The current HDFS Slurper was created as part of writing “Hadoop in Practice,” and it just so happened that it also happened to fulfill a need that we had at work. The one-sentence description of the Slurper is that it’s a utility that copies files between Hadoop file systems. It’s particularly useful in situations where you want to automate moving files from local disk to HDFS, and vice-versa.

While it has worked well for us, with the addition of a few choice features it could be even more useful:

• Filter and projection, to remove or reduce data from input files
• Write to multiple output files from a single input file
• Keep source files intact

As such I have come up with a high-level architecture for what v2 may look like (subject to change of course).