Performance Zone is brought to you in partnership with:

Gary Sieling is a software developer interested in dev-ops, database technologies, and machine learning. He has a computer science degree from the Rochester Institute of Technology. He has worked on many products in the legal and regulatory industries, having worked on and supported several data warehousing applications. Gary is a DZone MVB and is not an employee of DZone and has posted 62 posts at DZone. You can read more from them at their website. View Full User Profile

Moving files and folders into hashed subfolders

04.25.2013
| 1890 views |
  • submit to reddit

The following will move a series of files into subfolders. It hashes the file names, building a two character, two folder deep hierarchy to split the files, e.g. a1/b2. The motivation for this is to split 500,000 folders into a manageable hierarchy, to avoid file system limits – in NTFS this is quite slow, and well over the 64,000 limit in ext4.

vi f
for f in `ls pacer/`; do echo -n $f | md5sum | sed 's`\(..\)\(..\).*`\1/\2/`' | tr -d '\n'; echo $f; done;
./f > t
mkdir ../pacer2
./t | sed 's`\(..\)/\(..\)/\(.*\)`mkdir pacer2/\1\nmkdir pacer2/\1/\2\nmv pacer/\3 pacer2/\1/\2/\3`' > dirs
./dirs




Published at DZone with permission of Gary Sieling, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)