Wednesday, January 26, 2011

tips to delete huge - extremely huge no.of millions

I have around six million files (only files ,no sub-dir) to delete in UFS file system. Any tips to increase the performance?

  • Get the file names with ls -f or ls -U (if supported) to avoid having ls or your shell sort out the names. Just ls -f | egrep -v '\.|\.\.' | xargs rm -f. If this is a frequent necessity, you might want to write a small C utility to do it.

    Dennis Williamson : That skips *any* file with a dot in its name, perhaps you meant `egrep -v '^\.$|^\.\.$'`
    mpez0 : @Dennis Williamson - you're right. Though if the million files have generated names, it's likely that none start with dot.
    Dennis Williamson : Your grep is intended to eliminate the directories `.` and `..` from the output of `ls`, but it will also eliminate such generated filenames as `tmp.C9rDc96tca`
    From mpez0
  • Not for this time but in the future would it be possible for you to create them in a separate file system? this would at least give you the option of just wiping the whole FS if that were appropriate.

    From Chopper3
  • find /mydir -type f -exec rm {} \;

0 comments:

Post a Comment