Dariusz on Software Quality

Locate duplicates quickly: I mean only size+filename check, not expensive MD5 sum computation:

find . -printf "%f:%s:%p\n" -type f | \
    awk -F: '
        {
            key=$1 " " $2;
            occur[key]++;
            loc[key]=loc[key] $3 " "
        }
        END {
            for(key in occur) {
                if(occur[key]>1) {
                    print key ": " loc[key]
                }
            }
        }
    ' | sort

A bit of explanation of above magic:

• printf: tells find command to output file metadata instead of only file path (the default), this metadata (size, filename) will be used later
• -F: :We want to handle properly paths with spaces, that’s why special separator is used
• key=$1 ” ” $2: we use file name (without dir) and file size to create ID for this file
• occur: table (key -> number of file occurences)
• loc: maps file ID to list of locations found
• occur[key]>1: we want to show only files that have duplicates
• sort: results are sorted alphabetically for easier navigation

Sometimes you want to separate old files (for example older than 30 days) into separate subdirectory to make navigation easier. Using Linux tools it’s pretty easy task:

mkdir -p OLD; find . -maxdepth 1 -mtime +30 -exec mv \{\} OLD \;

Explanation:

• mkdir -p OLD: create OLD subdirectory if not exists
• -maxdepth 1: search only one level under current working directory
• -mtime +30: find only files that are older than 30 days
• -exec …. \;: for every file/directory found execute the following command
• \{\}: will be replaced by filename

Above command will move all old files/subdirectories into OLD subdirectory.