Weed out duplicate files and some tips for Windows 7

We’re going to take a look at files in XP that are duplicated – ie more than one copy is stored on your hard disk. First, files can have the same name but not be duplicates. If you do a search for ‘desktop.ini’ or ‘thumbs.db’ you’ll probably find hundreds, but they will all have different content. The former store folder settings, and the latter thumbnail views of images in a folder.

Reader Alan Harrison used the Duplicate File Finder that comes with the free version of Glary Utilities and found he had more than 2,600 duplicated files on his hard disk.

So I ran the same utility on my C: drive looking for files of the same name, size and time and found 2,445. However further investigation showed that 600 of these were in the System Volume Information folder. In other words they were components of different Restore Points.

Each Restore Point is a standalone entity, so if certain aspects of a system have not changed between their creation, the corresponding files will be identical. It was then I found that the utility was somewhat flawed, as – and you may have already spotted this – those System Restore files will not have the same time stamp.

Furthermore, repeating the scan with the System Volume Information folder excluded, but with all other parameters the same showed 2,231 duplicates, rather than the expected 1,845. Time to get a second opinion, I felt.

Download the free Windows XP SP2 Support Tools and make sure you install the full version, which will get you many tools as obscure as a left-handed monkey wrench or a metric pickaxe, but the one we want is Dupfinder.exe which you can launch from the Start Menu Run box. This offers fewer options than Glary, and takes far longer, but found a total of 33,617 duplicates consuming 6,006,932KB. Yes that’s around 5.7GB.

They were kidding! Well yes, they were, as it was treating various hefty Thunderbird Inbox files – under different accounts and of different sizes as duplicates. So I installed yet another utility named Duplicate File Finder, which found 13,679 duplicated files wasting 1.81GB.

So how do all these duplicates arise – if indeed they are duplicates? Unruly application installations account for some – several applications install fonts, for example, in their rightful place in the Windows\Fonts folder, but leave copies in the program folder.

Java loves to proliferate, but I found that Thinkfree Office had installed another set of Java Runtime files in its own program folder. Quicktime and Flash also seem to love duplication and you’ll find that there are duplicates of libraries and resources in different user profiles.

And then, of course, there’s Windows itself. You’ll find, for example, copies of DLLs in Windows\System32 duplicated in System32\DLLCache. This is meant to be so and is an essential part of Windows File Protection. Other system files will be duplicated in Service Pack folders, Uninstall folders and elsewhere. Microsoft’s Dupfinder found a boggling 6,160 duplicated DLLs occupying 1.4GB.

So what do we advise? You could spend many unhappy hours tracking down and removing duplicates and it’s almost inevitable that you’ll remove a wrong one and something will stop working. Windows is a mess and storage is cheap. Live with it.