Is there a hard drive file organizer that will ...

On Sun, 26 Oct 2008 16:23:16 -0400, "FromTheRafters"
<erratic RemoveThis @nomail.afraid.org> put finger to keyboard and composed:

>> AFAICS, a fundamental flaw in duplicate finder software is that it
>> relies on direct binary comparisons. With programs like FindDup, if we
>> have 3 files of equal size, then we would need to compare file1 with
>> file2, file1 with file3, and file2 with file3. This requires 6 reads.
>> For n equally sized files, the number of reads is n(n-1).
>>
>> Alternatively, if we relied on MD5 checksums, then each file would
>> only need to be read once.
>
>So...once it is found to be the same checksum, what should the
>program do next? How important are these files? A fundamental
>flaw would be to trust MD5 checksums as an indication that the
>files are indeed duplicates. You can mostly trust MD5 checksums
>to indicate two files are different, but the other way around?

OK, I retract my ill-informed comment, but it still seems to me that
the benefits far outweigh the risks. FindDup has been running for the
past 18 hours or so as I write this, so I'm happy to accept a 30
minute alternative. In any case, all programs appear to require that
the user decides whether or not a file can be safely deleted. To this
end the programmer could allow for a binary comparision in those cases
where there is any doubt.

- Franc Zabkar
--
Please remove one 'i' from my address when replying by email.

"Bill in Co." wrote:

> > A fundamental flaw would be to trust MD5 checksums as an
> > indication that the files are indeed duplicates.
>
> Since when? What is the statistical likelyhood of that being
> true?

If there was no malicious intent or source involved, I'd say the odds
are pretty low. But even if you had 2 identical hashs, it's simple
enough to just see if the files are the same length, and if they were,
then you do a byte-by-byte comparison.

"Franc Zabkar" <fzabkar.RemoveThis@iinternode.on.net> wrote in message
news:rpn9g4h6d10d3kv20ud3j02e2phuq66ucg@4ax.com...
> On Sun, 26 Oct 2008 16:23:16 -0400, "FromTheRafters"
> <erratic.RemoveThis@nomail.afraid.org> put finger to keyboard and composed:
>
>>> AFAICS, a fundamental flaw in duplicate finder software is that it
>>> relies on direct binary comparisons. With programs like FindDup, if we
>>> have 3 files of equal size, then we would need to compare file1 with
>>> file2, file1 with file3, and file2 with file3. This requires 6 reads.
>>> For n equally sized files, the number of reads is n(n-1).
>>>
>>> Alternatively, if we relied on MD5 checksums, then each file would
>>> only need to be read once.
>>
>>So...once it is found to be the same checksum, what should the
>>program do next? How important are these files? A fundamental
>>flaw would be to trust MD5 checksums as an indication that the
>>files are indeed duplicates. You can mostly trust MD5 checksums
>>to indicate two files are different, but the other way around?
>
> OK, I retract my ill-informed comment, but it still seems to me that
> the benefits far outweigh the risks. FindDup has been running for the
> past 18 hours or so as I write this, so I'm happy to accept a 30
> minute alternative. In any case, all programs appear to require that
> the user decides whether or not a file can be safely deleted. To this
> end the programmer could allow for a binary comparision in those cases
> where there is any doubt.

It all depends on the risk you are willing to assume. It would be nice
to have a hybrid case where you could switch between the MD5
mode and the byte by byte mode depending on such factors as type
or location of files etc.

"Bill in Co." <not_really_here DeleteThis @earthlink.net> wrote in message
news:%23RrPjC7NJHA.1908@TK2MSFTNGP04.phx.gbl...
> FromTheRafters wrote:
>>> AFAICS, a fundamental flaw in duplicate finder software is that it
>>> relies on direct binary comparisons. With programs like FindDup, if we
>>> have 3 files of equal size, then we would need to compare file1 with
>>> file2, file1 with file3, and file2 with file3. This requires 6 reads.
>>> For n equally sized files, the number of reads is n(n-1).
>>>
>>> Alternatively, if we relied on MD5 checksums, then each file would
>>> only need to be read once.
>>
>> So...once it is found to be the same checksum, what should the
>> program do next? How important are these files?
>
>> A fundamental
>> flaw would be to trust MD5 checksums as an indication that the
>> files are indeed duplicates.
>
> Since when?

Forever.

Checksums are often smaller than the file they are derived from
(thats kinda the point, eh?).

> What is the statistical likelyhood of that being true?

Greater than zero.

FromTheRafters wrote:
> "Bill in Co." <not_really_here.DeleteThis@earthlink.net> wrote in message
> news:%23RrPjC7NJHA.1908@TK2MSFTNGP04.phx.gbl...
>> FromTheRafters wrote:
>>>> AFAICS, a fundamental flaw in duplicate finder software is that it
>>>> relies on direct binary comparisons. With programs like FindDup, if we
>>>> have 3 files of equal size, then we would need to compare file1 with
>>>> file2, file1 with file3, and file2 with file3. This requires 6 reads.
>>>> For n equally sized files, the number of reads is n(n-1).
>>>>
>>>> Alternatively, if we relied on MD5 checksums, then each file would
>>>> only need to be read once.
>>>
>>> So...once it is found to be the same checksum, what should the
>>> program do next? How important are these files?
>>
>>> A fundamental
>>> flaw would be to trust MD5 checksums as an indication that the
>>> files are indeed duplicates.
>>
>> Since when?
>
> Forever.
>
> Checksums are often smaller than the file they are derived from
> (thats kinda the point, eh?).

No, that's not the point. Your statement was that the checksums did not
assure the integrity of the file, whatsoever - i.e., that two files could
have the same hash valus and yet be different, which I still say is *highly*
unlikely. A statistically insignificant probability, so that using hash
values is often prudent and is much more expedient, of course.

"Bill in Co." <not_really_here.TakeThisOut@earthlink.net> wrote in message
news:ugvJco8NJHA.3636@TK2MSFTNGP05.phx.gbl...
> FromTheRafters wrote:
>> "Bill in Co." <not_really_here.TakeThisOut@earthlink.net> wrote in message
>> news:%23RrPjC7NJHA.1908@TK2MSFTNGP04.phx.gbl...
>>> FromTheRafters wrote:
>>>>> AFAICS, a fundamental flaw in duplicate finder software is that it
>>>>> relies on direct binary comparisons. With programs like FindDup, if we
>>>>> have 3 files of equal size, then we would need to compare file1 with
>>>>> file2, file1 with file3, and file2 with file3. This requires 6 reads.
>>>>> For n equally sized files, the number of reads is n(n-1).
>>>>>
>>>>> Alternatively, if we relied on MD5 checksums, then each file would
>>>>> only need to be read once.
>>>>
>>>> So...once it is found to be the same checksum, what should the
>>>> program do next? How important are these files?
>>>
>>>> A fundamental
>>>> flaw would be to trust MD5 checksums as an indication that the
>>>> files are indeed duplicates.
>>>
>>> Since when?
>>
>> Forever.
>>
>> Checksums are often smaller than the file they are derived from
>> (thats kinda the point, eh?).
>
> No, that's not the point. Your statement was that the checksums did not
> assure the integrity of the file, whatsoever

I didn't say anything about the integrity of a file, and I also didn't
say 'whatsoever'. You can still read what I said above.

If you want to ensure they are duplicates - compare the files exactly.
If you only need to be reasonably sure they are duplicates, checksums
are adequate.

> - i.e., that two files could have the same hash valus and yet be
> different, which I still say is *highly* unlikely.

Highly unlikely -yes. But files can be highly valuable too. Just
how fast does such a program need to be? How much speed
is worth how much accuracy?

> A statistically insignificant probability, so that using hash values is
> often prudent and is much more expedient, of course.

True, but to aim toward accuracy instead of speed is not a flaw.