Add config flag to boards for detecting dupes

[karai17]

In some cases, it might be preferred to allow boards to have duplicate content.

[raidho36]

Really allowing duplicate content is pointless, it just wastes your hard drive space. When enabled, it should substitute duplicate file for a reference to already existing file. Then you, however, can't manage files by nuking everything with sunk thread, you'll have to find another thread that references duplicate file and attach it there.

[karai17]

hdd space isn't really that big of an issue tbh. The small amount of dupes probably wouldn't be more than a few hundred mb per board in very active boards, and some of the very active boards like /b/ usually turn archiving off so once it falls off of page 10 or 20 or whatever, it's all freed up.

I think it'd be easier to just allow dupe files than to try and micromanage kilobytes.

[shakesoda]

I'd strongly prefer just using the hash check to point to an existing file

[raidho36]

You'd only think it's "kilobytes" if your board only allows small files, has few threads, and is in general not popular. Something as simple as webm thread can tank hard drive usage by several gigabytes. Additionally, having duplicate files reside in the filesystem prevents using same file cache for the duplicate file, i,e. both files will have to be cached, taking up server memory and displacing other files from cache, which gives additional hard drive load to look up files that are missing from cache. Speaking of webm threads and the like, those usually contain large amount of various files, and may be prime cause of duplicate file "errors". So it's not helpful to refuse a post containing a duplicate file. Again, if you're facing problems with flood of identical files, that's a problem of moderation, not software. It is an incredibly naive idea that people will not bypass it if they wanted to post it, defeating the whole purpose of such "feature" and getting annoyed in the process, so again, it's not helpful.