Deduplicating and Decontaminating a Dataset

Exact and near-duplicate detection with MinHash and suffix arrays, n-gram decontamination against your eval sets, and what the deduplication studies reported.