Duplicate product detection
A Rust engine that finds duplicate listings in an e-commerce catalog. R&D on contract, never released.
Pick a pair or edit either title, then watch the green matches add up to a verdict.
1Cut each title into pieces
Spaces and punctuation drop out, then every run of 3 characters becomes a piece. Green pieces are in both titles.
A 28 pieces
B 28 pieces, 23 shared
2Shrink each to a signature
A signature is a short fingerprint of a title: 32 pieces, picked by the same rules for both. Cells light up where A (top) and B (bottom) agree. A group of 4 that fully agrees puts both in the same bucket.
3Compare only what shares a bucket
- True overlap
- 70%
- Signature match
- 84%
- Shared buckets
- 4 of 8
- Verdict
- Same product
4 groups matched, so they share a bucket and get compared. 84% of the signature matches, at or above the 50% line.
On a contract with Betty I built a Rust engine to find duplicate products in their catalog. It never shipped, and it is the most substantial Rust I have written.
MinHash sketches and locality-sensitive hashing bucket near-identical listings, which avoids comparing every pair. Canonical clustering collapses chains (A matches B, B matches C) into one group with one canonical record. Scans are incremental.
Precision, recall and F1 came from a harness I built. Performance came from Criterion benchmarks.