Project 2 · Part 1 · Record linkage
Same product, two shops
Abt and Buy describe the same electronics in different words: one lists Sony SLV-D380P Black DVD VHS Combo Player - SLVD380P
, the other Sony SLVD380P DVD/VCR Combo
. Task 1a links a small sample of both catalogues with four increasingly fuzzy passes. Task 1b prepares the full catalogues for linkage by blocking them on brand, so only products from the same maker are compared.
- precision (1a)
- 0.864precision (1a)
- recall (1a)
- 0.852recall (1a)
- pair completeness (1b)
- 0.957pair completeness (1b)
- reduction ratio (1b)
- 0.945reduction ratio (1b)
The four-pass matcher
Each pass only looks at products the earlier passes left unmatched. The first two trust model numbers; the third compares descriptions within a brand using three similarity measures; the last falls back to name similarity. All of it runs in a background worker, ported so the scores are bit-identical to textdistance and fuzzywuzzy.
Blocking explorer
The submitted key is the brand: the first word of the Abt name, and the first word of the Buy manufacturer (or of the name when the manufacturer is blank). The alternatives below are not part of the coursework; they show how coarser or finer keys move along the same trade-off.
Data source: Abt-Buy product matching benchmark, Leipzig University database group; the subset used in COMP20008; ids, names, descriptions and manufacturers only.
Next: Part 2
Predicting life expectancy from World Bank indicators.