feat(IT): remap cities to metropolitan cities and provinces (#1349)

[dr5hn]

Refs #1349 (Italy data: cities reported as 'totally wrong'). Do not auto-close — this PR is the city-level follow-up to the earlier states-only fix; the issue should remain open until cleanup of the duplicates flagged below.

Summary

• 9,947 IT cities were parented to the 20 region-level entities. None pointed at any of the 14 metropolitan cities, and the more granular provinces / free consortia / autonomous provinces / decentralized regional entities were similarly under-used.
• This PR remaps 9,828 cities to the correct ISO 3166-2:IT province-level entity. 119 records (mostly Ravenna and Barletta-Andria-Trani frazioni already correctly parented) are unchanged. 0 records remain unmapped.

After-state distribution

Type	Cities
Metropolitan city	1,682
Province	7,337
Autonomous province	390
Free municipal consortium	176
Decentralized regional entity	271
Autonomous region (Aosta only)	91
Total	9,947

Approach

Authoritative source: ISTAT Elenco dei comuni italiani (CC-BY 3.0 IT, 7,896 comuni), bundled at bin/scripts/fixes/data/istat-elenco-comuni-italiani.csv. The join key is Sigla automobilistica — the 2-letter province plate code (RM for Rome, TO for Turin, BZ for Bolzano), which matches our state.iso2 1:1 for every province-level entity.

bin/scripts/fixes/italy_remap_cities.py resolves each city in this order:

1. Name match, region-validated. Folds diacritics / apostrophes, looks up ISTAT denomination, requires the candidate's region to equal the city's current region (walked up via parent_id). 7,373 cities matched.
2. English-name aliases (Venice → Venezia, Florence → Firenze, etc.).
3. Conjunction-half match for "X e Y"-style comuni (Lampedusa → Lampedusa e Linosa). 25 cities.
4. k-NN proximity vote (k=5, ≤25km cap) for frazioni and historical names. 2,512 cities. More robust than single-nearest at province borders — fixes Mestre / Murano / Lido / Burano correctly snapping to Venice metro instead of Treviso province.

The script is idempotent: re-running on the rewritten data produces 0 changes.

Local validation (mirrors .github/scripts/validate-*)

• ✅ Schema: required fields present, country_code=IT, country_id=107.
• ✅ Cross-reference: every state_id resolves to an IT state; every state_code equals state.iso2.
• ✅ wikiDataId: all match ^Q\d+$.
• ✅ Coordinates: 9,946/9,947 within country-bounds.json IT box. Lampedusa (lat 35.5°) sits just south of the bounds, but that's a pre-existing country-bounds.json gap — Lampedusa is the southernmost Italian comune, geographically closer to Tunisia than the mainland. Not caused by this remap.
• ✅ Same-name + ≤5km duplicate scan: 0 hits.
• ✅ JSON parses cleanly.

Possible duplicates flagged for maintainer review (NOT deleted)

8 pairs / groups now map to a single comune. Per the task constraints, no records were deleted:

Sigla	Comune	Cities
CR	Pozzaglio ed Uniti	"Pozzaglio", "Pozzaglio ed Uniti"
FI	Capraia e Limite	"Capraia e Limite", "Limite"
MN	Sermide e Felonica	"Sermide", "Felonica"
NA	Napoli	"Naples", "Napoli"
PV	Corteolona e Genzone	"Corteolona", "Genzone"
PV	Inverno e Monteleone	"Inverno", "Inverno e Monteleone"
SS	Trinità d'Agultu e Vignola	"Trinità d'Agultu", "Trinità d'Agultu e Vignola"
TO	Torino	"Torino", "Turin"

Most are conjunction-merger artefacts (older comuni that ISTAT now lists under unified names) or English/Italian language duplicates. Recommend keeping the ISTAT-canonical row and removing the legacy half in a follow-up.

Out-of-scope but flagged for follow-up

• Corrupt native field. Many native values look machine-translated (e.g. Pero → native Ma; Postal → native Postale). The script intentionally matches on name rather than native. A separate fix should restore correct Italian comune names to native.
• country-bounds.json IT box does not include the Pelagie islands (Lampedusa/Linosa). Worth widening IT.minLat to 35.5° in a one-line PR.
• Tessera (Venice airport area) still maps to Treviso because its 5-NN cluster is mixed; not enough impact to hand-tune.

Commits

1. feat(IT): add italy_remap_cities.py + ISTAT comune mapping data (#1349) — script + bundled CSV + structured JSON report.
2. feat(IT): remap 9,828 cities to provinces / metropolitan cities (#1349) — pure data diff (only state_id and state_code fields touched).
3. docs(IT): add FIX_1349_SUMMARY for cities remap (#1349) — fix-docs entry.

Test plan

• python3 bin/scripts/fixes/italy_remap_cities.py --dry-run reports 9947 input / 0 changes after merge (idempotent).
• jq '[.[] | select(.country_code=="IT" and .type=="metropolitan city")] | length' contributions/states/states.json returns 14, and each metro state has cities (jq '[.[] | select(.state_code=="MI")] | length' contributions/cities/IT.json etc.).
• CI's validate-schema, validate-cross-reference, validate-coordinates, detect-duplicates pass.
• After Phinx import + JSON export, the round-trip preserves the new state assignments.

🤖 Generated with Claude Code

[dr5hn]

Weekly data-quality review (2026-04-27)

Verdict: clean

Checks

• Schema: ✅ Only state_id and state_code modified on existing records; no new records and no auto-managed fields added.
• FK integrity: ✅ Author ran cross-reference validator: all 9,947 cities resolve state_id to an IT state whose iso2 matches state_code. ISTAT CSV join + k-NN proximity fallback methodology is sound and independently reproducible.
• Coordinates: ✅ No coordinate changes. Pre-existing edge case: Lampedusa (~35.5°N) sits just south of the country-bounds.json IT minLat: 35.49 — not introduced by this PR.
• Wikidata: N/A (no Wikidata field changes)
• Naming convention: ✅ No name/native field changes. English-name convention preserved (Turin, Naples retained); duplicate Italian-name rows handled in PR fix(IT): drop 6 duplicate pairs flagged by remap (#1349 follow-up) #1399.

Advisory (non-blocking)

• Scale — 9,828 of 9,947 records remapped in a single PR. Individual mapping verification is impractical; the ISTAT join methodology, the idempotency --dry-run check, and the bundled it_remap_report.json are the primary audit trails.
• 8 pending duplicate pairs — Sermide/Felonica and Corteolona/Genzone (both ambiguous merged comuni where neither row carries the modern ISTAT canonical name) remain open; requires maintainer sign-off on which ID to rename and which to drop. The 6 unambiguous pairs are addressed in PR fix(IT): drop 6 duplicate pairs flagged by remap (#1349 follow-up) #1399.
• country-bounds.json IT minLat gap — The Pelagie islands (Lampedusa ~35.5°N, Linosa ~35.8°N) lie south of the current bound. A one-line change widening IT.minLat to 35.0 would eliminate false coordinate-validation failures for those islands.

🤖 Automated weekly review — Claude (sonnet-4-6).

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR addresses issue #1349 by remapping Italy’s city records from region-level parents to the correct province-level entities (metropolitan cities, provinces, autonomous provinces, etc.), using ISTAT’s official comune list as the authoritative mapping source.

Changes:

• Adds a reproducible remap script that resolves each city via name/alias/conjunction matching with a proximity (k-NN) fallback.
• Bundles ISTAT mapping data and a generated structured remap report to support auditing/reproducibility.
• Documents the methodology, counts, validation, and flagged potential duplicates for follow-up.

Reviewed changes

Copilot reviewed 1 out of 5 changed files in this pull request and generated 1 comment.

File	Description
bin/scripts/fixes/italy_remap_cities.py	Implements the two-pass city→province remapping logic (ISTAT join + proximity fallback) and rewrites contributions/cities/IT.json.
bin/scripts/fixes/data/it_remap_report.json	Stores the structured output from a remap run (counts, distributions, possible duplicates, samples).
.github/fixes-docs/FIX_1349_SUMMARY.md	Fix documentation summarizing rationale, approach, and validation results for the Italy city remap.

[Copilot]

In the counts block, the key proximity_skipped doesn’t match the script/report field name (proximity_skipped_or_far). For consistency (and to make it easier to cross-check with it_remap_report.json), consider updating the label in this doc to use the same key name as the generated report.

Suggested change

	input 9947
	name_unique 7373 exact ISTAT match in same region
	name_region 13 multiple ISTAT matches, region tie-break
	name_ambiguous 69 matches existed but in another region (rejected -> proximity)
	name_conjunction 25 matched via "X e Y" half
	no_match 2467 no ISTAT name (frazione / hamlet / historical name)
	proximity_assigned 2512 resolved by 5-NN vote within 25 km
	proximity_skipped 0 none rejected for being too far
	changed 9828 state_id and/or state_code rewritten
	unchanged 119 already pointed at the correct province (mostly RA, BT)
	unmapped 0 every record reached a final assignment
	input 9947 exact ISTAT match in same region
	name_unique 7373 exact ISTAT match in same region
	name_region 13 multiple ISTAT matches, region tie-break
	name_ambiguous 69 matches existed but in another region (rejected -> proximity)
	name_conjunction 25 matched via "X e Y" half
	no_match 2467 no ISTAT name (frazione / hamlet / historical name)
	proximity_assigned 2512 resolved by 5-NN vote within 25 km
	proximity_skipped_or_far 0 none rejected for being too far
	changed 9828 state_id and/or state_code rewritten
	unchanged 119 already pointed at the correct province (mostly RA, BT)
	unmapped 0 every record reached a final assignment