Skip to content

feat(exports): wire postcodes table into all export formats (#1039)#1403

Merged
dr5hn merged 1 commit into
masterfrom
feat/postcodes-exports
Apr 25, 2026
Merged

feat(exports): wire postcodes table into all export formats (#1039)#1403
dr5hn merged 1 commit into
masterfrom
feat/postcodes-exports

Conversation

@dr5hn
Copy link
Copy Markdown
Owner

@dr5hn dr5hn commented Apr 25, 2026

Summary

Completes the deferred export-pipeline work from #1398. With the postcodes table landed and #1401/#1402 starting to populate it, every export command and the export workflow itself now emit postcode data alongside the existing 5 tables.

Files changed (9)

PHP commands (bin/Commands/)

File Change
ExportJson.php SELECT from postcodes (graceful skip if table missing); emit /json/postcodes.json
ExportCsv.php Add postcodes to FILES; guard empty arrays so empty source files no longer crash on $csc[0] access
ExportXml.php Add postcodes to FILES; replace fragile ?: throw on empty arrays with explicit is_array() check
ExportYaml.php Same as XML
ExportSqlServer.php Add postcodes to TABLES with full CREATE TABLE schema (FKs to countries/states/cities, nullable state_id/city_id)
ExportMongoDB.php Add postcodes to COLLECTIONS plus new processPostcodes() method with country/state DBRef refs and GeoJSON Point location

Python helpers (bin/scripts/)

File Change
export/export_plist.py Include postcodes.csv with missing-file guard so the script no-ops cleanly when no postcodes have landed yet
sync/sync_mysql_to_json.py New sync_postcodes() per-country file writer (mirrors sync_cities); export_schema() includes postcodes when present

Workflow (.github/workflows/export.yml)

  • postcode_count env var with graceful fallback to 0 if table absent
  • mysqldump postcodessql/postcodes.sql
  • pg_dump postcodespsql/postcodes.sql
  • mysql2sqlite postcodessqlite/postcodes.sqlite3
  • mongoimport gated on non-empty postcodes.json (uses jq)
  • gzip postcodes.sql in sql/ and psql/ when present
  • POSTCODE_COUNT exposed to Release body and PR body
  • DROP order updated: postcodes drops first (FKs to countries/states/cities)

Behaviour with empty postcodes table

This PR is designed to ship cleanly even if postcodes is empty:

  • Importer/JSON/XML/YAML produce empty postcodes.json / .xml / .yml files without erroring
  • CSV writes an empty file and emits a "no records" note
  • mongoimport skipped via jq length check
  • mysqldump still emits the (empty) DDL — so consumers can rely on the table existing in every export format
  • sync_postcodes skips writing files when no rows exist (no spurious empty LI.json etc.)

Why bundle 9 files in one PR

Each file's change is small (1–60 lines), but they're functionally interdependent: ExportJson produces postcodes.json, which ExportCsv/Xml/Yaml consume; the workflow expects all formats to exist. Splitting would create CI-broken intermediate states.

Test plan

  • All 6 PHP files lint clean (php -l)
  • Both Python scripts compile (python3 -m py_compile)
  • Workflow YAML is valid (yaml.safe_load)
  • DROP TABLE order respects FK dependencies (postcodes → cities → states → countries → subregions → regions)
  • Reviewer runs php console export:json against a world DB with postcodes table populated
  • Reviewer runs php console export:json against a world DB without postcodes table — should not crash
  • Reviewer triggers the workflow once feat(postcodes/LI): add Liechtenstein postcodes (#1039) #1401 / feat(postcodes/MC): add Monaco postcode (#1039) #1402 country data has merged

Out of scope / follow-ups

  • GeoJSON / Toon converters (bin/scripts/export/json_to_geojson.py, json_to_toon.py) — these have hardcoded ["cities", "states", "countries"] lists. Postcode geo-coordinates are sparse (most rows null) and Toon serialisation of postcode arrays would benefit from a separate review. Tracked for a follow-up PR.
  • PostgreSQL schema dump (psql/schema.sql) — auto-generated by pg_dump --schema-only in the workflow, so picks up postcodes automatically once the table exists in PostgreSQL via NMIG.

Refs: #1039
Builds on: #1398

🤖 Generated with Claude Code

Completes the deferred export-pipeline work from #1398. With the
postcodes table now landed, every export command and the workflow
itself learns to emit postcode data alongside the existing 5 tables.

PHP commands (Symfony Console)
- ExportJson: SELECT from postcodes (graceful skip if table missing),
  emit /json/postcodes.json
- ExportCsv: add 'postcodes' to FILES; guard empty arrays so empty
  source files no longer crash on $csc[0] access
- ExportXml / ExportYaml: add 'postcodes' to FILES; replace fragile
  ?: throw on empty arrays with explicit is_array() check
- ExportSqlServer: add 'postcodes' to TABLES with full CREATE TABLE
  schema (FKs to countries/states/cities, nullable state/city)
- ExportMongoDB: add 'postcodes' to COLLECTIONS plus processPostcodes()
  with country/state DBRef references and GeoJSON Point location

Python helpers
- export_plist.py: include postcodes.csv with missing-file guard so
  the script no-ops cleanly until first country PR lands
- sync_mysql_to_json.py: new sync_postcodes() per-country file writer
  mirroring sync_cities; export_schema includes postcodes when present

Workflow
- postcode_count env var (graceful 0 if table absent)
- mysqldump postcodes -> sql/postcodes.sql
- pg_dump postcodes -> psql/postcodes.sql
- mysql2sqlite postcodes -> sqlite/postcodes.sqlite3
- mongoimport gated on non-empty postcodes.json
- gzip postcodes.sql in sql/ and psql/ when present
- POSTCODE_COUNT exposed to Release body and PR body

Behaviour with empty postcodes table
- Importer/JSON/CSV/XML/YAML produce empty postcodes.json (or skip in
  CSV's case) without erroring
- mongoimport skipped via jq length check
- mysqldump still emits the (empty) DDL, so consumers can rely on the
  table existing in every export format

Refs: #1039
Builds on: #1398

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 25, 2026 14:54
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. enhancement New feature or request exports labels Apr 25, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Wires the new postcodes table into the export pipeline so that all export formats (and the export workflow) emit postcode data alongside existing tables.

Changes:

  • Added postcodes to JSON/CSV/XML/YAML/MongoDB/SQL Server exporters and to the export workflow artifacts.
  • Added MySQL→contributions reverse-sync support for per-country contributions/postcodes/<ISO2>.json output when the table exists.
  • Updated plist export helper to include postcodes.csv and skip cleanly when the source file is missing.

Reviewed changes

Copilot reviewed 1 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
bin/scripts/sync/sync_mysql_to_json.py Adds sync_postcodes() and conditionally includes postcodes in schema export when present.
bin/scripts/export/export_plist.py Includes postcodes.csv and adds missing-file guard.
bin/Commands/ExportYaml.php Adds YAML export for /json/postcodes.json and hardens JSON decode handling for empty arrays.
bin/Commands/ExportXml.php Adds XML export for /json/postcodes.json and hardens JSON decode handling for empty arrays.
bin/Commands/ExportSqlServer.php Adds postcodes table DDL and includes it in SQL Server export set.
bin/Commands/ExportMongoDB.php Adds postcodes collection export with DBRefs and optional GeoJSON location.
bin/Commands/ExportJson.php Emits /json/postcodes.json, skipping gracefully if the DB table is missing.
bin/Commands/ExportCsv.php Adds postcodes to CSV exports and avoids crashing on empty datasets.
.github/workflows/export.yml Exports/archives postcodes across SQL/SQLite/MongoDB and exposes POSTCODE_COUNT in release/PR metadata.

@dr5hn dr5hn merged commit 5de5169 into master Apr 25, 2026
4 checks passed
@dr5hn dr5hn deleted the feat/postcodes-exports branch April 25, 2026 15:01
dr5hn added a commit that referenced this pull request Apr 28, 2026
These were committed at tiny placeholder sizes during #1039's initial
exports wiring (#1403), but the export.yml workflow regenerates them
at full size every run — 239 MB json/postcodes.json, 263 MB
xml/postcodes.xml, 171 MB yml/postcodes.yml, 123 MB psql/postcodes.sql,
105 MB sqlite/postcodes.sqlite3, 90 MB sql/postcodes.sql, 73 MB
sqlserver/postcodes.sql — all over GitHub's 100 MB hard limit.

#1490 added .gitignore entries for the same paths but gitignore is
inert against tracked files, so the export PR's git push still failed.
Untrack here so the gitignore actually applies; large compressed
.gz versions continue to ship via GitHub Releases.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request exports size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants