feat(exports): wire postcodes table into all export formats (#1039)#1403
Merged
Conversation
Completes the deferred export-pipeline work from #1398. With the postcodes table now landed, every export command and the workflow itself learns to emit postcode data alongside the existing 5 tables. PHP commands (Symfony Console) - ExportJson: SELECT from postcodes (graceful skip if table missing), emit /json/postcodes.json - ExportCsv: add 'postcodes' to FILES; guard empty arrays so empty source files no longer crash on $csc[0] access - ExportXml / ExportYaml: add 'postcodes' to FILES; replace fragile ?: throw on empty arrays with explicit is_array() check - ExportSqlServer: add 'postcodes' to TABLES with full CREATE TABLE schema (FKs to countries/states/cities, nullable state/city) - ExportMongoDB: add 'postcodes' to COLLECTIONS plus processPostcodes() with country/state DBRef references and GeoJSON Point location Python helpers - export_plist.py: include postcodes.csv with missing-file guard so the script no-ops cleanly until first country PR lands - sync_mysql_to_json.py: new sync_postcodes() per-country file writer mirroring sync_cities; export_schema includes postcodes when present Workflow - postcode_count env var (graceful 0 if table absent) - mysqldump postcodes -> sql/postcodes.sql - pg_dump postcodes -> psql/postcodes.sql - mysql2sqlite postcodes -> sqlite/postcodes.sqlite3 - mongoimport gated on non-empty postcodes.json - gzip postcodes.sql in sql/ and psql/ when present - POSTCODE_COUNT exposed to Release body and PR body Behaviour with empty postcodes table - Importer/JSON/CSV/XML/YAML produce empty postcodes.json (or skip in CSV's case) without erroring - mongoimport skipped via jq length check - mysqldump still emits the (empty) DDL, so consumers can rely on the table existing in every export format Refs: #1039 Builds on: #1398 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Wires the new postcodes table into the export pipeline so that all export formats (and the export workflow) emit postcode data alongside existing tables.
Changes:
- Added
postcodesto JSON/CSV/XML/YAML/MongoDB/SQL Server exporters and to the export workflow artifacts. - Added MySQL→contributions reverse-sync support for per-country
contributions/postcodes/<ISO2>.jsonoutput when the table exists. - Updated plist export helper to include
postcodes.csvand skip cleanly when the source file is missing.
Reviewed changes
Copilot reviewed 1 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
bin/scripts/sync/sync_mysql_to_json.py |
Adds sync_postcodes() and conditionally includes postcodes in schema export when present. |
bin/scripts/export/export_plist.py |
Includes postcodes.csv and adds missing-file guard. |
bin/Commands/ExportYaml.php |
Adds YAML export for /json/postcodes.json and hardens JSON decode handling for empty arrays. |
bin/Commands/ExportXml.php |
Adds XML export for /json/postcodes.json and hardens JSON decode handling for empty arrays. |
bin/Commands/ExportSqlServer.php |
Adds postcodes table DDL and includes it in SQL Server export set. |
bin/Commands/ExportMongoDB.php |
Adds postcodes collection export with DBRefs and optional GeoJSON location. |
bin/Commands/ExportJson.php |
Emits /json/postcodes.json, skipping gracefully if the DB table is missing. |
bin/Commands/ExportCsv.php |
Adds postcodes to CSV exports and avoids crashing on empty datasets. |
.github/workflows/export.yml |
Exports/archives postcodes across SQL/SQLite/MongoDB and exposes POSTCODE_COUNT in release/PR metadata. |
This was referenced Apr 27, 2026
dr5hn
added a commit
that referenced
this pull request
Apr 28, 2026
These were committed at tiny placeholder sizes during #1039's initial exports wiring (#1403), but the export.yml workflow regenerates them at full size every run — 239 MB json/postcodes.json, 263 MB xml/postcodes.xml, 171 MB yml/postcodes.yml, 123 MB psql/postcodes.sql, 105 MB sqlite/postcodes.sqlite3, 90 MB sql/postcodes.sql, 73 MB sqlserver/postcodes.sql — all over GitHub's 100 MB hard limit. #1490 added .gitignore entries for the same paths but gitignore is inert against tracked files, so the export PR's git push still failed. Untrack here so the gitignore actually applies; large compressed .gz versions continue to ship via GitHub Releases. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the deferred export-pipeline work from #1398. With the
postcodestable landed and #1401/#1402 starting to populate it, every export command and the export workflow itself now emit postcode data alongside the existing 5 tables.Files changed (9)
PHP commands (
bin/Commands/)ExportJson.phppostcodes(graceful skip if table missing); emit/json/postcodes.jsonExportCsv.phppostcodestoFILES; guard empty arrays so empty source files no longer crash on$csc[0]accessExportXml.phppostcodestoFILES; replace fragile?: throwon empty arrays with explicitis_array()checkExportYaml.phpExportSqlServer.phppostcodestoTABLESwith fullCREATE TABLEschema (FKs to countries/states/cities, nullable state_id/city_id)ExportMongoDB.phppostcodestoCOLLECTIONSplus newprocessPostcodes()method with country/state DBRef refs and GeoJSONPointlocationPython helpers (
bin/scripts/)export/export_plist.pypostcodes.csvwith missing-file guard so the script no-ops cleanly when no postcodes have landed yetsync/sync_mysql_to_json.pysync_postcodes()per-country file writer (mirrorssync_cities);export_schema()includespostcodeswhen presentWorkflow (
.github/workflows/export.yml)postcode_countenv var with graceful fallback to0if table absentmysqldump postcodes→sql/postcodes.sqlpg_dump postcodes→psql/postcodes.sqlmysql2sqlite postcodes→sqlite/postcodes.sqlite3mongoimportgated on non-emptypostcodes.json(usesjq)gzip postcodes.sqlinsql/andpsql/when presentPOSTCODE_COUNTexposed to Release body and PR bodypostcodesdrops first (FKs to countries/states/cities)Behaviour with empty postcodes table
This PR is designed to ship cleanly even if
postcodesis empty:postcodes.json/.xml/.ymlfiles without erroringmongoimportskipped via jq length checksync_postcodesskips writing files when no rows exist (no spurious emptyLI.jsonetc.)Why bundle 9 files in one PR
Each file's change is small (1–60 lines), but they're functionally interdependent:
ExportJsonproducespostcodes.json, whichExportCsv/Xml/Yamlconsume; the workflow expects all formats to exist. Splitting would create CI-broken intermediate states.Test plan
php -l)python3 -m py_compile)yaml.safe_load)php console export:jsonagainst aworldDB with postcodes table populatedphp console export:jsonagainst aworldDB without postcodes table — should not crashOut of scope / follow-ups
bin/scripts/export/json_to_geojson.py,json_to_toon.py) — these have hardcoded["cities", "states", "countries"]lists. Postcode geo-coordinates are sparse (most rows null) and Toon serialisation of postcode arrays would benefit from a separate review. Tracked for a follow-up PR.psql/schema.sql) — auto-generated bypg_dump --schema-onlyin the workflow, so picks up postcodes automatically once the table exists in PostgreSQL via NMIG.Refs: #1039
Builds on: #1398
🤖 Generated with Claude Code