Fetch new data and update the repository #884
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Fetch new data and update the repository | |
| on: | |
| workflow_dispatch: | |
| schedule: | |
| - cron: "5 8 */3 * *" | |
| jobs: | |
| update-data: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Checkout code | |
| uses: actions/checkout@v3 | |
| - name: Setup Python | |
| uses: actions/setup-python@master | |
| with: | |
| python-version: 3 | |
| - name: Grab latest alltheplaces | |
| run: | | |
| git clone --depth=1 https://github.com/alltheplaces/alltheplaces.git | |
| - name: Install scrapy | |
| run: | | |
| pip install --quiet --upgrade pip scrapy xlrd geonamescache json5 chompjs xmltodict phonenumbers reverse_geocoder xmltodict requests_cache pycountry ijson openpyxl pyproj pygeohash unidecode phpserialize shapely babel geopandas h2 pdfplumber scrapy_zyte_api | |
| - name: Run USPS spider | |
| run: | | |
| cd alltheplaces | |
| scrapy crawl usps_collection_boxes --output ../usps_collection_boxes.ndjson:ndgeojson --loglevel=ERROR | |
| - name: Sort the data | |
| run: | | |
| mkdir -p data | |
| python sort.py usps_collection_boxes.ndjson usps_collection_boxes_sorted.ndjson --prefix-dir=data --prefix-chars=3 | |
| - name: Push changes to remote | |
| run: | | |
| git config user.email "ian.dees+bot@gmail.com" | |
| git config user.name "USPS Bot" | |
| echo "$(date +%Y-%m-%dT%H:%M:%SZ)" > data/last_updated.txt | |
| git add data | |
| git commit -m "Adding data from $(date +%Y-%m-%d)" | |
| git push origin master |