Backend — Setup Your OSM Harvester

Purpose

The Pumps Harvester is also Python script, packed as an Docker image that is run as a GitHub action from our giessdenkiez-de repository. See our current active action here. It harvests pumps data from OSM and outputs the harvested file. Subsequently, for our use case, the harvested pumps data is pushed to Supabase storage.

How to use

Refer to the following diagram to understand the relationship between Frontend, Backend and the OSM Harvester. We are aware that this is overly complex and should be simplified. It grew historically into this.

In the example below we omit all comments that are the same as in the DWD harvester example. Create the file .github/workflows/pumps-harvester.yml and add the following code to it.

See https://github.com/technologiestiftung/giessdenkiez-de/blob/master/.github/workflows/pumps.yml for reference.

name: Update Pumps in Supabase Storage

on:
  workflow_dispatch:
  repository_dispatch:
    # This action can be triggered via Github API webook (see https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#repository_dispatch)
    types: [pumps_cron]

jobs:
  collect_pumps:
    # Using the payload of the repository_dispatch webhook to specify the environment
    environment: "${{ github.event.client_payload.environment }}"
    runs-on: ubuntu-latest
    name: A job to aggregate pumps data from open street maps and push to GDK Supabase
    steps:
      - name: Pumps data generate step
        id: pumps
        uses: technologiestiftung/[email protected]
        with:
          outfile-path: "public/data/pumps.geojson"
      - name: File output
        run: echo "The file was written to ${{ steps.pumps.outputs.file }}"
      - name: Upload file to supabase
        run: |
          getStatusCode=$(curl -s -o /dev/null -w "%{http_code}" \
            -X GET \
            ${{ vars.SUPABASE_URL }}/storage/v1/object/info/public/${{ vars.SUPABASE_DATA_ASSETS_BUCKET }}/pumps.geojson)
          if [ "$getStatusCode" = "200" ]; then
            putStatusCode=$(curl -s -o /dev/null -w "%{http_code}" \
              -X PUT \
              -H "Authorization: Bearer ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }}" \
              -H "Content-Type: application/geo+json" \
              -d "@${{ steps.pumps.outputs.file }}" \
              ${{ vars.SUPABASE_URL }}/storage/v1/object/${{ vars.SUPABASE_DATA_ASSETS_BUCKET }}/pumps.geojson)
            if [ "$putStatusCode" = "200" ]; then
              echo "Uploading to Supabase successful"
            else
              echo "Uploading to Supabase failed"
              exit 1
            fi
          else
            postStatusCode=$(curl -s -o /dev/null -w "%{http_code}" \
              -X POST \
              -H "Authorization: Bearer ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }}" \
              -H "Content-Type: application/geo+json" \
              -d "@${{ steps.pumps.outputs.file }}" \
              ${{ vars.SUPABASE_URL }}/storage/v1/object/${{ vars.SUPABASE_DATA_ASSETS_BUCKET }}/pumps.geojson)
            if [ "$postStatusCode" = "200" ]; then
              echo "Uploading to Supabase successful"
            else
              echo "Uploading to Supabase failed"
              exit 1
            fi
          fi