Validate latitude/longitude on Locations by reverse geocoding

[shajith]

Solving this:

Uses: https://github.com/thampiman/reverse-geocoder

Checks that the latitude/longitude, when geocoded, produces a spot that is in the state that the location is claimed to be in.

Does this if not:

[codecov]

The author of this PR, shajith, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at [email protected] with any questions.

[simonw]

I don't think this is the right approach here, for a couple of reasons.

Firstly, this is a really heavy new dependency - it pulls in the whole of SciPy and NumPy, which is a big increase in our overall dependency size. I don't think it's worth adding that to our application server footprint for this single use-case.

More importantly, having this as a hard validation check is going to cause us problems for any location that is close to a border.

41.9950887,-123.7249003 for example is a location in California right next to the Oregon border - but when I run this:

reverse_geocoder.search((41.9950887,-123.7249003))

I get back:

[{'lat': '42.16289',
  'lon': '-123.64812',
  'name': 'Cave Junction',
  'admin1': 'Oregon',
  'admin2': 'Josephine County',
  'cc': 'US'}]

Finally, there's quite a performance overhead in running this function:

%timeit reverse_geocoder.search((41.9950887,-123.7249003))
799 ms ± 126 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Running the validator in the model itself means this will add 800ms to each save we make for a Location - which is a big problem for our bulk data import APIs like /api/importLocations and /api/updateLocations which could be run against hundreds of locations at once.

[simonw]

Rather than implementing this as validation logic in VIAL, I suggest we run this as a separate process. We could set something up that runs on a schedule to verify that locations in VIAL aren't dramatically outside their suggested boundaries - using either this library or some other mechanism, such as the API I built at https://us-counties.datasette.io/counties/county_for_latitude_longitude?longitude=-123.7249003&latitude=41.9950887 for resolving a location to a county and state based on geographic polygons from the U.S. census.

[simonw]

This will be a lot easier once I add the "stream all results" mechanism to the search API in #367 - since you'll be able to do things like retrieve all 10,000 CA locations in one stream of newline-delimited GeoJSON and feed that into a script that geocodes them all.

[shajith]

All that makes sense. This IS a terribly large dependency for this use case, and I had my misgivings about that. I do want to catch wildly off values during manual data entry though, but I'm happy wait to see whether this becomes a larger problem.

How should I go about setting up a separate process to verify or geocode lat/lng values on locations? Do we have any precedent for that in the codebase? It seems generally useful to be able to run that sort of thing on a schedule over the data.

[simonw]

We could definitely make an improvement here for manual data entry - especially if we could show people a warning there but let them override it (for weird edge cases like a cruise ship offering vaccinations just off the coast of Puerto Rico or whatever)

For that I'd suggest some element of client-side validation in the admin interface itself. I'd like us to show the lat/Lon there on a map and let people drag to adjust the marker, which could relate into this too.

[shajith]

Validating client-side seems sufficient for my purposes. I can go look for ways to inject JS into django admin. Would it be appropriate to use the datasette API you linked above for this purpose (as opposed to google maps or something)?

[shajith]

@simonw: here's where I'm heading with the JS validation, as a stopgap:

validating-locations.mov

This uses a query like: https://us-counties.datasette.io/counties.json?sql=select%0D%0A++STATEFP+as+state_fips%2C%0D%0A++states.name+as+state%2C%0D%0A++STATEFP+%7C%7C+COUNTYFP+as+county_fips%2C%0D%0A++counties.NAME+as+county_name%2C%0D%0A++COUNTYNS%2C%0D%0A++AFFGEOID%2C%0D%0A++GEOID%2C%0D%0A++LSAD%2C%0D%0A++ALAND%2C%0D%0A++AWATER%0D%0Afrom%0D%0A++counties+join+states+on+counties.STATEFP+%3D+states.fips%0D%0Awhere%0D%0A++within%28%0D%0A++++GeomFromText%28%0D%0A++++++%27POINT%28%27+%7C%7C+%3Alongitude+%7C%7C+%27+%27+%7C%7C+%3Alatitude+%7C%7C+%27%29%27%0D%0A++++%29%2C%0D%0A++++counties.Geometry%0D%0A++%29%0D%0Alimit%0D%0A++1&longitude=-121.96590885207218&latitude=40.7764473230209

There's probably a better way to use that API?