Changes to the SPR

[thisisaaronland]

What changes should be applied to a v3 of the Standard Places Response (SPR). See also:

• https://github.com/whosonfirst/go-whosonfirst-spr/blob/main/spr.go

[nvkelso]

Id love to see wof:population as population in the SPR. This helps rank any result set of SPR responses and on the data side we expend a lot of effort gathering this data.

[missinglink]

I'm not sure how relevant this still is but we have code in Pelias. which checks a variety of fields for population.

I'd be curious to see how much of an effect this has on the total fields output, I seem to recall that it was a good idea at the time 🤔

[missinglink]

Seems to be sourced from this comment: whosonfirst-data/whosonfirst-data#240 (comment)

[nvkelso]

Good find. My recollection is @stepps00 crawled the repo(s) and applied that source property ordered coalesce into the wof:population field for all data records. If that's not the case then we should file a data issue for the same.

Either way, I like updating the SPR to have a single population – ideally from the single source, or a business logic coalesce.

[stepps00]

Yeah iirc, that order of property preference was used to construct the wof:population values while crawling all admin repos. Happy to file an issue / start another crawl if Pelias is seeing different values when using that logic vs the existing wof:population value.

And agreed that population_rank (maybe full population, too?)would be helpful in the SPR.

[missinglink]

The Pelias code is 6 years old, so I'd have to do a new evaluation to speak to the current state of affairs.

[nvkelso]

Potentially we could clarify that the bounding box min/max lat/long floats are also biased towards "label"?

	// The latitude for the principal centroid (typically "label") of the place result
	Latitude() float64
	// The longitude for the principal centroid (typically "label") of the place result
	Longitude() float64
	// The minimum latitude of the bounding box of the place result
	MinLatitude() float64
	// The minimum longitude of the bounding box of the place result
	MinLongitude() float64
	// The maximum latitude of the bounding box of the place result
	MaxLatitude() float64
	// The maximum longitude of the bounding box of the place result
	MaxLongitude() float64

[nvkelso]

While these alternate geoms don't exist today... They should, and including a pointer to a "200 kb" or less "display" geometry directly in the SPS would be amazing.

[missinglink]

This is fairly easy to do now without adding any new columns:

SELECT *
FROM spr
JOIN geojson ON (spr.id = geojson.id AND is_alt = 1 AND alt_label = 'small' )

[missinglink]

Some fields we use in Pelias which we might consider:

wof:label
wof:lang_x_official
wof:shortcode
wof:abbreviation

[nvkelso]

Agree these are all useful – especially when trying to guess which label (in what language locale) should be shown for people from that place's location.

[stepps00]

How about the mz:hierarchy_label and mz:is_funky properties? These are helpful in generating place labels/address strings and understanding general "confidence" of a place.

[thisisaaronland]

A few initial comments:

1. The SPR can change and it can have more properties than it does today. I just want to make it clear that's not the issue.
2. The SPR is designed to be a minimalist, common representation of a record so we should be diligent in adding new properties.
3. If the goal is for every record to have a "small" geometry then it's not clear what the benefit of including that in the SPR since it's mostly a question of documenting what the URI syntax for that geometry is. We've also talked about making the small version the default geometry.
4. I am not sure what to think about names/labels yet. Maybe something like including wof:lang_x_official/spoken and a has_official/spoken property to indicate that a language-specific translation is available?
5. Some of the properties discussion seem like they are derivatives (as in they are derived from other properties and logic). I would favour including the derivatives in place of all their component pieces. Some sort of confidence property would be a good addition. Likewise population seems like it's really a stand-in for logic about label display, and ordering, so I would opt for being explicit about that.

cc @tomtaylor @vicchi

[missinglink]

I agree that spr should be fairly conservative, it's beneficial to have a small subset of the data which is generally useful, and to keep that as concise as possible.

A good yardstick here is to ask what's 'generally useful' vs. what's 'useful for a specific purpose'.

I would argue that small geometries (for instance) are not useful for geocoding and therefore not generally useful, although I would to some degree argue that the zoom levels should be included even though I won't use them.

Rather than extending the spr we could consider better documenting how to extract whatever data is required for each consumer's use case.

Users with high technical knowledge can always create their own extract, so spr is mostly useful to less technical users and as a pivot table to apply filter conditions.

[thisisaaronland]

Part of the rationale for the SPR was that it would/could/should be the common response for API methods (remember the API...) or really any other query whether that was a general-purpose text search or a spatial lookup. The SPR was designed with the idea that there would always be a common response (regardless of the overall heterogeneity of the data) with enough data to plot the results on a map and, importantly, perform subsequent lookups/rendering by fetching the whole record or using something like the wof-browser's "select" endpoint [1].

[1] https://github.com/whosonfirst/go-whosonfirst-browser#select

[missinglink]

The SPR was designed with the idea that there would always be a common response

However if this is the intent then I guess these should also be the fields available in shapefile, for example, to be consistent?

[nvkelso]

I miss the API, those were the days :)

[thisisaaronland]

I agree that if there is going to be an "extended" response, which starts to get us back to the good old days of the "meta" files, it would be best if it encapsulated and extended the SPR.

[missinglink]

The recent motivation for extending spr came from a discussion around providing WOF in formats such as shapefile, geopackage, geoparquet etc.

In these cases it's nice to have a bunch of fields available, and that requirement may not 100% align with the original motivation when designing spr, which was to provide a more concise view of the data.

It's worth considering if a new view of the data may make sense for this purpose, an 'extended place response` (epr) for instance, or whether the two concepts can be expressed in the same view.

[nvkelso]

Agree less is better for the spr.

For localization, I imagine the spr would return from an API an appropriately localized name the responses name field – if the request included a locale param or header. That it wouldn't need to return name_eng and name, instead just name.

I wonder if the tension here is between flat file storage versus query response? Or what @missinglink was saying with "well known" extended place response variants? And how an epr might work in an API, but also be applied to generated flat files?

So if you make a request that includes a (default false) optional true flag for "give me names in top locales", then it'd also give you back the set of locales we're exporting in the flat files (like 20 of them), with additional properties to indicate the wof:lang stuff mentioned above?

wof:lang_x_official/spoken and a has_official/spoken

3-char	2-char	Language	Native script
ara	ar	Arabic	العربية
ben	bn	Bengali	বাংলা
deu	de	German	Deutsch
eng	en	English	English
ell	el	Greek (modern)	ελληνικά
spa	es	Spanish	español
fas	fa	Farsi	فارسی
fra	fr	French	français
heb	he	Hebrew	עִבְרִית‎
hin	hi	Hindi	हिन्दी
hun	hu	Hungarian	magyar
ind	id	Indonesian	Bahasa Indonesia
ita	it	Italian	italiano
jpn	ja	Japanese	日本語
kor	ko	Korean	한국어
nld	nl	Dutch	Nederlands
pol	pl	Polish	Polski
por	pt	Portuguese	Português
rus	ru	Russian	Русский
swe	sv	Swedish	Svenska
tur	tr	Turkish	Türkçe
ukr	uk	Ukrainian	українська
urd	ur	Urdu	اردو
vie	vi	Vietnamese	Tiếng Việt
zho	zh	Chinese (simplified and/or traditional)	中文

See also:

• https://github.com/nvkelso/natural-earth-vector/tree/master/tools/wikidata#supported-languages--now-26
• https://github.com/tilezen/vector-datasource/blob/master/docs/SEMANTIC-VERSIONING.md#languages-are-not-versioned

And if you're interested in concordances then the slimmed set of concordances that also ends up in the flat files?

'gn:id' AS gn_id,
'wd:id' AS wd_id,
'uscensus:geoid' AS usgeo_id,
'hasc:id' AS hasc_id

NOTE: gp:id has great coverage, too but is perhaps less germane for people now a days?

If you're interested in slim hierarchy, then just:

country_id,
dependency_id,
region_id,
county_id,
locality_id,

And if you're interested in labeling extras:

"mz:min_zoom" as min_zoom,
"mz:max_zoom" as max_zoom,
"lbl:min_zoom" as min_label,
"lbl:max_zoom" as max_label,

In the case of population that's widely available and I think deserves to be in the spr (rather than an epr).

In the case of mz:is_funky I could go either way. When I think of the SHP export I want to see them there, even though we otherwise mask out features that aren't current" from that export...

Should the properties repo also add properties as to the sprs / epr shortened property name is for each fully qualified and namespaced WOF property?

[thisisaaronland]

Conceptually, the SPR is an "interface" so there can be multiple implementations/representations. For example, there are different implementations for alt and not-alt geometries:

• https://github.com/whosonfirst/go-whosonfirst-spr/blob/main/whosonfirst.go
• https://github.com/whosonfirst/go-whosonfirst-spr/blob/main/whosonfirst_alt.go

That's convenient for strictly-typed languages (which only know about interface methods rather than property values) but also helps to enforce a measure of commonality across the entire dataset.

So there is technically nothing to prevent a dynamically generated SPR from picking and choosing from the available language names but we should think carefully about how that works at a technical level. For example, all of the PIP code is predicated on only having to store a SPR alongside the spatial data used to perform a query rather than an entire WOF record. This is a practical consideration to account for the database size.

One approach might be to introduce a URI-based syntax for creating SPRs, along the lines of the go-reader and go-writer packages:

• https://github.com/whosonfirst/go-reader#available-readers
• https://github.com/whosonfirst/go-writer#writers

That would allow for an SPR to be instantiated along the lines of wof:// or wof://?lang=deu or wof-alt://?lang=SOME_LABEL_RULE.

The point being that there would be a consistent and portable way to define "labeled" SPRs and to define rules and reference implementations for how a SPR URI translates a WOF record into SPR JSON.

In the meantime, remember that the select endpoint can be used to look up atomic properties for a WOF record. For example:

https://data.whosonfirst.org/select/85922499?select=properties.name:zho_x_preferred