DBS-derived organizations without geo coordinates #89

acka47 · 2015-12-02T12:48:58Z

As said in #49 (comment), there are a lot DBS-derived entries without geo coordinates. See, e.g. this list: http://beta.lobid.org/organisations/search?q=@id=DBS&size=20&from=250

At öleast for some entries, the reason for the missing geo data is that no street address is provided by DBS. From the csv file, I can see that 124 of the entries.

First of all, I would like to know how many entries in total are lacking geo corrdinates. Can I find this out with elasticsearch, i.e. can I get all entries where a particular field is missing?

hauschke · 2015-12-09T18:09:13Z

I have found 148 institutions without geonames, and subsequently(?) without lat&lon:
https://gist.github.com/hauschke/b4c998dbee1bb0f75947#file-missing_geonames-csv

Related?

dr0i · 2015-12-10T08:31:24Z

Just as a note - in the old lobid-organisations, most (all?)of this data having an ISIL do have geo-data, e.g. DE-198 and DE-MUS-430817.

fsteeg · 2015-12-16T13:20:26Z

To fix this, we need to

do a full reindexing to get coordinates for updated entries
implement automatic geo lookup during updates as in the old lobid-organisations

acka47 · 2015-12-17T13:55:06Z

Re. containedIn, as asked by @fsteeg on twitter (https://twitter.com/fsteeg/status/677483772451954688). This is added based on the "Regionalschlüssel" (rs). See #17 (comment) where it reads:

Add look-up of geonames in map using 8-digit "Gemeindeschlüssel" ("032P.n" of Sigel and a 8-digit derivation of "gemeindekennzahl" in DBS) for field "containedIn"

fsteeg · 2015-12-17T13:59:49Z

Some confusion here:

Original issue was about missing geo coordinates for DBS entries, this is a geo lookup issue and remains open
@hauschke mentioned missing geonames (not geo coordinates). As far as I can tell, only 1/4 of the the orgs even have a geo name in the containedIn field: http://beta.lobid.org/organisations/search?q=containedIn:* vs. http://beta.lobid.org/organisations/search?q=*
@dr0i mentioned all Sigel-Orgs have geo coordinates in the old API, but this is true for the new one too (at least for the two examples given) (@dr0i: I made the new issue ISIL organizations without geo coordinates #93 for that)

So what I wrote in the comment above about the steps we need to take is nonsense. The original issue remains (geo coordinates for DBS entries), and geonames are what they are and we have no plans to change anything. For Sigel-Orgs everything seems to be fine, thus I'll remove the nwbib-launch label, unassign myself and move to backlog.

@hauschke If there's anything you need, get in contact (perhaps in a new issue).

hauschke · 2015-12-17T13:59:53Z

Example of organisation without lat+long, and without geonamed id:
http://beta.lobid.org/organisations/DBS-WA339

Should have:
"containedIn" : "http://www.geonames.org/2875558"

acka47 · 2015-12-17T14:23:14Z

Example of organisation without lat+long, and without geonamed id:
http://beta.lobid.org/organisations/DBS-WA339

Should have:
"containedIn" : "http://www.geonames.org/2875558"

Generally, only DBS entries contain "Regionalschlüssel" which are then mapped to GeoNames. There can be three reasons why entries don't have a GeoNames link.

~~Entries solely derived from ISIL registry (i.e. that are not also part of DBS) don't have a link to GeoNames as the "Regionalschlüssel" is missing.~~ [Edit: This is wrong as the key is in "032P.n as said above. The key simply doesn't get into the RDF.]
For some DBS-derived entries, the base data has a "Regionalschlüssel" in it, it can not be found in the RDF, though, and probably isn't used for matching with Geo Names.
Some entries might have a "Regionalschlüssel" and can't be mapped to GeoNames as they key is missing there. (I am not sure that these cases actually exist.)

acka47 · 2015-12-17T14:42:22Z

At least 739 entries don't have rs and containedIn (link to GeoNames) because they are entries for moved/renamed libraries: http://beta.lobid.org/organisations/search?q=name:fr%C3%BCher

See also #76.

dr0i · 2015-12-18T07:39:21Z

@hauschke wrote:

I have found 148 institutions without geonames, and subsequently(?) without lat&lon:

How do you have created that data? Seems that many/all ISILs mentioned there in fact do have lat/lon & geonames data.

hauschke · 2015-12-18T11:06:46Z

My mistake. Geonames is missing, lat&long is mostly there. I used #55 (comment) to gather the data and extracted lat&long from geoname.rdf.

fsteeg · 2016-01-15T10:42:39Z

So #92 and #93 look good, we have a new issue #90.

I think we can close this. Assiging to @acka47 for review.

acka47 · 2016-01-18T14:27:49Z

Checking DBS entries as we did for ISIL entries in #93 (comment):

get total of entries with dbsID: https://beta.lobid.org/organisations/search?q=dbsID:* => 12.299
get total of entries with dbsID and geo data : "https://beta.lobid.org/organisations/search?q=dbsID:*%20AND%20location.geo:* => 9.714
DBS-derived entires that are not expected to have geo data (because of missing street address: => 124

=> 2461 (still ~20 %) DBS-derived entries without geo data

acka47 · 2016-09-01T07:45:48Z

\o/ +1

acka47 · 2016-09-01T08:28:43Z

For checking accuracy of OSM lookups, we can at some point automatically compare geo coordinates with regionalschlüssel. E.g. I can easily see querying for libraries in NRW via field rs that at least the two lookups went wrong: http://beta.lobid.org/organisations/search?q=rs:05*

Use correct separation of street address and city in query. This commit also fixes a UI issue for organisations without classification (discovered with updated test data).

acka47 · 2016-09-01T11:58:07Z

Here are some checks based on rs that currently don't look that good:

Schleswig-Holstein: http://staging.lobid.org/organisations/search?q=rs%3A01*
Niedersachsen: http://staging.lobid.org/organisations/search?q=rs%3A03*
Hessen: http://staging.lobid.org/organisations/search?q=rs%3A06*
Rheinland-Pfalz: http://staging.lobid.org/organisations/search?q=rs%3A07*
Baden-Württemberg: http://staging.lobid.org/organisations/search?q=rs%3A08*

To improve lookup result quality and avoid false positives.

fsteeg · 2016-09-01T14:15:59Z

Deployed a new version to staging where these queries look better. The confidence treshold used is too high though, resulting in missing location data for 3056 organisations. Plus, due to a bug, these organisations are skipped entirely. Will continue by fixing the bug and lowering the treshold.

To improve lookup result quality and avoid false positives.

fsteeg · 2016-09-01T16:01:42Z

Processed with treshold of 0.7, yields similar values to current beta, slight improvement:

http://test.lobid.org/organisations/search?q=dbsID:*+AND+_missing_:location.geo
http://beta.lobid.org/organisations/search?q=dbsID:*+AND+_missing_:location.geo

I think we should move the treshold down a little further and check the results again.

acka47 · 2016-09-02T07:11:38Z

Yes, the results look very good. We might even test a lower threshold.

Accept result independent of treshold if street and city match.

fsteeg · 2016-09-02T11:43:27Z

Reprocessed with slightly lower treshold (0.675) and additional rule: if (street AND (postal code OR city)) match, use the result, no matter what the confidence is (I saw cases with missing postal codes in Mapzen but complete street and city match resulting in 0.5 confidence).

Results in 1218 missing locations (1921 in beta):

http://test.lobid.org/organisations/search?q=dbsID:*+AND+_missing_:location.geo

False positives look good too:

http://test.lobid.org/organisations/search?q=rs:01*
http://test.lobid.org/organisations/search?q=rs:03*
http://test.lobid.org/organisations/search?q=rs:05*
http://test.lobid.org/organisations/search?q=rs:06*
http://test.lobid.org/organisations/search?q=rs:07*
http://test.lobid.org/organisations/search?q=rs:08*

fsteeg · 2016-09-02T11:47:23Z

Log output for the 1218 missing locations: missing-geo.txt

acka47 · 2016-09-02T11:55:49Z

+1

Resolves #89 See: http://lobid.org/organisations/search?q=dbsID:*+AND+_missing_:location.geo http://lobid.org/organisations/search?q=rs:01* http://lobid.org/organisations/search?q=rs:03* http://lobid.org/organisations/search?q=rs:05* http://lobid.org/organisations/search?q=rs:06* http://lobid.org/organisations/search?q=rs:07* http://lobid.org/organisations/search?q=rs:08*

fsteeg self-assigned this Dec 16, 2015

fsteeg added ready nwbib-launch and removed ready labels Dec 16, 2015

fsteeg removed their assignment Dec 17, 2015

fsteeg removed nwbib-launch working labels Dec 17, 2015

acka47 mentioned this issue Dec 17, 2015

DBS-derived entries missing "Regionalschlüssel" #90

Closed

This was referenced Dec 18, 2015

Missing new ISILs #92

Closed

ISIL organizations without geo coordinates #93

Closed

fsteeg added the review label Jan 15, 2016

fsteeg assigned acka47 Jan 15, 2016

acka47 changed the title ~~Organizations without geo coordinates~~ DBS-derived organizations without geo coordinates Jan 18, 2016

acka47 added ready and removed review labels Jan 25, 2016

acka47 removed their assignment Jan 25, 2016

fsteeg mentioned this issue May 23, 2016

Varying number of geo fields with same data and code #97

Closed

fsteeg assigned acka47 Sep 1, 2016

acka47 assigned fsteeg and unassigned acka47 Sep 1, 2016

fsteeg added a commit that referenced this issue Sep 1, 2016

Include postcode in geo lookup, use confidence treshold (#89)

df64aac

To improve lookup result quality and avoid false positives.

fsteeg added a commit that referenced this issue Sep 1, 2016

Include postcode in geo lookup, use confidence treshold (#89)

f509d5b

To improve lookup result quality and avoid false positives.

fsteeg added a commit that referenced this issue Sep 1, 2016

Include postcode in geo lookup, use confidence treshold (#89)

8a1975a

To improve lookup result quality and avoid false positives.

fsteeg added a commit that referenced this issue Sep 2, 2016

Improve logging for error analysis (#89)

18c71ca

fsteeg added a commit that referenced this issue Sep 2, 2016

Lower confidence treshold (#89)

bf4d7c9

fsteeg added a commit that referenced this issue Sep 2, 2016

Tweak logging, match address of best result (#89)

f24cc8e

Accept result independent of treshold if street and city match.

fsteeg added a commit that referenced this issue Sep 2, 2016

Tweak logging, match address of best result (#89)

37822d4

Accept result independent of treshold if street and city match.

fsteeg assigned acka47 and unassigned fsteeg Sep 2, 2016

acka47 assigned fsteeg and unassigned acka47 Sep 2, 2016

fsteeg added deploy and removed review labels Sep 6, 2016

fsteeg closed this as completed in #252 Sep 6, 2016

fsteeg removed the deploy label Sep 6, 2016

dr0i mentioned this issue May 23, 2019

Geo lookup is broken #419

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DBS-derived organizations without geo coordinates #89

DBS-derived organizations without geo coordinates #89

acka47 commented Dec 2, 2015 •

edited by fsteeg

Loading

hauschke commented Dec 9, 2015

dr0i commented Dec 10, 2015

fsteeg commented Dec 16, 2015

acka47 commented Dec 17, 2015

fsteeg commented Dec 17, 2015

hauschke commented Dec 17, 2015

acka47 commented Dec 17, 2015

acka47 commented Dec 17, 2015

dr0i commented Dec 18, 2015

hauschke commented Dec 18, 2015

fsteeg commented Jan 15, 2016

acka47 commented Jan 18, 2016

acka47 commented Sep 1, 2016

acka47 commented Sep 1, 2016

acka47 commented Sep 1, 2016 •

edited by fsteeg

Loading

fsteeg commented Sep 1, 2016

fsteeg commented Sep 1, 2016

acka47 commented Sep 2, 2016

fsteeg commented Sep 2, 2016

fsteeg commented Sep 2, 2016

acka47 commented Sep 2, 2016

DBS-derived organizations without geo coordinates #89

DBS-derived organizations without geo coordinates #89

Comments

acka47 commented Dec 2, 2015 • edited by fsteeg Loading

hauschke commented Dec 9, 2015

dr0i commented Dec 10, 2015

fsteeg commented Dec 16, 2015

acka47 commented Dec 17, 2015

fsteeg commented Dec 17, 2015

hauschke commented Dec 17, 2015

acka47 commented Dec 17, 2015

acka47 commented Dec 17, 2015

dr0i commented Dec 18, 2015

hauschke commented Dec 18, 2015

fsteeg commented Jan 15, 2016

acka47 commented Jan 18, 2016

acka47 commented Sep 1, 2016

acka47 commented Sep 1, 2016

acka47 commented Sep 1, 2016 • edited by fsteeg Loading

fsteeg commented Sep 1, 2016

fsteeg commented Sep 1, 2016

acka47 commented Sep 2, 2016

fsteeg commented Sep 2, 2016

fsteeg commented Sep 2, 2016

acka47 commented Sep 2, 2016

acka47 commented Dec 2, 2015 •

edited by fsteeg

Loading

acka47 commented Sep 1, 2016 •

edited by fsteeg

Loading