-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TG2-VALIDATION_COORDINATESSTATEPROVINCE_CONSISTENT #56
Comments
Comment by Lee Belbin (@Tasilee) migrated from spreadsheet: |
Comment by Arthur Chapman (@ArthurChapman) migrated from spreadsheet: |
Comment by John Wieczorek (@tucotuco) migrated from spreadsheet: |
Comment by Paula Zermoglio (@pzermoglio) migrated from spreadsheet: |
Comment by Arthur Chapman (@ArthurChapman) migrated from spreadsheet: |
Difficult to get a standard vocabulary for StateProvince names and boundaries that work. |
Just to add to the difficulty of getting standard vocabulary, data from some country may be in another language and it could also use another alphabets and I don't think they should be flagged as INCONSISTENT. |
@cgendreau if the stateProvince name string cannot be found in either the GIS data source or a thesaurus used to find variant and internatinonalized forms of the names, then the expectation would be that this test would return a result status of data/internal prerequisites not met, with no value for result value, rather than a result value of compliant or not compliant (noting that validation result values under the framework are only CONSISTENT or INCONSISTENT). |
I wonder - rather than using TGN - we can use the ISO subdivision codes ISO 3166-2 (https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes) as the Default authority. What do you think here @tucotuco ? |
I think name matching is not the actual issue here, and not the authority we are after for this test. This test should use a standardized stateProvince to do the lookup, and so should ideally pass through geography standardization first. TGN is probably not the right authority, because it can't do the spatial intersection needed. An authority service based on GADM would be great. I do not know of a production level one. |
Level 1 layers seem to be available, but Level 2 and lower not apparently. However the UN is apparently preparing a Level 2 DB at 1:1 million (https://www.unsalb.org/) The FAO GAUL is (http://www.fao.org/geonetwork/srv/en/metadata.show%3Fid%3D12691) but I understand the licencing is a problem with its use "The GAUL always maintains global layers with a unified coding system at country, first (e.g. departments) and second administrative levels (e.g. districts). Where data is available, it provides layers on a country by country basis down to third, fourth and lowers levels". ESRI has a World Administrative Divisions (to first level) at https://www.arcgis.com/home/item.html?id=f0ceb8af000a4ffbae75d742538c548b. There are also some OpenStreetmap layers that I haven't looked at but appear to be Vector layers only and in Mercator projections |
Thanks @tucotuco and @ArthurChapman: If there isn't a current service that can provide spatial intersection at Level 2, this test is not operable? |
The Google Maps API can return geocoding information at administrative levels more specific than country - administrative_area_level_1 is the equivalent for dwc:stateProvince. So, this is theoreticaly operable. |
Level 1 doesn't seem a big problem. Level 2 is still a long way off globally. Looking at the datasets available (https://www.unsalb.org/data?page=3) so far only about 27 out of 197 countries are covered. Google Maps seems a good option for Level 1 (I haven't checked - but do they include Level 2 at all (for example for the 27 countries that SALB have?)) |
Level 2 exists, but I do not know what the coverage is, not do I know where
to find out what the coverage is.
…On Fri, Aug 9, 2019 at 7:06 PM Arthur Chapman ***@***.***> wrote:
Level 1 doesn't seem a big problem. Level 2 is still a long way off
globally. Looking at the datasets available (
https://www.unsalb.org/data?page=3) so far only about 27 out of 197
countries are covered. Google Maps seems a good option for Level 1 (I
haven't checked - but do they include Level 2 at all (for example for the
27 countries that SALB have?))
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#56?email_source=notifications&email_token=AADQ725IMCJZZKIAS3JHCBLQDXS6VA5CNFSM4EKSMWV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD374OOQ#issuecomment-520079162>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADQ7232HGDTQPA6EOTCLC3QDXS6VANCNFSM4EKSMWVQ>
.
|
Wouldn't that take another test though. One would need to test against something to decide if it was valid or not. I don't think this is what we were intending. INTERNAL_PREREQUISITES_NOT_MET is EMPTY is just testing if something is there or not - if nothing there the test can't be run. To determine if it is valid or not would require some sort of further testing |
It would require multiple other tests, and I don't think this is an isolated example. The coordinates might have to be interpreted first. The point is that the test can not be run meaningfully unless all of the right conditions are met, and having real coordinates is definitely a requirement for running the test. |
Can't go through all the tests at the moment - but don't we have another test that tests for validity of Coordinates? |
… On Fri, Jan 27, 2023 at 7:10 AM Arthur Chapman ***@***.***> wrote:
Can't go through all the tests at the moment - but don't we have another
test that tests for validity of Coordinates?
—
Reply to this email directly, view it on GitHub
<#56 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADQ725Q3ZGBY3C2QN6GYTTWUONINANCNFSM4EKSMWVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I agree that it would be good for something that is not a valid coordinate would mean that the INTERNAL_PREREQUISITES_NOT_MET. I suppose my initial reluctance was looking at things in order - i.e. EXTERNAL_PREREQUISITES_NOT_MET - can't go any further. If this isn't the case - move to the next step - i.e. EMPTY so INTERNAL_PREREQUISITES_NOT_MET - go to the test. But - this is probably not necessarily how it works, as while running the test you determine that they values can't be interpreted as coordinates for the test - so fails. In this case it is because the INTERNAL_PREREQUISTES_NOT_MET and that would be noted in the Response.result as such rather than "RUN_HAS_RESULT with Response.result as NOT_COMPLIANT" Bottom Line: I agree that "INTERNAL_PREREQUISITES_NOT_MET if the values of dwc:decimalLatitude, dwc:decimalLongitude are EMPTY or not valid ..." is good (also in #50). Perhaps we need a third example - something like [dwc:stateProvince="Neuquén", dwc:decimalLatitude="-141.0525925872862", dwc:decimalLongitude="-71.5310546742521", dwc:geodeticDatum="": Response.status=INTERNAL_PREREQUISTES_NOT_MET, Response.comment="Input fields contain invalid values"] |
Thanks @ArthurChapman and @tucotuco. Do I take it that there is agreement to my suggestion above (#56 (comment)) that we do need to add "or not valid" to this and similar tests? |
@Tasilee From me, yes. |
@Tasilee Yes, I'll concur as well. For this test, data have quality if the coordinates match the state/province. Data lack a particular kind of property if they coordinates are not consistent with the state province. By considering validity of the coordinates (and state/province) within the path for internal prerequisites not met, a failure (NOT_COMPLIANT) of this test isolates a class of problematic data - those where the coordinates exist, the state/province is known, and the coordinates fall outside of the state/province. For data quality control, isolating this class of problem has value, and other tests can highlight problematic data where the coordinates are invalid or the state/province is unknown. |
Yes from me too |
Changed ER and added 3rd example |
I changed the third example - in the response from "fields" to "field" as only one is invalid. |
I changed "not valid" in the Expected Response to "invalid" to standardize the phrasing. |
Restructured Parameter(s) and Source authority |
Post Zoom 11/7/2023, I have aligned the Source Authority with the suggested syntax: bdq:sourceAuthority default = "ADM1 boundaries" {[https://gadm.org]} |
Splitting bdqffdq:Information Elements into "Information Elements ActedUpon" and "Information Elements Consulted". Also changed "Field" to "TestField" and "Output Type" to "TestType". This one I am also not sure about. Please check. |
@Tasilee If it is a validation, nothing is acted upon, no? Or am I missing the point? |
@tucotuco - If I interpreted the Zoom conversation correctly with @chicoreus last week, my thought was what Information Element/s were the FOCUS of the 'test' (=ActedUpon). Originally, I also wondered if all the Information Elements associated with VALIDATIONs and ISSUEs would be 'Consulted'. @chicoreus ? It would be good to get this nailed down as I would like to finish all the edits ASAP this week. |
On Sat, 16 Sep 2023 08:20:30 -0700 John Wieczorek ***@***.***> wrote:
@Tasilee If it is a validation, nothing is acted upon, no? Or am I
missing the point?
ActedUpon indicates which specific information elements a test is making an assertion about.
Consulted are other information elements that were examined to make an assertion, but are not ones that a test makes an assertion about.
Think of highlighting cells that either have NOT_COMPLIANT or COMPLIANT assertions made upon them by a validation in a spreadsheet of flat darwin core, ActedUpon are cells about which such an assertion is made. Consulted cells were examined, but assertions were not made about them. Many validations have only a single information element that is ActedUpon, and no information elements that are Consulted.
For a guide to which information elements are ActedUpon and which are Consulted, see the @ActedUpon and @consulted annotations in the method signatures in the java implementations.
|
Changed all Information Elements to "ActedUpon" as per Paul's Java Code |
Standardized reference to bdq:sourceAuthority in Expected Response to "EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available" |
Amended TERM-ACTION COORDINATES_STATE-PROVINCE_CONSISTENT to COORDINATES_STATEPROVINCE_CONSISTENT |
…nce name in spatial data set.
Removing hyphen or underscore from names/labels to make TERM_ACTION consistent |
…ations: tdwg/bdq#56 tdwg/bdq#59 tdwg/bdq#187, noting in 59 potential of change needed as specification does not conform to Darwin Core value (which may just need rationale management in the issue, or may need a change).
The text was updated successfully, but these errors were encountered: