TG2-AMENDMENT_OCCURRENCESTATUS_ASSUMEDDEFAULT

[iDigBioBot]

TestField	Value
GUID	96667a0a-ae59-446a-bbb0-b7f2b0ca6cf5
Label	AMENDMENT_OCCURRENCESTATUS_ASSUMEDDEFAULT
Description	Proposes an amendment of the value of dwc:occurrenceStatus to the default parameter value if dwc:occurrenceStatus, dwc:individualCount and dwc:organismQuantity are empty.
TestType	Amendment
Darwin Core Class	dwc:Occurrence
Information Elements ActedUpon	dwc:occurrenceStatus
Information Elements Consulted	dwc:individualCount
	dwc:organismQuantity
Expected Response	INTERNAL_PREREQUISITES_NOT_MET if dwc:occurrenceStatus is bdq:NotEmpty; FILLED_IN the value of dwc:occurrenceStatus using the bdq:defaultOccurrenceStatus Parameter value if dwc:occurrenceStatus, dwc:individualCount and dwc:organismQuantity are bdq:Empty; otherwise NOT_AMENDED
Data Quality Dimension	Completeness
Term-Actions	OCCURRENCESTATUS_ASSUMEDDEFAULT
Parameter(s)	bdq:defaultOccurrenceStatus
Source Authority	bdq:defaultOccurrenceStatus default = "present"
Specification Last Updated	2024-11-13
Examples	[dwc:occurrenceStatus="", dwc:individualCount="", dwc:organismQuantity="": Response.status=FILLED_IN, Response.result=dwc:occurrenceStatus="present", Response.comment="dwc:occurrenceStatus is bdq:Empty; assumed "Present""]
	[dwc:occurrenceStatus="X", dwc:individualCount="10", dwc:organismQuantity="": Response.status=NOT_AMENDED, Response.result="", Response.comment="dwc:occurrenceStatus is bdq:NotEmpty"]
Source	ALA
References
Example Implementations (Mechanisms)
Link to Specification Source Code
Notes	There is currently a mismatch between https://dwc.tdwg.org/terms/#dwc:occurrenceStatus recommended values and the vocabulary at bdq:sourceAuthority that we are using (https://api.gbif.org/v1/vocabularies/OccurrenceStatus/concepts)

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
Widespread assumption in vouchered occurrence data. Likely to be important when aggregating with any data with 'absent' values. However, this is an amendment: a value of "present" is being provided for dwc:occurrenceStatus when dwc:occurrenceStatus is either empty or not present.

[ArthurChapman]

Why could this not be incorporated under #115 AMENDMENT_OCCURRENCESTATUS_STANDARDIZED?

[Tasilee]

That I would tend to agree with as we could easily add this special case. Comments from others?

[Tasilee]

In retrospect, if we are going to effectively treat EMPTY or an uninterpretable value as "present" then it is indeed an amendment. Sigh.

[tucotuco]

Agreed at TDWG 2018 DQIG meeting that this amendment can only be applied if the the value of dwc:occurrenceStatus is empty.

[ArthurChapman]

I wonder if we should change the name of this test to AMENDMENT_OCCURRENCESTATUS_ASSUMEDDEFAULT to parallel #102. Any comments? I realise the default has only the one possible value - i.e. "Present" but I am attempting to reduce the things we have to define.

[Tasilee]

After reviewing all, I'd agree. Changed accordingly

[chicoreus]

Inconsistency snuck in somewhere in the editing history, this is now clearly labeled as an amendment, but retains an output type of Notification, changing this to Amendment for consistency.

[Tasilee]

Changed "AMENDED" to "FILLED_IN" in accordance with discussions April 16. I also moved the INTERNAL_PREREQUISITES_NOT_MET test into the FILLED_IN part as this aligns with similar amendments.

[Tasilee]

Edited Example 2 as there is no "INTERNAL_PREREQUISITES_NOT_MET". There was an error in the test data, now fixed.

[dwc:occurrenceStatus="X": Response.status=NOT_AMENDED, Response.result="", Response.comment="dwc:occurrenceStatus is not EMPTY"]

[ymgan]

Hey, out of curiousity, may I know why the amendment of occurrenceStatus (to default = "present") does not consider the value of individualCount or organismQuantity and organismQuantityType?

I am thinking that it is possible to have situation where occurrenceStatus is empty but individualCount is >0 individualCount = 0. Please see:

• Improve handling of records declaring absence data gbif/pipelines#268 (comment)

Thank you!!

Edit: sorry, I noticed that I made a mistake in this comment

[ymgan]

Agreed - this simplifies it. @ymgan does this satisfy your issues?

Yes, thank you very much for your hard work here! I really appreciate it!

[Tasilee]

I have updated the Description and the Examples accordingly and will amend the test data.

[Tasilee]

I have added dwc:individualCount and dwc:organismQuantity to the Information Elements.

[Tasilee]

Restructured Parameter(s) and Source authority

[ArthurChapman]

Change sourceAuthority from "dwc:occurrenceStatus = "present"" to "dwc:occurrenceStatus default = "present""

[Tasilee]

Changed all Information Elements to "ActedUpon" as per Paul's Java Code.

@chicoreus: You will need to amend your code to include dwc:individualCount and dwc:organismQuantity ?

[chicoreus]

The parameter can't be the same as an information element.

Propose changing the parameter from dwc:occurrenceStatus to bdq:defaultOccurrenceStatus

[Tasilee]

Thanks @chicoreus - that seems a reasonable solution to me. Amending.

[Tasilee]

Changed Expected Response from

FILLED_IN the value of dwc:occurrenceStatus using the Parameter value if dwc:occurrence.Status, dwc:individualCount and dwc:organismQuantity are EMPTY; otherwise NOT_AMENDED

to

FILLED_IN the value of dwc:occurrenceStatus using the Parameter value if dwc:occurrenceStatus, dwc:individualCount and dwc:organismQuantity are EMPTY; otherwise NOT_AMENDED

[ymgan]

May I know if we need a VALIDATION_ORGANISMQUANTITY_NOTEMPTY please? We already have

• TG2-VALIDATION_INDIVIDUALCOUNT_NOTEMPTY #232

[ArthurChapman]

Thanks @ymgan - #232 is Supplementary at this stage and another test for VALIDATION_ORGANISMQUANTITY_NOTEMPTY could be valuable for some, but at this stage we don't think it is widely applicable. But it is certainly one worth considering in the future if required. There are quite a few tests in a similar position that we don't believe are CORE.

[ymgan]

Thanks @ArthurChapman !! Good morning :D To make sure that I understand, even if this test is core and its prerequisite include individualCount and organismQuantity are empty, it does not mean we need the notempty tests for individualCount and organismQuantity. Am I correct?

[chicoreus]

On Mon, 19 Aug 2024 07:10:09 -0700 Yi-Ming Gan ***@***.***> wrote: Thanks @ArthurChapman !! Good morning :D To make sure that I understand, even if this test is core and its prerequisite include individualCount and organismQuantity are empty, it does not mean we need the notempty tests for individualCount and organismQuantity. Am I correct?

Non-empty tests for individualCount and organismQuantity would ba aspirational at this point. If we adopted them we would be asserting that these terms would be important enough for everyone to try to put in the effort to populate them. For natural science collections data at least, this would be non-trivial, collections may know how many parts they have for some specimen, but not be readily able to work out how many individuals those represent. So, yes, these do make natural related tests, but not really within the scope of what we want to accomplish. Others, for whom quality in this portion of the data, can easily propose a use case and suite of tests.

[ymgan]

got it, thanks @chicoreus !

[chicoreus]

This and #102 are "AssumedDefault" tests for which non-empty values aren't preventing execution of the test, that should probably both have the internal prerequisites clause removed and be able to reach the NOT_AMENDED clause:

FILLED_IN the value of dwc:occurrenceStatus using the Parameter value if dwc:occurrenceStatus, dwc:individualCount and dwc:organismQuantity are bdq:Empty; otherwise NOT_AMENDED

[Tasilee]

I agree with removing the INTERNAL_PREREQUISITES_NOT_MET phrase on this Test.

[Tasilee]

Do we change default to "Present" for now as "present" won't currently validate against the GBIF vocabulary?

[chicoreus]

On Mon, 11 Nov 2024 18:48:40 -0800 Lee Belbin ***@***.***> wrote: Do we change default to "Present" for now as "present" won't currently validate against the GBIF vocabulary?

Yes. Good catch. Probably worth adding a note as well.

[Tasilee]

Added to Notes " There is currently a mismatch between https://dwc.tdwg.org/terms/#dwc:occurrenceStatus recommended values and the vocabulary at bdq:sourceAuthority that we are using (https://api.gbif.org/v1/vocabularies/OccurrenceStatus/concepts)"

[chicoreus]

Corrected the parameter namespace for bdq:defaultOccurrenceStatus from dwc to bdq.