occurrenceStatus, individualCount checks

[ymgan]

obis-qc checks

Check	Fields	Flags	Absence	Dropped	Vandepitte et al. flag number
occurrenceStatus should be present.	occurrenceStatus
occurrenceStatus should be absent or present.	occurrenceStatus		x
If individualCount equals 0, record is absence.	individualCount		x

TDWG BDQ TG2 Tests

Validation

• TG2-VALIDATION_OCCURRENCESTATUS_STANDARD tdwg/bdq#116
• TG2-VALIDATION_OCCURRENCESTATUS_NOTEMPTY tdwg/bdq#117

Amendment

• TG2-AMENDMENT_OCCURRENCESTATUS_ASSUMEDDEFAULT tdwg/bdq#75
• TG2-AMENDMENT_OCCURRENCESTATUS_STANDARDIZED tdwg/bdq#115

Implementation

• Improve handling of records declaring absence data gbif/pipelines#268 (comment)

individualCount	occurrenceStatus	inferred occurrenceStatus	flag
NULL	NULL	PRESENT
NULL	present*	PRESENT
NULL	absent*	ABSENT
NULL	rubbish	PRESENT	OCCURRENCE_STATUS_UNPARSABLE
>0	NULL	PRESENT	OCCURRENCE_STATUS_INFERRED_FROM_INDIVIDUAL_COUNT
>0	present*	PRESENT
>0	absent*	ABSENT	INDIVIDUAL_COUNT_CONFLICTS_WITH_OCCURRENCE_STATUS
>0	rubbish	PRESENT	OCCURRENCE_STATUS_UNPARSABLE,
0	NULL	ABSENT	OCCURRENCE_STATUS_INFERRED_FROM_INDIVIDUAL_COUNT
0	present*	PRESENT	INDIVIDUAL_COUNT_CONFLICTS_WITH_OCCURRENCE_STATUS
0	absent*	ABSENT
0	rubbish	ABSENT	OCCURRENCE_STATUS_UNPARSABLE, OCCURRENCE_STATUS_INFERRED_FROM_INDIVIDUAL_COUNT
rubbish	NULL	PRESENT	INDIVIDUAL_COUNT_UNPARSABLE
rubbish	present*	PRESENT	INDIVIDUAL_COUNT_UNPARSABLE
rubbish	absent*	ABSENT	INDIVIDUAL_COUNT_UNPARSABLE
rubbish	rubbish	PRESENT	INDIVIDUAL_COUNT_UNPARSABLE, OCCURRENCE_STATUS_UNPARSABLE

*= or similar values

GBIF flags

https://github.com/gbif/gbif-api/blob/dev/src/main/java/org/gbif/api/vocabulary/OccurrenceIssue.java

/**
   * Example: individual count value > 0, but occurrence status is absent.
   */
  INDIVIDUAL_COUNT_CONFLICTS_WITH_OCCURRENCE_STATUS(WARNING, DwcTerm.individualCount),

  /**
   * Occurrence status value can't be assigned to {@link OccurrenceStatus}
   */
  OCCURRENCE_STATUS_UNPARSABLE(WARNING, DwcTerm.occurrenceStatus),

  /**
   * Occurrence status was inferred from the individual count value
   */
  OCCURRENCE_STATUS_INFERRED_FROM_INDIVIDUAL_COUNT(WARNING, DwcTerm.occurrenceStatus),

  /**
   * Occurrence status was inferred from basis of records
   */
  OCCURRENCE_STATUS_INFERRED_FROM_BASIS_OF_RECORD(WARNING, DwcTerm.occurrenceStatus),

The definition of the flags are the document as above: https://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/OccurrenceIssue.html

occurrenceStatus dictionary in GBIF: https://github.com/gbif/parsers/blob/master/src/main/resources/dictionaries/parse/occurrence_status.tsv

Related issues

• Infer individualCount based on organismQuantity when organismType = individuals, to include counts in the simple download format gbif/pipelines#855
• populate occurrence fields based on MeasurementOrFact records obis-issues#197
• Use restricted vocabulary when possible obis-issues#190

[ymgan]

I guess we will get to the question on how do we deal with the interpreted values. Do we replace the value (e.g. dwc:occurrenceStatus) when it is inferred based on individualCount? I do not have the answer and do not have the time to go through all the comments in other issues. Linking relevant issue here:

• Process of running an AMENDMENT tdwg/bdq#196

From 2023-02-16 meeting:

• OBIS portal does present interpreted value, but needs to know where to look for. Interpreted values are not included in downloads.
• Team agrees to align the test occurrence_status_inferred_from_individual_count and use the same flags as GBIF whenever possible.
• Acknowledge that good documentation is needed to document the flags and their definition.

From 2023-03-16 meeting:

• Agrees to not amend occurrenceStatus value and populate a custom field "absence" from OBIS. The field is used to hide absence data from map by default.