Update post-transform methods to handle None values

[jonavellecuerdo]

Purpose and background context

An error was encountered when trying to run the latest updates to the transformer against the full gisogm source:

File "/Users/jcuerdo/Documents/repos/transmogrifier/transmogrifier/sources/transformer.py", line 256, in _transform
    fields = self.create_locations_from_spatial_subjects(fields)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcuerdo/Documents/repos/transmogrifier/transmogrifier/sources/transformer.py", line 417, in create_locations_from_spatial_subjects
    if subject_location not in fields.setdefault("locations", []):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

When a field method returns None values (resulting from or None statements), the fields dictionary will be populated with a key-value pair formatted as <field_name>: None. The fields.setdefault(<field_name>, []) method will only set values to an empty list if the key is not present. The solution proposed is to have post-transform methods (a) check if the value for a field in fields is None and (b) if None, set the value to an empty list.

How can a reviewer manually see the effects of these changes?

1. Temporarily set AWS credentials for TimdexManagers for Dev1 in your terminal.

2. Run transform on full set of gisogm records:

   • Run the following command from your local clone of transmogrifier:

     pipenv run transform -i s3://timdex-extract-dev-222053980223/gisogm/gisogm-2024-03-01-full-extracted-records-to-index.jsonl -o output/output_gisogm.json -s gisogm -v
     

3. Verify completion without error!

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/GDT-217
• https://mitlibraries.atlassian.net/browse/GDT-210

Developer

• All new ENV is documented in README
• All new ENV has been added to staging and production environments
• All related Jira tickets are linked in commit message(s)
• Stakeholder approval has been confirmed (or is not needed)

Code Reviewer(s)

• The commit message is clear and follows our guidelines (not just this PR message)
• There are appropriate tests covering any new functionality
• The provided documentation is sufficient for understanding any new functionality introduced
• Any manual tests have been performed and verified
• New dependencies are appropriate or there were no changes

[ghukill]

Approved!

I like the added tests, specifically that they test these new "enrichment" (name still TBD) methods specifically.

And as mentioned in discussions, I think this is a good solution given our current architecture, and we can revisit (if needed) in any future refactorings.

[ehanson8]

Great tests!