Transformation function raises FileNotFoundError regarding data YAML file

[eecavanna]

Summary

While running a transformation function named fix_award_dois from within a Python notebook that imports the containing Python class from the nmdc-schema Python package on PyPI, Python raised a FileNotFoundError exception because it couldn't find a file at the path specified.

Screenshot

Related snippets

Invocation of load_yaml_file function:

nmdc-schema/nmdc_schema/migration_recursion.py

Lines 332 to 333 in 918056b

	study_doi_data = load_yaml_file(
	filename='assets/misc/study_dois_changes.yaml')

Definition of load_yaml_file function:

nmdc-schema/nmdc_schema/migration_recursion.py

Lines 90 to 94 in 918056b

	def load_yaml_file(filename):
	"""Loads a YAML file into a Python dict."""
	with open(filename, "r") as f:
	data = yaml.safe_load(f)
	return data

Proposal

Reference the file in a way that works regardless of where the function is being invoked.

Suggestion (untested):

+ from importlib import resources

# ...

 def load_yaml_file(filename): 
     """Loads a YAML file into a Python dict.""" 
-    with open(filename, "r") as f: 
+    with resources.open(filename, "r") as f: 
        data = yaml.safe_load(f) 
     return data 

Docs:

• Official: https://docs.python.org/3/library/importlib.resources.html#module-importlib.resources
• Official: https://docs.python.org/3/library/importlib.resources.abc.html#importlib.resources.abc.Traversable.open
• Third-party: https://dev.to/bowmanjd/easily-load-non-python-data-files-from-a-python-package-2e8g

[eecavanna]

As a workaround for the current migration, I can download the referenced file from the nmdc-schema GitHub repo and put it at the path assets/misc/study_dois_changes.yaml relative to the Python notebook.

[turbomam]

Thanks @eecavanna

@brynnz22 would be it be OK for me to change the assignee to you?

[eecavanna]

The underlying issue is still present in nmdc-schema, and will come into play again the next time code that imports the nmdc-schema Python package calls the load_yaml_function (or calls something that calls that function). That's because that function uses Python's open() function with a relative path, which is relative to the current working directory.

Python docs about open(): https://docs.python.org/3/library/functions.html#open

[turbomam]

apologies for closing

[eecavanna]

For future reference: I found this code in the nmdc-schema repo (while working on something else). Maybe the same approach (i.e. way of getting the contents of a file) can be used in this case.

nmdc-schema/nmdc_schema/nmdc_data.py

Line 28 in 387ed3b

return io.BytesIO(pkgutil.get_data(__name__, "nmdc_schema_merged.yaml"))

Docs: https://docs.python.org/3/library/pkgutil.html#pkgutil.get_data