Non-obvious local file loading paths

[gsantia]

It took me a bit to figure out how to fetch a local file for an extract config, because it's not really clear in the documentation or the library what to use for the working directory. My directory structure looks like this:

my_study/
└── data/
    └── gen_manifest.csv
    └── s3_scrape.csv
└── ingest_package/
    └── extract_configs/
       └── genomic.py

After a bit of fooling around, I managed to get what I wanted (load gen_manifest.csv for extraction and use s3_scrape.csv to supplement it), but it was really just after guessing what the paths should be. Here's what I mean:

genomic.py

source_data_url = "file://../../data/gen_manifest.csv"

def do_after_read(df):
    s3_scrape = pd.read_csv("data/s3_scrape.csv")
    merged_df = df.merge(
        right=s3_scrape,
        left_on="filepath",
        right_on="Filepath",
        how="inner"
    )
    return merged_df

So using the file protocol I have to go up two directories, but for reading with pandas I don't. Changing how this works in the library probably breaks some previous extract configs, so I assume a change to the documentation is preferable.

[fiendish]

Side question: Is there a reason why you're loading a second file manually and merging it in do_after_read instead of combining them in the transform module?

[gsantia]

Side question: Is there a reason why you're loading a second file manually and merging it in do_after_read instead of combining them in the transform module?

No. I haven't done an ingest in a while, so I was referencing the Chung one I did almost a year ago. But I did it there because I was again referencing someone else's ingest package. So it seems like it happens sometimes.

[fiendish]

It seems like any compatibility concerns from a code change could be addressed by #504

[gsantia]

Is changing the code preferable? I don't mind adding a snippet to the documentation.

[fiendish]

It might be!