Refactor integration tests to remove random collection sampling

[mfisher87]

Resolves #215

cc @betolink just getting started on this. I have some sample code that can generate a list of 100 most popular collections, in order, given a provider.

Before I continue, I would like input!

Next steps may be:

• Refactor things to be less quick-and-dirty. First consideration for me is the mismatch between "DAACs" in the tests and "providers" in the script that generates the popular lists.
• Modify the tests to read the new source instead of randomly selecting. Can use first N rows from the file, depending on the test; currently not every test uses the same number of collections. Can create a fixture for providing the data in these files as a mapping of DAACs -> list of collections.
• A "blocklist" of known-bad collections
• ?

[mfisher87]

I just copied this out of my browser dev tools' network tab after doing a similar query in earthdata search client. I'm sure we can run an equivalent query with earthaccess.

[mfisher87]

Worked on this with @itcarroll during hack day. Notes: #755

[mfisher87]

We considered the usefulness of random sampling tests. We don't think we should be doing this for integration tests, especially when they execute on every PR. We could, for example, run them on a cron job and create reports, but that seems like overkill when we have a community to help us identify datasets and connect with the right support channel if there's an issue with the provider.

We may still consider a cron job for, for examle, recalculating the most popular datasets on a monthly basis.

[mfisher87]

We decided we can hardcode a small number and expand the list as we go. Other things like random tests on a cron or updating the list of popular datasets on a cron can be addressed separately.

[mfisher87]

@betolink will take on work to update generate.py to generate top N collections for all providers.

@mfisher87 will continue working on test_onprem_download.py for just NSIDC_ECS for now to make it use the new source of collections.

[mfisher87]

We will update the .txt files to .csv files and add boolean field for "does the collection have a EULA?" and then we'll use that field to mark those tests as xfail.

[mfisher87]

Two major milestones:

1. @danielfromearth updated the script which generates the top collection lists to use all providers supported by earthaccess 🎉 Still TODO: Make them CSVs with a boolean representing whether the collection has a EULA
2. We just got the test_onprem_download.py module working without randomization! 🎉 Still TODO: Refactor the other 3 integration test modules to share this behavior. Let's try and remove duplicate code while we're at it!

Thanks to @DeanHenze and @Sherwin-14 for collaborating on this on today's hackathon!

[mfisher87]

@betolink @chuckwondo This seems reasonable to me, but please validate me :)

[mfisher87]

Thinking about it for like 5 minutes, this is obviously a bad idea. This class is subclassing dict. We'd need to implement like a frozendict.

[mfisher87]

Also still TODO: Run generate.py in GHA on a monthly/quarterly cron and auto-open a PR with the changes to top collections?

[mfisher87]

If we want to determine whether a collection has a EULA, this example was provided:

curl -i -XGET "https://cmr.earthdata.nasa.gov/search/collections.json?concept_id=C1808440897-ASF&pretty=true"

The metadata "eula_identifiers" : [ "1b454cfb-c298-4072-ae3c-3c133ce810c8" ] is present in the response. We're not 100% sure whether this can be used authoritatively. Discussion in progress: https://nsidc.slack.com/archives/C2LRKMDEV/p1724179804149239

[mfisher87]

TODO: Add tests for OBDAAC on-prem open. Related to #828 - we want to make sure the data streams successfully. Opening data from OBDAAC on-prem relies on both #828 and a (potentially) unreleased change to fsspec! (check the September release notes)

[danielfromearth]

Looks like part of this issue may be related to work on EULAs in this issue.