GBIF download notes

[eculler]

Sorry for the delay on the video for the GBIF download! If folks need some more time on that, that's totally ok -- we'll have a week before you start your final assignments to revisit any portfolio posts that need some extra time. In the meantime, there have been a couple of common issues:

1. You don't need to modify the code that stores your credentials. It seems like folks have been trying to write credentials into the code, which we want to avoid because it publishes your password on GitHub! Instead, run the code cell, and you will be prompted (waay up at the top and hard to see :/ ) to enter your credentials. Your credentials should then be saved until you restart your Codespace.
2. If you accidentally enter your credentials incorrectly, you can set reset_credentials=True and re-run the cell.
3. The download code is meant to work if you have to restart and stop, or if you have an error and re-run the code, but it actually only works if you get it right the first time. Go ahead and use this code instead if you have to re-run the download:

# Only download once
gbif_pattern = os.path.join(gbif_dir, '*.csv')
if not glob(gbif_pattern):
    # Only submit one request
    if not 'GBIF_DOWNLOAD_KEY' in os.environ:
        # Submit query to GBIF
        gbif_query = occ.download([
            "speciesKey = ",
            "hasCoordinate = ",
            "year = ",
        ])
        os.environ['GBIF_DOWNLOAD_KEY'] = gbif_query[0]

    # Wait for the download to build
    download_key = os.environ['GBIF_DOWNLOAD_KEY']
    wait = occ.download_meta(download_key)['status']
    while not wait=='SUCCEEDED':
        wait = occ.download_meta(download_key)['status']
        time.sleep(5)

    # Download GBIF data
    download_info = occ.download_get(
        os.environ['GBIF_DOWNLOAD_KEY'], 
        path=data_dir)

    # Unzip GBIF data
    with zipfile.ZipFile(download_info['path']) as download_zip:
        download_zip.extractall(path=gbif_dir)

# Find the extracted .csv file path (take the first result)
gbif_path = glob(gbif_pattern)[0]

[ZimmZone]

Hi Elsa,

I have tried this and I receive the following error.

Exception: error: , with error status code 401check your number of active downloads.

[annaposlednik]

Having the same issue, please let me know if you resolve the error!

[gpb3037]

me too...

[BioNCphilic]

It would help to know which notebook # to use when applying this code.

[ZimmZone]

Hi Everyone - it turns out that my error was caused because I accidentally put in the incorrect credentials into the password at the top of the page (when logging into GBIF). I changed the updated credentials to = True and reset them and it seems to have worked. @hanried also posted a different solution for the same error in the '401 Unauthorized Error' discussion thread.

Also - DO NOT put your credentials and passwords into the code! When you run the code a little box will drop down from the top of the window and you enter the credentials there. I made this mistake the first time and Elsa helped me sort it out in office hours.

[byandell]

Not sure if this indicates problems with credentials or problem with arguments to occ.download()

Exception Traceback (most recent call last)
Cell In[17], line 7
3 if not glob(gbif_pattern):
4 # Only submit one request
5 if not 'GBIF_DOWNLOAD_KEY' in os.environ:
6 # Submit query to GBIF
----> 7 gbif_query = occ.download([
8 "speciesKey = 2474953",
9 "hasCoordinate = True"
10 "year = 2023",
11 ])
12 os.environ['GBIF_DOWNLOAD_KEY'] = gbif_query[0]
14 # Wait for the download to build

File /opt/conda/lib/python3.11/site-packages/pygbif/occurrences/download.py:271, in download(queries, format, user, pwd, email, pred_type)
268 for predicate in keyval:
269 req.add_predicate_dict(predicate)
--> 271 out = req.post_download(user, pwd)
272 return out, req.payload

File /opt/conda/lib/python3.11/site-packages/pygbif/occurrences/download.py:482, in GbifDownload.post_download(self, user, pwd)
475 r = requests.post(
476 self.url,
477 auth=requests.auth.HTTPBasicAuth(user, pwd),
...
488 )
489 else:
490 self.request_id = r.text

Exception: error: , with error status code 401check your number of active downloads.

[eculler]

@byandell this error seems to come up in a couple of situations, but the most common one is an issue with the credentials. So far, I've seen it in the following situations:

1. Used GitHub or Google to create the account on GBIF. I wasn't ever able to get an account created with these services to work with the API -- make sure to create the account with a username, email, and password. You can link services after to fact to handle log ins, but not create the account with them.
2. Incorrect credentials -- set reset_credentials=True and re-enter the credentials. The email must also match!
3. When there actually are too many active downloads...you can check this in your account page on the GBIF website. If you use the corrected download code, it doesn't create a new download every time, which helps avoid this problem.

[byandell]

I think I solved this. The issue is that I had set GBIF_USER to my email and it may have gotten confused by the @ sign. I changed GBIF_USER to yandell.brian and it seems to be working now. Having said that, the download is at 20m and counting.