Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

alexklwong · 2020-03-25T08:42:11Z

I want to download a 16GB zip file on Google Drive using:

gdown https://drive.google.com/uc?id=1InfIal4y7OBMGNUDeldEmDxtD0MrewY8

I have already set the permissions on the file to ``Anyone with the link''

But I get this error:

Permission denied: https://drive.google.com/uc?id=1InfIal4y7OBMGNUDeldEmDxtD0MrewY8
Maybe you need to change permission over 'Anyone with the link'?

mrlzla · 2020-03-25T12:16:30Z

The same error

zenithfang · 2020-03-26T00:35:36Z

same error

wkentaro · 2020-03-31T06:08:11Z

Maybe the same issue as #26.

I found this happens with large data + many access (e.g., public dataset) #42. However, I have no idea how to fix this.

If they are the same issue, there's no solution at the moment from gdown side.

Rmao99 · 2020-03-31T06:12:48Z

I have this issue as well with a 15GB dataset and a 4.3 GB dataset. But when I copy the 4.3GB to my local drive and manually get the link on my copied dataset it works. Can't make a copy of the 15GB on free drive though.

For reference the datasets im downloading are:
https://drive.google.com/file/d/1NrqOLbMa_RwHbG3KIXJFWLnlND2kiIpj/view
https://drive.google.com/file/d/15w0q3Nuye2ieu_aUNdTS_FNvoVzM4RMF/view

I was actually able to successfully download both datasets once from the original links but after needing to redo the downloads they no longer work even when uninstalling and reinstalling gdown

wkentaro · 2020-03-31T06:18:31Z

@Rmao99 Thanks for the information. That behavior makes sense if that's the issue of Google-Drive's side. It seems they restrict access from the command-line for data with many access (e.g., public dataset).

wkentaro · 2020-04-29T17:33:01Z

I've added a new feature to keep cookies #51.
I hope this fixes this issue.

You can try pre-released version of this.

pip install -U --no-cache-dir gdown --pre

by default, cookies is enabled and saved into ~/.cache/gdown/cookies.json.
You can disable it by gdown --no-cookies XXX

Jackie-LJQ · 2020-05-05T00:41:40Z

I have the same issue. I tried the above version, not fix.

whulixiya · 2020-06-01T03:17:46Z

I met the same error when I try to download many big files from my google drive, have anyone fixed it?

carlmalamud · 2020-06-03T14:23:24Z

I find if I rm ~/.cache/gdown/cookies.json and then restart my script, things start downloading again. Is your cookie expiring perhaps? I'm good for 1,000 or so files before I get the error.

Thanks for gdown. Very helpful!

wkentaro · 2020-06-03T16:28:32Z

@carlmalamud Thanks for info. Do you have any sharable links to reproduce this?

whulixiya · 2020-06-04T00:34:45Z

Unfortunately, there is nothing in ~/.cache/gdown/

…

On Wed, Jun 3, 2020 at 10:23 PM Carl Malamud ***@***.***> wrote: I find if I rm ~/.cache/gdown/cookies.json and then restart my script, things start downloading again. Is your cookie expiring perhaps? I'm good for 1,000 or so files before I get the error. Thanks for gdown. Very helpful! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDECDYFMKEGB3BS3FBLVETRUZMGZANCNFSM4LTI65IQ> .

-- *Xiya Li* *State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China* *(M**aster**)*

carlmalamud · 2020-06-04T18:32:29Z

@wkentaro What is happening is your cookie keeps growing and growing until it is full. I believe every time there is a warning, a new line gets added. I've seen my cookie grow from 11 warnings to 36 warnings in about 2 hours. After about 1,000 files, there are presumably enough that there isn't room for more warnings, so you when present the cookie that doesn't mention a particular download, Google rejects it and your code is thinking that it is a permission issue.

Perhaps if your cookie fails, you could remove it, try again, and only then if it fails a second time assume it is a permissions issue.

Here is a snippet of what my cookie looks like:

[
[
"AUTH_22nmp4ombsvnvmjrl3nb3u3hsrucc95e",
"14651752893030988712Z|1591294950000|hospjjjbfh4rapv59tff247rc9fhcbr3"
],
[
"NID",
"204=x4h9LoiFMtrKRVT3aYaAXNxcePhv5SjsTw9qrioaMhrWD61BLoO_hbjosmcIBEYKSEf-pqz0U5D9SyUKwCiHJJ1Ys_xGBVg8oLqizpJaVYdWUoSIgSZKNL3xpkoGvIQs9lz8hTQD9EgvcwunJ--j5FtM7OPAn_W8mnVaHEVweSs"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQEMExkMlBpOUJoQ28",
"IcVe"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQEMmVDYVJUckNQZlE",
"Ricd"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQENGlEZDNabFRwVzg",
"579P"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQENk9JWmNMNm81R0k",
"o4Bf"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQENlVqTnJsc2R5dzA",
"osl-"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQEOGpkZzFad25SdFE",
"8XbE"
],

carlmalamud · 2020-06-06T21:15:25Z

It looks like the magic number is 110 download_warnings and the cookie is full.

wkentaro · 2020-06-06T21:22:48Z

@carlmalamud thanks!

wkentaro · 2020-06-06T21:28:48Z

Sent a PR here: #55

wkentaro · 2020-06-06T21:32:50Z

Released as v3.11.1

wkentaro · 2020-06-06T21:33:35Z

@carlmalamud can you try if the new version eases the issue?

grudloff · 2020-06-10T18:26:03Z

Same problem, upgraded to latest release and I get the following message:

Access denied with the following error:

 	Too many users have viewed or downloaded this file recently. Please
	try accessing the file again later. If the file you are trying to
	access is particularly large or is shared with many people, it may
	take up to 24 hours to be able to view or download the file. If you
	still can't access a file after 24 hours, contact your domain
	administrator. 

You may still be able to access the file from the browser:
%%%URLHERE

carlmalamud · 2020-06-16T20:39:10Z

I was chugging along nicely for a while, but I now am also getting same error as @grudloff. I was in fact able to pull the file up in a browser and start the download as indicated above. I looked in the gdown cookie and didn't see any download_warning lines, so evidently we never got the cookie back from them with the additional authorization.

(I have now installed 3.11.1, but was running on 3.11.0 before, I simply inserted a "rm ~/.cache/gdown/cookies.json" every fifty lines. Sorry, it was working so I didn't want to bother it. :). The upgrade doesn't seem to help on the above error.

carlmalamud · 2020-06-16T20:46:19Z

I am imagining that the error might have something to do with my being an authenticated user in the browser and not via gdown. I'm not sure is presenting an oauth token might make a difference here?

wkentaro · 2020-06-17T00:39:23Z

I'm not sure is presenting an oauth token might make a difference here?

I don't think so, gdown uses requests inside without using any browsers.

kshitijagrwl · 2020-07-06T09:30:01Z

Solved by disabling certificate verification by changing line 111 in download.py

res = sess.get(url, stream=True, verify=False)

However it does display this warning
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning

wkentaro · 2020-07-06T12:44:40Z

Solved by disabling certificate verification

In my case, it didn't work.

aasyedd1 · 2024-01-16T13:04:00Z

@e96031413 worked like a charm. thanks alot

neeeal · 2024-01-17T13:36:40Z

pip install gdown==4.6.0

actually solved my issue. thanks

wkentaro · 2024-01-20T04:05:58Z

This PR #300 should help this issue in some regard. I also added FAQ to the README.
https://github.com/wkentaro/gdown?tab=readme-ov-file#i-set-the-permission-anyone-with-link-but-still-cant-download

Releasing v5.0.0 soon.

pmeier · 2024-01-29T08:39:33Z

When hitting large files that GDrive wants extra confirmation before it starts the download, e.g. https://drive.google.com/uc?id=1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK, we currently hit this branch:

gdown/gdown/download.py

Lines 31 to 35 in ae122f4

    
           m = re.search('id="download-form" action="(.+?)"', line) 
        
           if m: 
        
               url = m.groups()[0] 
        
               url = url.replace("&amp;", "&") 
        
               break

The relevant corresponding HTML looks like

<form id="download-form" action="https://drive.usercontent.google.com/download" method="get">
	<input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/>
	<input type="hidden" name="id" value="1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK">
	<input type="hidden" name="confirm" value="t">
	<input type="hidden" name="uuid" value="ce320bab-e9f0-4d0d-a9d8-8cede0161915">
</form>

The branch above just extracts https://drive.usercontent.google.com/download from it and this later fails other checks as this doesn't provide a file.

What we should do here is to extract the other parts of the form as well and append them as query. So for this example https://drive.usercontent.google.com/download?id=1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK&confirm=t&uuid=ce320bab-e9f0-4d0d-a9d8-8cede0161915. As you can see this link also includes confirm=t that multiple people in this thread have already mentioned. I'll send a PR to fix this.

pmeier · 2024-01-29T08:53:30Z

@wkentaro see #308. Could you please have a look?

wkentaro · 2024-02-01T14:05:39Z

Hi everyone.

I've just merged #308 by @pmeier, which adds extra pattern to parse the url to download.

Since I've never seen that pattern, I suspect it is a regional issue (I'm accessing from Japan, @pmeier is accessing from Germany). But at least It should help people who have the same issue as him.

vanloc1808 · 2024-02-23T13:51:21Z

@wkentaro Hi, I observed that only downgrading to 4.6.0 helps, new version does not help :(

wkentaro · 2024-02-23T13:57:43Z

Can you please give me an example url?

vanloc1808 · 2024-02-24T03:54:41Z

https://drive.google.com/file/d/1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb/view?usp=drive_link
Sure, here is it

vanloc1808 · 2024-02-24T03:56:04Z

I tried both

gdown --fuzzy 'https://drive.google.com/file/d/1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb/view?usp=drive_link'

and

gdown https://drive.google.com/uc?id=1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb

But it only works for me with gdown==4.6.0

wkentaro · 2024-02-25T02:48:58Z

@vanloc1808 It seems working for me with the latest.

Maybe the cookies file is not helping? If so remove it by rm ~/.cache/gdown/cookies.txt

vanloc1808 · 2024-02-25T04:15:15Z

@wkentaro it works for me, thank you!

Junyi42 · 2024-03-02T07:20:39Z

Older version gdown also helps.
pip install gdown==4.6.0
I finally solved this issue by using 4.6.0 gdown to download files.

Restarting the terminal after the downgrading works for me!

AndrewLt · 2024-03-07T16:14:05Z

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

julienboussard · 2024-03-08T15:16:56Z

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

Yeah this happened to me as well several times (with different versions). I'm not sure why, but changing the version (sometimes to a newer one, sometimes to an older one) seem to work, but always with a different version. Maybe it clears something and allows gdown to download again? I usually try versions 4.6.0, 5.1.0, or the latest (pip install --upgrade gdown). I also remove cookies.txt, and I end up (randomly) with a version that works. Hope this helps

ArianAmani · 2024-03-12T14:39:23Z

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

Weirdly this worked for me too. And I was using the default gdown (pip install gdown) on colab using gdown --id ID without a problem until just a few days ago. Thanks!

ain-soph · 2024-06-14T05:01:35Z

@wkentaro I'm still suffering this issue today. (gdown version 5.2.0)

I'm trying to download my own uploaded dataset (10G)
https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

And it raises the error

gdown https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_


Failed to retrieve file url:

        Too many users have viewed or downloaded this file recently. Please
        try accessing the file again later. If the file you are trying to
        access is particularly large or is shared with many people, it may
        take up to 24 hours to be able to view or download the file. If you
        still can't access a file after 24 hours, contact your domain
        administrator.

You may still be able to access the file from the browser:

        https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

but Gdown can't. Please check connections and **permissions.**

@JiaHeng-DLUT 's workaround works for me (https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work)

wkentaro · 2024-06-14T11:10:17Z

@ain-soph It works for me now, so looks like too many requests?

@JiaHeng-DLUT 's workaround works for me (https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work)

We can't do this because the scope of gdown is without logging in.

this is another workaround: https://github.com/wkentaro/gdown?tab=readme-ov-file#i-set-the-permission-anyone-with-link-but-i-still-cant-download

cleanerspam · 2024-08-08T06:37:29Z

#!/bin/bash

# Prompt for access token
read -p "Enter your Google Drive access token: " ACCESS_TOKEN

# Define the input file containing URLs
INPUT_FILE="file_urls.txt"

# Define the output directory
OUTPUT_DIR="/mnt/harddrive/gdrive"

# Check if the input file exists
if [[ ! -f "$INPUT_FILE" ]]; then
    echo "Input file '$INPUT_FILE' not found!"
    exit 1
fi

# Check if the output directory exists
if [[ ! -d "$OUTPUT_DIR" ]]; then
    echo "Output directory '$OUTPUT_DIR' does not exist!"
    exit 1
fi

# Loop through each line in the input file
while IFS= read -r URL; do
    # Extract the file ID from the URL
    FILE_ID=$(echo "$URL" | grep -oP 'd/\K[^/]+')

    # Check if FILE_ID was extracted successfully
    if [[ -z "$FILE_ID" ]]; then
        echo "Failed to extract file ID from URL: $URL"
        continue
    fi

    # Get the file metadata to retrieve the original name
    FILE_METADATA=$(curl -s -H "Authorization: Bearer $ACCESS_TOKEN" "https://www.googleapis.com/drive/v3/files/$FILE_ID?fields=name")
    ORIGINAL_NAME=$(echo "$FILE_METADATA" | jq -r '.name')

    # Check if the original name was retrieved successfully
    if [[ -z "$ORIGINAL_NAME" ]]; then
        echo "Failed to retrieve original name for file ID: $FILE_ID"
        continue
    fi

    # Use curl to download the file to the specified output directory
    OUTPUT_FILE="$OUTPUT_DIR/$ORIGINAL_NAME"
    echo "Downloading file with ID: $FILE_ID as '$ORIGINAL_NAME' to '$OUTPUT_FILE'"
    curl -H "Authorization: Bearer $ACCESS_TOKEN" -C - "https://www.googleapis.com/drive/v3/files/$FILE_ID?alt=media" -o "$OUTPUT_FILE"

    # Check if the download was successful
    if [[ $? -eq 0 ]]; then
        echo "Downloaded: $OUTPUT_FILE"
    else
        echo "Failed to download file with ID: $FILE_ID"
    fi
done < "$INPUT_FILE"

To anybody reading this

Install jq from your distro package manager
Change the output_location in script to something of your liking
Save the above code as download_drive_files.sh and
enter all links of files in a text file named file_urls.txt in the same directory as the script
Get AccessToken by following guide below

Going to OAuth 2.0 Playground https://developers.google.com/oauthplayground/
In the Select the Scope box, paste https://www.googleapis.com/auth/drive.readonly
Click Authorize APIs and then Exchange authorization code for tokens
Copy the Access token

This will save all the files that were mentioned in the file_urls.txt , just make there is only one url per line.

secluls · 2024-10-14T20:05:06Z

@cleanerspam does the script works with gdrive folders?

cleanerspam · 2024-10-15T04:05:54Z

@cleanerspam does the script works with gdrive folders?

Try and tell , as far as I remember it was for individual links stored in a .txt file

secluls · 2024-10-15T09:33:26Z

@cleanerspam I tried it out, but unfortunately it didn’t work. The script also didn’t work for some reason, as it only stores the files with no actual data inside. Still, a massive thank you for your efforts!

wkentaro self-assigned this Mar 31, 2020

wkentaro added more-information-needed labels Mar 31, 2020

wkentaro mentioned this issue Apr 29, 2020

Add cookies saved in home cache directory #51

Merged

wkentaro mentioned this issue May 14, 2020

Permission denied after few donwload #53

Closed

chi0tzp mentioned this issue May 15, 2020

[R2V2 dataset download] Permission denied - Maybe you need to change permission over 'Anyone with the link'? danielgordon10/vince#1

Closed

wkentaro mentioned this issue Jun 6, 2020

Remove download_warning_ from cookies #55

Merged

maltelueken mentioned this issue Jan 17, 2024

Prepare release 1.0 mexca/mexca#111

Merged

wkentaro unpinned this issue Jan 20, 2024

pmeier mentioned this issue Jan 29, 2024

Add extra pattern to extract url from download-form returned by Google Drive for a large file #308

Merged

wkentaro closed this as completed in #308 Feb 1, 2024

wkentaro reopened this Feb 1, 2024

Jieran-S mentioned this issue Mar 18, 2024

Update dataset gdown version SpatialHackathon/SpaceHack2023#249

Merged

kkaytekin mentioned this issue Aug 14, 2024

Too many users have viewed or downloaded this file recently. [Google Drive Link] zhuhao-nju/facescape#126

Closed

Vujas-Eteph mentioned this issue Aug 23, 2024

Access denied with gdown Vujas-Eteph/DatasetHarvester#2

Closed

Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

Comments

alexklwong commented Mar 25, 2020

mrlzla commented Mar 25, 2020

zenithfang commented Mar 26, 2020

wkentaro commented Mar 31, 2020 • edited Loading

Rmao99 commented Mar 31, 2020

wkentaro commented Mar 31, 2020 • edited Loading

wkentaro commented Apr 29, 2020

Jackie-LJQ commented May 5, 2020 • edited Loading

whulixiya commented Jun 1, 2020

carlmalamud commented Jun 3, 2020 • edited Loading

wkentaro commented Jun 3, 2020

whulixiya commented Jun 4, 2020 via email

carlmalamud commented Jun 4, 2020

carlmalamud commented Jun 6, 2020

wkentaro commented Jun 6, 2020

wkentaro commented Jun 6, 2020

wkentaro commented Jun 6, 2020

wkentaro commented Jun 6, 2020

grudloff commented Jun 10, 2020

carlmalamud commented Jun 16, 2020

carlmalamud commented Jun 16, 2020

wkentaro commented Jun 17, 2020

kshitijagrwl commented Jul 6, 2020 • edited Loading

wkentaro commented Jul 6, 2020

aasyedd1 commented Jan 16, 2024

neeeal commented Jan 17, 2024

wkentaro commented Jan 20, 2024

pmeier commented Jan 29, 2024

pmeier commented Jan 29, 2024

wkentaro commented Feb 1, 2024

vanloc1808 commented Feb 23, 2024

wkentaro commented Feb 23, 2024

vanloc1808 commented Feb 24, 2024

vanloc1808 commented Feb 24, 2024 • edited Loading

wkentaro commented Feb 25, 2024

vanloc1808 commented Feb 25, 2024

Junyi42 commented Mar 2, 2024 • edited Loading

AndrewLt commented Mar 7, 2024 • edited Loading

julienboussard commented Mar 8, 2024 • edited Loading

ArianAmani commented Mar 12, 2024

ain-soph commented Jun 14, 2024

wkentaro commented Jun 14, 2024 • edited Loading

cleanerspam commented Aug 8, 2024

secluls commented Oct 14, 2024

cleanerspam commented Oct 15, 2024

secluls commented Oct 15, 2024

wkentaro commented Mar 31, 2020 •

edited

Loading

wkentaro commented Mar 31, 2020 •

edited

Loading

Jackie-LJQ commented May 5, 2020 •

edited

Loading

carlmalamud commented Jun 3, 2020 •

edited

Loading

kshitijagrwl commented Jul 6, 2020 •

edited

Loading

vanloc1808 commented Feb 24, 2024 •

edited

Loading

Junyi42 commented Mar 2, 2024 •

edited

Loading

AndrewLt commented Mar 7, 2024 •

edited

Loading

julienboussard commented Mar 8, 2024 •

edited

Loading

wkentaro commented Jun 14, 2024 •

edited

Loading