[YouTube] Failed download of a playlist page is not fatal and causes a false success.

[coletdjnz]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2020.12.05
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

youtube-dl --flat-playlist --skip-download UUm9K6rby98W8JigLoZOh6FQ --proxy http://localhost:8889 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--flat-playlist', '--skip-download', 'UUm9K6rby98W8JigLoZOh6FQ', '--proxy', 'http://localhost:8889', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.12.05
[debug] Python version 3.8.6 (CPython) - Linux-5.9.11-3-MANJARO-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {'http': 'http://localhost:8889', 'https': 'http://localhost:8889'}
[youtube:tab] UUm9K6rby98W8JigLoZOh6FQ: Downloading webpage
[download] Downloading playlist: Uploads from LockPickingLawyer
[youtube:tab] Downloading page 1
[youtube:tab] Downloading page 2
WARNING: Unable to download JSON metadata: <urlopen error Remote end closed connection without response>
[youtube:tab] playlist Uploads from LockPickingLawyer: Downloading 200 videos
[download] Downloading video 1 of 200
[download] Downloading video 2 of 200
[download] Downloading video 3 of 200
[download] Downloading video 4 of 200
[download] Downloading video 5 of 200
[download] Downloading video 6 of 200
[download] Downloading video 7 of 200
[download] Downloading video 8 of 200
[download] Downloading video 9 of 200
[download] Downloading video 10 of 200
[download] Downloading video 11 of 200
[download] Downloading video 12 of 200
[download] Downloading video 13 of 200
[download] Downloading video 14 of 200
[download] Downloading video 15 of 200
[download] Downloading video 16 of 200
[download] Downloading video 17 of 200
[download] Downloading video 18 of 200
[download] Downloading video 19 of 200
[download] Downloading video 20 of 200
[download] Downloading video 21 of 200
[download] Downloading video 22 of 200
[download] Downloading video 23 of 200
[download] Downloading video 24 of 200
[download] Downloading video 25 of 200
[download] Downloading video 26 of 200
[download] Downloading video 27 of 200
[download] Downloading video 28 of 200
[download] Downloading video 29 of 200
[download] Downloading video 30 of 200
[download] Downloading video 31 of 200
[download] Downloading video 32 of 200
[download] Downloading video 33 of 200
[download] Downloading video 34 of 200
[download] Downloading video 35 of 200
[download] Downloading video 36 of 200
[download] Downloading video 37 of 200
[download] Downloading video 38 of 200
[download] Downloading video 39 of 200
[download] Downloading video 40 of 200
[download] Downloading video 41 of 200
[download] Downloading video 42 of 200
[download] Downloading video 43 of 200
[download] Downloading video 44 of 200
[download] Downloading video 45 of 200
[download] Downloading video 46 of 200
[download] Downloading video 47 of 200
[download] Downloading video 48 of 200
[download] Downloading video 49 of 200
[download] Downloading video 50 of 200
[download] Downloading video 51 of 200
[download] Downloading video 52 of 200
[download] Downloading video 53 of 200
[download] Downloading video 54 of 200
[download] Downloading video 55 of 200
[download] Downloading video 56 of 200
[download] Downloading video 57 of 200
[download] Downloading video 58 of 200
[download] Downloading video 59 of 200
[download] Downloading video 60 of 200
[download] Downloading video 61 of 200
[download] Downloading video 62 of 200
[download] Downloading video 63 of 200
[download] Downloading video 64 of 200
[download] Downloading video 65 of 200
[download] Downloading video 66 of 200
[download] Downloading video 67 of 200
[download] Downloading video 68 of 200
[download] Downloading video 69 of 200
[download] Downloading video 70 of 200
[download] Downloading video 71 of 200
[download] Downloading video 72 of 200
[download] Downloading video 73 of 200
[download] Downloading video 74 of 200
[download] Downloading video 75 of 200
[download] Downloading video 76 of 200
[download] Downloading video 77 of 200
[download] Downloading video 78 of 200
[download] Downloading video 79 of 200
[download] Downloading video 80 of 200
[download] Downloading video 81 of 200
[download] Downloading video 82 of 200
[download] Downloading video 83 of 200
[download] Downloading video 84 of 200
[download] Downloading video 85 of 200
[download] Downloading video 86 of 200
[download] Downloading video 87 of 200
[download] Downloading video 88 of 200
[download] Downloading video 89 of 200
[download] Downloading video 90 of 200
[download] Downloading video 91 of 200
[download] Downloading video 92 of 200
[download] Downloading video 93 of 200
[download] Downloading video 94 of 200
[download] Downloading video 95 of 200
[download] Downloading video 96 of 200
[download] Downloading video 97 of 200
[download] Downloading video 98 of 200
[download] Downloading video 99 of 200
[download] Downloading video 100 of 200
[download] Downloading video 101 of 200
[download] Downloading video 102 of 200
[download] Downloading video 103 of 200
[download] Downloading video 104 of 200
[download] Downloading video 105 of 200
[download] Downloading video 106 of 200
[download] Downloading video 107 of 200
[download] Downloading video 108 of 200
[download] Downloading video 109 of 200
[download] Downloading video 110 of 200
[download] Downloading video 111 of 200
[download] Downloading video 112 of 200
[download] Downloading video 113 of 200
[download] Downloading video 114 of 200
[download] Downloading video 115 of 200
[download] Downloading video 116 of 200
[download] Downloading video 117 of 200
[download] Downloading video 118 of 200
[download] Downloading video 119 of 200
[download] Downloading video 120 of 200
[download] Downloading video 121 of 200
[download] Downloading video 122 of 200
[download] Downloading video 123 of 200
[download] Downloading video 124 of 200
[download] Downloading video 125 of 200
[download] Downloading video 126 of 200
[download] Downloading video 127 of 200
[download] Downloading video 128 of 200
[download] Downloading video 129 of 200
[download] Downloading video 130 of 200
[download] Downloading video 131 of 200
[download] Downloading video 132 of 200
[download] Downloading video 133 of 200
[download] Downloading video 134 of 200
[download] Downloading video 135 of 200
[download] Downloading video 136 of 200
[download] Downloading video 137 of 200
[download] Downloading video 138 of 200
[download] Downloading video 139 of 200
[download] Downloading video 140 of 200
[download] Downloading video 141 of 200
[download] Downloading video 142 of 200
[download] Downloading video 143 of 200
[download] Downloading video 144 of 200
[download] Downloading video 145 of 200
[download] Downloading video 146 of 200
[download] Downloading video 147 of 200
[download] Downloading video 148 of 200
[download] Downloading video 149 of 200
[download] Downloading video 150 of 200
[download] Downloading video 151 of 200
[download] Downloading video 152 of 200
[download] Downloading video 153 of 200
[download] Downloading video 154 of 200
[download] Downloading video 155 of 200
[download] Downloading video 156 of 200
[download] Downloading video 157 of 200
[download] Downloading video 158 of 200
[download] Downloading video 159 of 200
[download] Downloading video 160 of 200
[download] Downloading video 161 of 200
[download] Downloading video 162 of 200
[download] Downloading video 163 of 200
[download] Downloading video 164 of 200
[download] Downloading video 165 of 200
[download] Downloading video 166 of 200
[download] Downloading video 167 of 200
[download] Downloading video 168 of 200
[download] Downloading video 169 of 200
[download] Downloading video 170 of 200
[download] Downloading video 171 of 200
[download] Downloading video 172 of 200
[download] Downloading video 173 of 200
[download] Downloading video 174 of 200
[download] Downloading video 175 of 200
[download] Downloading video 176 of 200
[download] Downloading video 177 of 200
[download] Downloading video 178 of 200
[download] Downloading video 179 of 200
[download] Downloading video 180 of 200
[download] Downloading video 181 of 200
[download] Downloading video 182 of 200
[download] Downloading video 183 of 200
[download] Downloading video 184 of 200
[download] Downloading video 185 of 200
[download] Downloading video 186 of 200
[download] Downloading video 187 of 200
[download] Downloading video 188 of 200
[download] Downloading video 189 of 200
[download] Downloading video 190 of 200
[download] Downloading video 191 of 200
[download] Downloading video 192 of 200
[download] Downloading video 193 of 200
[download] Downloading video 194 of 200
[download] Downloading video 195 of 200
[download] Downloading video 196 of 200
[download] Downloading video 197 of 200
[download] Downloading video 198 of 200
[download] Downloading video 199 of 200
[download] Downloading video 200 of 200
[download] Finished downloading playlist: Uploads from LockPickingLawyer

Description

If youtube-dl fails to download any of the playlist pages, the error reported is not fatal. I believe this is an issue as it produces a false success, allowing an incorrect playlist size to be reported and downloaded, as well as making it difficult to handle the error externally in the case of using the youtube_dl module.

See the example above in the log - the playlist above has 1209 videos as of writing. Using a proxy to forcibly trigger a connection error to show this, it fails to download page 3, hence only gathered 200 videos. The issue is it treats the playlist extraction as successful, so it continues to try download these 200 videos (and report there is 200 in the playlist which is incorrect), rather than producing an error and stopping.

I believe this has something to do with this part:

youtube-dl/youtube_dl/extractor/youtube.py

Lines 2984 to 2987 in 082da36

	browse = self._download_json(
	'https://www.youtube.com/browse_ajax', None,
	'Downloading page %d' % page_num,
	headers=headers, query=continuation, fatal=False)

[coletdjnz]

I think this is different. In the code snippet above I mentioned, if I set fatal=True it behaves as expected (prevents false success by failing). I'm just unsure if there was a reason why it was set to False.