403 Forbidden errors from downloading videos off twitter profile

[pxssy]

I'm not exactly sure what causes it but i'm quite suspicious of the volume i'm downloading at, so it might be because of limits imposed by twitter. This doesn't appear for the first several profiles i download, but gradually begins to appear. This never appears for images so i believe its only isolated to videos.

This is rather frustrating as i only get the videos from the first portion of the profiles, then the subsequent are all undownloaded. The error log i get are formatted like this

[twitter][warning] Unable to fetch video data for 1261633266638974976 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1261633266638974976 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1261626981864464386 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1247907279048822789 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1239114797989281792 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1238947146071597057 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1208730230807912449 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1208710041374584832 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1208074933877657601 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1197543392592293889 ('403 Forbidden') [twitter][warning] Unable to fetch video data for 1196798282439127040 ('403 Forbidden')

Hardly tells me which profile isn't downloading properly, makes it very difficult to go back and follow up on those errors.

Furthermore, i've been using the archive.sqlite3 functionality. This isn't the first time i'm running gallery-dl to save these profiles, so i believe a majority of the photos are already pulled, its only the videos that have issues. Shouldn't then the archive database then prevent gallery-dl from querying twitter for already saved posts, it would then only query for the videos that are errored and not saved like the ones above? I might still get 403s from limits, but at least the list of 403ed would go down everytime i run it.

what gives?

[outlaw240]

Same 403 errors here, I have to run through my list at least twice to get most of the content from a profile. I've noticed that it's more likely to happen on longer videos, like 60 sec + ones.

[mikf]

Fixed during the Twitter rewrite (#806)