Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include filenames in error logs #574

Closed
panhartstuff opened this issue Jan 12, 2020 · 2 comments
Closed

Include filenames in error logs #574

panhartstuff opened this issue Jan 12, 2020 · 2 comments

Comments

@panhartstuff
Copy link

Is there a way to include filenames in error logs such that it looks something like this

[downloader.http][warning][k3_spaceybear-772534025424474113-20160904_203719-p1.jpg] '404 Not Found' for 'https://pbs.twimg.com/media/CriXYFKUAAApwh-.jpg:orig'
[download][info][k3_spaceybear-772534025424474113-20160904_203719-p1.jpg] Trying fallback URL #1

The default currently looks like this:

[downloader.http][warning] '404 Not Found' for 'https://pbs.twimg.com/media/CriXYFKUAAApwh-.jpg:orig'
[download][info] Trying fallback URL #1

I tried a bunch of configuration, but I can't get it to work.

@mikf
Copy link
Owner

mikf commented Jan 12, 2020

For extractor logging messages, you can access the current job and extractor objects and use their attributes including filenames, but currently not in downloader messages (easy enough to fix, though).

The potential problem here is that there might be no filename available at that point in time, either for videos downloaded through youtube-dl or when the filename extension couldn't be determined yet. Hence, there is no general solution to this issue.

You should've also gotten a logging message after all the downloader failures which does include the filename

[download][error] Failed to download k3_spaceybear-772534025424474113-20160904_203719-p1.jpg

@panhartstuff
Copy link
Author

I see, makes sense.

It's just that sometimes there are cases where the downloader times out and it resorts to using a lower quality image (in the case of Twitter) when it doesn't need to.
In that situation, the download does "succeeds", so the filename doesn't get written into the logfile.
Since I use the logfile to check which file I should redownload, it makes it difficult to see which ID the error happens on.

But I guess I could just increase the timeout limit for now.

mikf added a commit that referenced this issue May 18, 2020
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.

__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.

(#574, #575)
@mikf mikf closed this as completed Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants