Add yt-dlp based archiving for TwitterArchiver #138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msramalho
reviewed
Apr 15, 2024
| else: | ||
| variant = var.get("src") if not variant else variant | ||
| variant = var if not variant else variant |
There was a problem hiding this comment.
these key changes in variant_choosing may break the other scrapers, but as they are well... broken ... it's just fine for now.
msramalho
approved these changes
Apr 15, 2024
There was a problem hiding this comment.
This is a really good and impactful contribution, thank you for taking the time to properly implementing this new alternative. As we're building on 3 and ytdlp is the most likely to work now, I would accept refactoring the priorities of the code but that's an extra and makes little impact in the running time. will merge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a yt-dlp based twitter archiving function to
TwitterArchiveras a fallback to the existing two archiving strategies. It uses the_extract_statusfunction of yt-dlp'sTwitterIEextractor to extract tweet metadata, and processes it in a similar way to the existing archiving implementation.Upon local testing, the existing snscrape(which seems to be unmaintained) and twitter-hack solution does not work reliably for tweets, but the yt-dlp based solution does. Happy to know if it can be replicated!
Also happy to add a configuration option to
TwitterArchiverfor specifying the preference of tweet archiving methods(snscrape/twitter-hack/yt-dlp)