Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local store of archived urls with check for content change locally to re-archive or return archived link #188

Open
ghost opened this issue Sep 10, 2023 · 0 comments

Comments

@ghost
Copy link

ghost commented Sep 10, 2023

Waybackpy has cdx which can check for archived content but there's the prospect of local checks.

The demo at asci cinema archives everytime you run it - that's unecessary.

During dev stage I ran into the throttling after archiving ~13 links every test. I considered a store in waybackpy with checks like hash or lash modified, I settled for last modified time. It's saved me the throttling notice and would conserve archive.org resources.

I'd suggest you add this as an optional feature, the size of the store can be configurable. Compression and other intelligent design decisions can assure users would not mind the local store.

Considerations of hash or last modified can be done when archiving local content which some use cases may be. Other methodologies of checking change in content can be thought or extended.

Consider?.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

0 participants