A collection of scripts and code snippets for data harvesting after generating the zip starter.
We welcome tools written in any language! Especially if they cover use cases we haven't described. To add a new tool to this repository please review our Contributing Guidelines
- Familiarize yourself with the harvesting instructions in the DataRescue workflow repo. Within the Archivers app, click
Download Zip Starterfrom the detail page related to a URL you have checked out - Unzip the zip stater file
- Choose a tool that seems likely to be helpful in capturing this particular resource, and copy the contents of its directory in this repo to the
toolsdirectory, e.g. with:cp -r harvesting-tools/TOOLNAME/* RESOURCEUUID/tools/ - Adjust the base URL for the dataset, along with any other relevant variables, and tweak the content of the tool as necessary
- After the dataset has been harvested, proceed with the further steps in the harvesting instructions
Each tool in this repo has a fairly specific use case. Your choice will depend on the shape and size of the data you're dealing with. Some datasetswill require more creativity/more elaborate tools. If you write a new tool, please add it to the repo.
wget-loop for largely static resources
If you encounter a page that links to lots of data (for example a "downloads" page), this approach may well work. It's important to only use this approach when you encounter data, for example pdf's, .zip archives, .csv datasets, etc.
The tricky part of this approach is generating a list of urls to download from the page.
- If you're skilled with using scripts in combination with html-parsers (for example python's wonderful beautiful-soup package), go for it.
- if the URL's you're trying to access are dynamically generated by JavaScript in the browser environment, we've also included the jquery-url-extraction guide], which will guide you through the process of exracting URL's directly from the browser console.
download_ftp_tree.py for FTP datasets
Government datasets are often stored on FTP; this script will capture FTP directories and subdirectories.
PLEASE NOTE that the Internet Archive has captured over 100Tb of government FTP resoures since December 2016. Be sure to check the URL using check-ia/url-check, the Wayback Machine Extension, or your own tool that uses the Wayback Machine's API (example 1, example 2 w/ wildcard. If the FTP directory you're looking at has not been saved to the Internet Archive, be sure that it has also been nominated as a web crawl seed.
Whether it has been saved or not, you may decide to download it for chain-of-custody preservation reasons. If so, this script should do what you need.
Ruby/Watir for full browser automation
The last resort of harvesting should be to drive it with a full web browser. It is slower than other approaches such as wget, curl, or a headless browser. Additionally, this implementation is prone to issues where the resulting page is saved before it's done loading. There is a ruby example in tools/example-hacks/watir.rb.
For search results from large document sets, you may need to do more sophisticated "scraping" and "crawling" -- check out tools built at previous events such as the EIS WARC archiver or the EPA Search Utils for ideas on how to proceed.
If you encounter an API, chances are you'll have to build some sort of custom solution, like epa-envirofacts-scraper or investigate a social angle. For example: asking someone with greater access for a database dump. Be sure to include your code in the tools directory of your zipfile, and if there is any likelihood of general application, please add to this repo.
the utils directory is for scripts that have been useful in the past but may not have very general application. But you still might find something youu like!