What Did It Look Like? (WDILL)

What Did It Look Like? is a service that uses the Memento framework to poll various public web archives, select an archived version from each calendar year, and create a visualization that shows the progression of the site through out the years. The visualization is shared to the specified Tumblr blog, Instagram page, and Twitter. In order to nominate a website, tweet #whatdiditlooklike URL.

A demo version of this service is deployed and posts to the following accounts:

• Tumblr blog: whatdiditlooklike.mementoweb.org
• Twitter: @_wdill
• Instagram: @_wdill

For more explanation of this service, visit:

• https://ws-dl.blogspot.com/2015/01/2015-02-05-what-did-it-look-like.html
• https://ws-dl.blogspot.com/2020/08/2020-08-20-automating-posting-videos-to.html

What Did It Look Like is heavily implemented in Python 3.x along with some JavaScript. The Python side of the service relies on the following libraries:

• Requests
• Appium Python Client
• Beautiful Soup
• PyTumblr
• Selenium
• Surt
• TinyTag
• TLDextract
• Tweepy
• Wikipedia
• Dateutil

The JavaScript dependencies include:

• Async
• Puppeteer

Usage

• Modify the config file; replace all place holders and insert the necessary credentials. WDILL requires credentials to the following services in order to function properly:
  • Twitter API
  • Tumblr API
  • Instagram login
  • BrowserStack Access Key
• The process will run as a cron job every 6 hours. To modify the timing, edit the crontab file following this documentation as needed.

Running as a Docker Container

Running WDILL in a Docker container can make the process of dependency management easier. This document assumes that you have Docker setup already, if not then follow the official guide.

In order to build the Docker image, clone the repository and change the working directory, make changes to the config file. Next, build and run the image as the following:

$ git clone https://github.com/dhruv282/wdill.git
$ cd wdill
<make necessary changes to config file>
<change cron job timing if needed>
$ docker image build -t wdill .
$ docker run --name wdill --shm-size=1G -it --rm wdill

This method will automatically install all required libraries and dependencies and also setup the cron job in the Docker container. The output generated by this service will be stored in /var/log/cron.log. Run the following command to view this log:

$ docker exec -it wdill cat /var/log/cron.log

Running Locally

Running locally requires the installing programs, libraries, and dependencies manually. This document assumes that you have the following programs installed:

• Python 3.x along with pip
• Node.js along with npm
• FFmpeg
• ImageMagick

Follow the steps below to run this service locally:

$ git clone https://github.com/dhruv282/wdill.git
$ cd wdill
<make necessary changes to config file>
<change cron job timing if needed>
$ ./setup_wdill.sh
$ cron && tail -f /var/log/cron.log

Citing Project

A tech report related to this project is available in arXiv.org (pdf). Please cite it as below:

Dhruv Patel, Alexander C. Nwala Michael L. Nelson, and Michele C. Weigle. What Did It Look Like: A service for creating website timelapses using the Memento framework. Technical report arXiv:2104.14041, April 2021.

@techreport{wdill-arxiv,
  title = {{What Did It Look Like}: A service for creating website timelapses using the {Memento} framework},
  author = {Dhruv Patel and Alexander C. Nwala and Michael L. Nelson and Michele C. Weigle},
  year = {2021},
  month = apr,
  number = {arXiv:2104.14041},
  url = {https://arxiv.org/abs/2104.14041}
}