Add download Wikidata dump command to CLI

[andrewtavis]

Terms

• I have searched open and closed feature requests
• I agree to follow Scribe-Data's Code of Conduct

Description

Scribe-Data will be expanding its functionality to work from Wikidata dumps. The first step in this is to add the ability for the CLI to download Wikidata Lexeme dumps. The following command should be added in this issue:

# Latest dump:
scribe-data download --wikidata-dump
scribe-data d -wd

# Specific dump:
scribe-data download --wikidata-dump YYYYMMDD
scribe-data d -wd YYYYMMDD

# Specific output directory:
scribe-data download --wikidata-dump --output-dir DIRECTORY_PATH
scribe-data d -wd -od DIRECTORY_PATH

The above will download the dumps from dumps.wikimedia.org/wikidatawiki/entities/. In the fist set of queries the latest .json.bz2 file will be downloaded, and in the second the URL for the given YYYYMMDD stamp will be checked and a .json.bz2 dump will be downloaded to the PWD. The third would add in an output directory path as is done on the get command, but let's not change the file name. We'll just allow the user to put it in a directory 😊

The functionality should be added in a file src/scribe_data/cli/download.py, with the option being added into src/scribe_data/cli/main.py :)

Contribution

Being worked on by @axif0 as a part of Outreachy! 📶🚀

[axif0]

This should be like-

sorry my network is too slow to fully download for now.

[axif0]

Also, should we add functionalities like if user give wrong date it'll ask user for available closest vailable old dumps, here we also check in a date-lexeme-folder .json.bz2 file are in or not. like 20241030 has and 20241122 not.

Available Closest old dumps only shows who has .json.bz2

[axif0]

Apologize early, if those features are not planed.. 😞

[andrewtavis]

No need to apologize, @axif0! You're a part of planning the features :) :)

This seems good to me! We'll just take the next most recent one? If there's no most recent one we'll take the last most recent one?

[axif0]

This seems good to me! We'll just take the next most recent one? If there's no most recent one we'll take the last most recent one?

Yes..

[andrewtavis]

Closed by #528 🚀 Thanks for the great work on making a sustainable and performant download functionality, @axif0! 😊