Skip to content

getpapers

Jump to bottom Edit New page
Priya-Jk-15 edited this page May 24, 2020 · 40 revisions

getpapers

primary URL

https://github.com/ContentMine/getpapers

purpose

queries a repository with RESTful API and downloads content in bulk

documentation.

overview and installation https://github.com/ContentMine/getpapers/blob/master/README.md For example https://github.com/petermr/tigr2ess/blob/master/getpapers/OVERVIEW.md

installation

Installation of getpapersinvolves steps for every operation system.

Instructions followed: https://github.com/ContentMine/getpapers/blob/master/README.md

See also: https://github.com/petermr/tigr2ess/blob/master/installation/INSTALLATION.md

example of use

Simple: https://github.com/ContentMine/getpapers/blob/master/README.md

For a full example: https://github.com/petermr/tigr2ess/tree/master/getpapers

comments

  • getpapers uses a headless browser (Phantom.js) which still works but is no longer maintained. It is customised for EPMC, IEEE, Crossref and ?arXiv. It needs a RESTful API.
  • the query syntax is different on different sites. Also escape characters (" or ')
  • default query format is EPMC

usefulness

help in downloading large files with full text content in bulk at a short time duration.

Installation problems

Users can face various problems during the installation process of getpapers. They may encounter errors in their process. Follow the instructions and in case of any installation problem, post an issue about the same in the issue section, or refer to an existing issue if it matches the problem.

Usage problems

For users facing any usage problems in getpapers they can create an issue regarding the same or may refer the existing ones.

Windows

More examples

Some EPMC queries

(please add some queries involving DATE, OR, AND, NOT)

tester experiences:

tester 1

Kareena Singh

operating system

Windows 10

INSTALLATION PROCESS

(1) Installation of nvm-windows

source of instructions

https://github.com/petermr/tigr2ess/blob/master/installation/windows/INSTALLATION.md

steps of installation

Go to the downloads page and download latest version of nvm-setup.zip.

Unzip the downloaded file and run the included installer.

installation

successfully installed and run

(2) Installation of node

source of instructions

https://github.com/petermr/tigr2ess/blob/master/installation/windows/INSTALLATION.md

steps of installation

Open your command prompt, and run the following commands one after the other.

nvm install 7 nvm use 7.10.1

installation of node

successful

installation problems

The following installation problem occured when I put node installation command in command line

Error: Access to the registry path is denied

  • reason insufficient privileges to install (requires "root" permission in windows)

  • solution

test of installation

successful

node --version
version 11.11.0

(3) Installation of getpapers

source of instructions

https://github.com/petermr/tigr2ess/blob/master/installation/windows/INSTALLATION.md

installation steps

Run the following command at command prompt:

npm install --global getpapers Now run the command getpapers at the command prompt, and you should see something as below:

installation problems

none reported

test of installation

You can run the test of installation by putting the command getpapers --version If you get the following, then installation is succesful. 0.4.17

Tester 2

Lakshmi Devi Priya

Operating System

Windows 10

Installation of node

source of instruction

Instructions from: https://github.com/ContentMine/blob/master/README.md

installation of node

Successful

test of installation

Successful

C:\Users>node -v v12.16.3

Installation of getpapers

source of instruction

Instructions from: https://github.com/ContentMine/blob/master/README.md

installtion of getpapers

Successful

installation problems

C:\Users>$ npm install --global getpapers

`$` is not recognized as an internal or external command,
operable program or batch file.

  • '$' is not a part of the command(it's UNIX prompt).

  • So just try as

npm install --global getpapers
  • getpapers is installed.

test of installation

Use

C:\Users>getpapers --help

  • The command option used for getpapers are viewed.

  • Installed getpapers.

Use of getpapers

Followed example from:

https://github.com/petermr/tigr2ess/tree/master/getpapers

To search query on a specified task

use the following syntax
getpapers -q <query> -n -k 100

-q, --query : search query(required)

-n, --noexecute : only reports how many queries match the query, but don't actually download anything

For eg: for the query of COVID-19

Use as

getpapers -q COVID-19 -n -k <int>

The results will be shown as below:

https://drive.google.com/file/d/1DP0_xcjC5GMQ2CflM7TQUoyIO3J3MyCW/view?usp=drivesdk

Output - Founds 46887 open acesss results. This much result cannot be downloaded, so the number of downloads should be limited.

-k, --limit : limits the number of hits and downloads

<int> refers to an integer. Hence, the number of files to be downloaded should be represented.

To download the files, use the following syntax
getpapers -q <query> -k <int> -o <path> -x -p

-o, --outdir : output directory(required - will be created if not found).

This command gives the path to the directory created in the system for the downloaded files.

-p, --pdf : downloads fulltext PDFs if available.

-x, --xml : downloads fulltext XMLs if available.

Thus, for the query COVID-19 the syntax

getpapers -q COVID-19 -k 100 -o covid -x -p

gives the result as follows.

https://drive.google.com/file/d/1H5k8ZooTFD1dHnMOK6-eTxckc95iJ6lG/view?usp=drivesdk

.xml files in the resultant folder are both machine-readable and human-readable.

Expected 100 .xml files were downloaded. But only 76 .pdf files were downloaded.

Add a custom sidebar
Clone this wiki locally