Batch Download

[Aznaveh]

I think most of people usually need a bunch of matrices. It would be great if there is a functionality to download in bulk.

[lahwaacz]

Or at least allow a recursive download with something like this:

wget --mirror --no-parent --force-directories --cut-dirs=1 --accept "*.gz" --delete-after "https://sparse.tamu.edu/MM/"

I'm getting a lot of 403 Forbidden responses after about 4 successful queries...

[DrTimothyAldenDavis]

Try the ssget java interface. It can download a batch of matrices, or all of them at once.

…

On Tue, Jan 14, 2020 at 8:45 AM Jakub Klinkovský ***@***.***> wrote: Or at least allow a recursive download with something like this: wget --mirror --no-parent --force-directories --cut-dirs=1 --accept "*.gz" --delete-after "https://sparse.tamu.edu/MM/" I'm getting a lot of 403 Forbidden responses after about 4 successful queries... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=AEYIIOLXAJPARD4JVQAZHFLQ5XFXXA5CNFSM4ECVM36KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI43JAA#issuecomment-574207104>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYIIONH7SM4KJIKK6WFNGDQ5XFXXANCNFSM4ECVM36A> .

[lahwaacz]

Relying on Java is not an option for me... 😞

[ScottKolo]

Or at least allow a recursive download with something like this:

wget --mirror --no-parent --force-directories --cut-dirs=1 --accept "*.gz" --delete-after "https://sparse.tamu.edu/MM/"

I'm getting a lot of 403 Forbidden responses after about 4 successful queries...

This should actually work. I don't have a good explanation for why you would be getting 403 responses for this. I just tried and saw the same behavior.

My hunch is that this has something do with our Apache configuration.

[lahwaacz]

If you could provide an FTP access like the MatrixMarket (ftp://math.nist.gov/pub/MatrixMarket2/) has, that would be great.

[neoblizz]

If it still helps, this worked for me in downloading ALL matrix market datasets:

wget --recursive --no-parent --force-directories -l inf -X RB,mat --accept "*.tar.gz" "https://suitesparse-collection-website.herokuapp.com/"

• --recursive recursively download
• --no-parent prevent wget from starting to fetch links in the parent of the website
• --l inf keep downloading for an infinite level
• -X RB,mat ignore subdirectories RB and mat, since I am only downloading matrix market MM, you can choose to download any of the others or remove this entirely to download all formats
• --accept accept the following extension only
• --force-directories create a hierarchy of directories, even if one would not have been created otherwise