Restore arbitrary file

[sergey-tihon]

It would be nice to have an ability to restore arbitrary file from internet (without ability to update reference).

Something like this
https://github.com/sergey-tihon/Stanford.NLP.NET/blob/master/src/Core.fsx#L53

Would be enough to download and put file somewhere.
It is useful when you need to restore large binaries that you do not want to commit in repository
https://github.com/sergey-tihon/Stanford.NLP.NET/blob/master/src/Stanford.NLP.CoreNLP.fsx#L17

[forki]

could you try ti come up with a syntax for dependencies file?

[sergey-tihon]

What do you think about following syntax

download http://server.com/file1.txt /path/to/folder/
downlaod http://server.com/file2.txt /path/to/folder/new_file_name.txt
download "http://server.com/space in url.txt" "/path/to/folder/space in path.txt"

[agross]

+1

Alex

Alexander Groß
Tiny phone, tiny mail

On Tue, Oct 28, 2014 at 8:10 PM, Sergey Tihon [email protected]
wrote:

What do you think about following syntax

download http://server.com/file1.txt /path/to/folder/
downlaod http://server.com/file2.txt /path/to/folder/new_file_name.txt
download "http://server.com/space in url.txt" "/path/to/folder/space in path.txt"

---

Reply to this email directly or view it on GitHub:
#298 (comment)

[sergey-tihon]

Also, I think that paket should not try to update such files. (at least it should be under control)
I am going to reference large zip files with models (~300Mb). Hash check is very expensive in this case.

[forki]

Do you want to try to implement it?
On Oct 28, 2014 9:19 PM, "Sergey Tihon" [email protected] wrote:

Also, I think that paket should not try to update such files. (at least
it should be under control)
I am going to reference large zip files with models (~300Mb). Hash check
is very expensive in this case.

—
Reply to this email directly or view it on GitHub
#298 (comment).

[sergey-tihon]

I have looked into the code and I am not sure about design now...
GitHub references does not allow to specify target location, but we are going to allow this for download. I think that it should be consistent across different reference types. There are two options:

• Download everything into paket-files (no configuration option)
• Add target path to githubreferences syntax.
  Your thoughts?

[forki]

Download to paket-files
On Oct 31, 2014 11:05 AM, "Sergey Tihon" [email protected] wrote:

I have looked into the code and I am not sure about design now...
GitHub references does not allow to specify target location, but we are
going to allow this for download. I think that it should be consistent
across different reference types. There are two options:

• Download everything into paket-files (no configuration option)
• Add target path to githubreferences syntax. Your thoughts?

—
Reply to this email directly or view it on GitHub
#298 (comment).

[forki]

what's the status? We can already download from arbitray url. So only unzip is missing right?

[sergey-tihon]

@forki Sorry, but status it not started yet =(

[sergey-tihon]

Paket needs 4min 40sec to restore 1 file ~20Mb
It is very strange ... How many times Paket download files?

[forki]

It needed 20s on my machine.

[sergey-tihon]

And does not work on my machine using latest released version ... Hmmm...

[forki]

Can you send me your repo?
On Dec 5, 2014 12:32 PM, "Sergey Tihon" [email protected] wrote:

And does not work on my machine using latest released version ... Hmmm...
[image: image]
https://cloud.githubusercontent.com/assets/1197905/5315207/91acd1f2-7c8b-11e4-8af6-bcd420a8016d.png

—
Reply to this email directly or view it on GitHub
#298 (comment).

[sergey-tihon]

Could you try to restore following dependencies and post your restore time here?

http http://www.frijters.net/ikvmbin-8.0.5449.0.zip
http http://nlp.stanford.edu/software/stanford-corenlp-full-2014-10-31.zip
http http://nlp.stanford.edu/software/stanford-ner-2014-10-26.zip
http http://nlp.stanford.edu/software/stanford-parser-full-2014-10-31.zip
http http://nlp.stanford.edu/software/stanford-postagger-full-2014-10-26.zip
http http://nlp.stanford.edu/software/stanford-segmenter-2014-10-26.zip

There are real dependencies for Stanford.NLP.NET. It should be ~1Gb of data.

It looks very unfriendly on my machine. Paket works more than hour without any progress information on the console - I have no idea what is going on inside. Some loading messages or status would be nice to be sure that everything is OK.

[forki]

and a little bit later:

As you can see the following is happening:

first the resolver downloads all the zip files sequentially and then the restore downloads again but in parallel. That's obviously sub-optimal for such big files.

[sergey-tihon]

@forki Thanks for help with implementation.