Docs fixes and improvements #102

sappelhoff · 2020-06-22T13:28:33Z

related to #97.

I am collecting several smaller fixes while going through the commands @bpoldrack recommended.

Questions that come up regarding `--mode export`

These questions may be answered and then used to improve the tutorial.

what would I have to do to have my data exported in a folder with the name of the dataset (as in datalad create DATASET_NAME) ... as opposed to having all files being loaded directly into the OSF? --> currently not possible, but may be a feature in the future (see Docs fixes and improvements #102 (comment))
the .datalad directory seems to be exported as well, ... how can I suppress this? (I may want to suppress writing the .datalad directory if using BIDS, and not wanting to add a .bidsignore with a single line .datalad) --> probably just have to live with it ... CAN be solved (but only in hacky ways) Docs fixes and improvements #102 (comment)
Given the hypothetical case that I already HAVE an OSF repository ... how do I add an OSF remote to a local datalad dataset to then just do git annex export HEAD --to NAME_OF_REMOTE ... without the OSF repo creation step? --> it's possible, but not yet convenient (future feature) Docs fixes and improvements #102 (comment)
when I update my data locally and then repeat git annex export HEAD --to NAME_OF_REMOTE ... will it overwrite my data, creating new OSF versions? --> unknown as of now: Docs fixes and improvements #102 (comment)

Questions that come up regarding `--mode annex`

when I pushed my datalad dataset to an OSF project via --mode annex, how can I clone or install my data from that OSF project (say, on a different machine, or after deleting locally) --> will be enabled with Rough draft of a Git remote helper to store Git repositories in OSF projects #100 (probably, see: Docs fixes and improvements #102 (comment))

bpoldrack · 2020-06-22T14:07:14Z

what would I have to do to have my data exported in a folder with the name of the dataset (as in datalad create DATASET_NAME) ... as opposed to having all files being loaded directly into the OSF?

For that we need to (re-)introduce a path option for the special remote. We discussed this during the brainhack and ended up not having it. Ultimately the argument was, that you can have a subproject (component) instead of a folder within your project and connect the special remote to that component instead. I do think that's a bit cleaner that way, but I don't see why we shouldn't allow for more flexibility. There's no "technical" reason. Ping @mih.

the .datalad directory seems to be exported as well, ... how can I suppress this? (I may want to suppress writing the .datalad directory if using BIDS, and not wanting to add a .bidsignore with a single line .datalad)

I don't think that's currently possible (except you create a dedicated commit w/ .datalad removed and export it). git annex export only allows to specify a subdir to restrict the export to (like: git annex export master:subdir --to myremote).

sappelhoff · 2020-06-22T15:11:22Z

Thanks @bpoldrack - I updated my OP (and added new questions).

Also stumbled over an issue while trying out using OSF credentials instead of an OSF token:

$ datalad create-sibling-osf sandbox --mode annex
[ERROR  ] 401 Client Error: Unauthorized for url: https://api.osf.io/v2//nodes/ [models.py:raise_for_status:941] (HTTPError)

I defined OSF_USERNAME and OSF_PASSWORD through:

export OSF_USERNAME="[email protected]"
and similarly for my password

the password and username are both correct and I can log in with them. Yet I still get an error from datalad-osf as pasted above. Any clues why?

bpoldrack · 2020-06-23T10:15:32Z

Given the hypothetical case that I already HAVE an OSF repository ... how do I add an OSF remote to a local datalad dataset to then just do git annex export HEAD --to NAME_OF_REMOTE ... without the OSF repo creation step?

For now:
git annex initremote NAME_OF_REMOTE type=external externaltype=osf encryption=none autoenable=true project=EXISTING_PROJECT exporttree=yes

If you wnat an annex-type remote instead, just leave out the exporttreeoption
EXISTING_PROJECT takes the URL to the project. However, the only important thing is, that it ends with the OSF ID, therefore you can also just pass the ID of your project (or component)

Eventually datalad create-sibling-osf should be enhanced to be able to do that more conveniently.

bpoldrack · 2020-06-23T10:17:13Z

when I update my data locally and then repeat git annex export HEAD --to NAME_OF_REMOTE ... will it overwrite my data, creating new OSF versions?

To be honest - I'm not sure, whether this generates new versions from the point of view of OSF, But I'd suspect it does. We need to make sure that's the case indeed.

bpoldrack · 2020-06-23T10:22:30Z

when I pushed my datalad dataset to an OSF project via --mode annex, how can I clone or install my data from that OSF project (say, on a different machine, or after deleting locally)

Not yet. PR #100 will allow for that.
As of now, it's just an annex store. However, thanks to the autoenabling (see option to initremote in above post), any clone of the repo you pushed there, will know about this data source and be able to get data from it. So, if you publish/update your dataset on github (or anywhere else), whoever clones from there and has datalad-osf installed will have access (assuming public access of your project or proper credentials, of course)

bpoldrack · 2020-06-23T10:24:48Z

the password and username are both correct and I can log in with them. Yet I still get an error from datalad-osf as pasted above. Any clues why?

There was some trouble on some machines in that regard (including mine). Current master should work better. You could install/update via pip install git+http://github.com/datalad/datalad-osf and report whether that fixes the issue in your case.

sappelhoff · 2020-06-23T12:16:25Z

Thanks a lot for all of these answers @bpoldrack. I'll finish up this PR later and mark it "ready for review".

sappelhoff · 2020-06-23T12:18:04Z

You could install/update via pip install git+http://github.com/datalad/datalad-osf and report whether that fixes the issue in your case

I am on the dev version (most recent commit on master), and I still get this issue.

sappelhoff · 2020-06-24T13:44:43Z

hey @all-contributors please add @sappelhoff for doc userTesting

allcontributors · 2020-06-24T13:44:54Z

@sappelhoff

I've put up a pull request to add @sappelhoff! 🎉

sappelhoff · 2020-06-24T13:47:47Z

I finished my changes for now, ready to get a review and improve the changes, or just to merge 🙂

I also took the liberty to add myself to the contributors, following the contributing guide

mih · 2020-06-25T11:06:05Z

datalad_osf/create_sibling_osf.py

+    'OSF_TOKEN', or both 'OSF_USERNAME' and 'OSF_PASSWORD'. If neither of these
+    is defined, the tool will fall back to the datalad credential manager and
+    inquire for credentials interactively.
+


FTR: When used with DataLad, it supports queries of DataLad's credential management and makes the definition of environment variables unnecessary.

What does FTR mean? Should I replace my sentence with yours?

mih · 2020-06-25T11:11:47Z

docs/source/exporthumandata.rst

+Note that the ``-s NAME_OF_REMOTE>`` flag is used to specify how ``git`` internally refers to your OSF project with the name `OSF_PROJECT_NAME`.
+It would be completely fine to use `OSF_PROJECT_NAME` also as a value for the ``-s`` flag.
+
+You can later on list your remotes from the command line using the ``git remote -v`` command.


ATM this names refers to the special remote, not a Git remote IIRC.

Still listed by git remote -v. Just w/o any details.

Is there a command to list special remotes that I should refer to instead?

Should I adjust the text to make it more clear that this is not a normal remote, but a special remote?

mih · 2020-06-25T11:15:35Z

docs/source/exporthumandata.rst


 .. code-block:: bash

    export OSF_TOKEN=YOUR_TOKEN_FROM_OSF.IO

-We are now going to use datalad to create a sibling dataset on OSF with name `osf` - this will create a new project called `OSF_PROJECT_NAME` on the OSF account associated with the OSF token in `$OSF_TOKEN`.
+We are now going to use datalad to create a sibling dataset on OSF with name `OSF_PROJECT_NAME`.
+This will create a new project called `OSF_PROJECT_NAME` on the OSF account associated with the OSF token in `$OSF_TOKEN`.


What is the rational to switch from osf to OSF_PROJECT_NAME in the example. Do you envision the need to have multiple different OSF projects for the same dataset as a common case?

Having something simple and uniform, such as osf makes a lot of sense. Especially in the case of a hierarchy of nested datasets, where one would want to be able to do a datalad push --to osf --recursive.

Do you envision the need to have multiple different OSF projects for the same dataset as a common case?

no, I don't think so,

But I would prefer to let users know (also within the example) that it's up to their discretion which name they want to use for their remote and the OSF project. --> after all, it doesn't matter whether it's called osf or something else.

But reading this again, it seems like I mixed up some things and it rather should be something like:

We are now going to use DataLad to create a sibling dataset on OSF as a "special remote". Within git-annex, we will refer to the special remote with the name $NAME_OF_REMOTE, while the project that will be created on the OSF account associated with the $OSF_TOKEN will be called $OSF_PROJECT_NAME.

mih · 2020-06-25T11:17:08Z

@sappelhoff thx for the PR. I made a bunch of comments. We also still seem to have an issue with testing of PRs from other forks.

mih · 2020-06-26T14:03:06Z

FTR: I am implementing git remote support in create-sibling-osf via #100. This requires API changes and we may want to investigate how the docs need to be bent in that light.

mih · 2020-06-26T20:46:25Z

#105 brings in a credential helper

sappelhoff · 2020-06-27T06:22:23Z

Super cool that development does not stop with the Brainhack 👍

re: my PR, I see two options

get it merged ... and then improve docs further step by step as new features get integrated
shut it down and wait with overhauling the docs until the main features have been integrated

given that I already invested some work, I am of course in favor of 1. - but I am interested to hear your opinions!

mih · 2020-06-27T09:03:12Z

@sappelhoff Will merge and go for (1)... and hoping that you will have another round of paying with it, once #100 is completed (hopefully later today).

sappelhoff · 2020-06-27T17:29:29Z

hoping that you will have another round of paying with it, once #100 is completed (hopefully later today).

sure, looking forward to it! Just ping me :-)

sappelhoff added 5 commits June 22, 2020 15:01

fix typo and flake8-docstring

10a1895

talk about env vars in docstr

1723e83

fix/enh exporthumandata example

e2a55e5

use placeholder in example

ce4dbb2

add @sappelhoff to zenodo

fec0823

use links.inc file for common links

8c83b58

sappelhoff marked this pull request as ready for review June 24, 2020 13:35

fix link, use anonymous syntax: trailing __ instead of _

cfb6a2f

sappelhoff changed the title ~~[WIP] Docs fixes and improvements~~ Docs fixes and improvements Jun 24, 2020

allcontributors bot mentioned this pull request Jun 24, 2020

docs: add sappelhoff as a contributor #103

Merged

mih reviewed Jun 25, 2020

View reviewed changes

mih merged commit a0ca891 into datalad:master Jun 27, 2020

sappelhoff deleted the docs-fixes branch June 27, 2020 17:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs fixes and improvements #102

Docs fixes and improvements #102

sappelhoff commented Jun 22, 2020 •

edited

Loading

bpoldrack commented Jun 22, 2020

sappelhoff commented Jun 22, 2020

bpoldrack commented Jun 23, 2020

bpoldrack commented Jun 23, 2020

bpoldrack commented Jun 23, 2020

bpoldrack commented Jun 23, 2020

sappelhoff commented Jun 23, 2020

sappelhoff commented Jun 23, 2020

sappelhoff commented Jun 24, 2020

allcontributors bot commented Jun 24, 2020

sappelhoff commented Jun 24, 2020

mih Jun 25, 2020

mih Jun 25, 2020

sappelhoff Jun 25, 2020

mih Jun 25, 2020

bpoldrack Jun 25, 2020

sappelhoff Jun 25, 2020

mih Jun 25, 2020

sappelhoff Jun 25, 2020

mih commented Jun 25, 2020

mih commented Jun 26, 2020

mih commented Jun 26, 2020

sappelhoff commented Jun 27, 2020

mih commented Jun 27, 2020

sappelhoff commented Jun 27, 2020

Docs fixes and improvements #102

Docs fixes and improvements #102

Conversation

sappelhoff commented Jun 22, 2020 • edited Loading

Questions that come up regarding --mode export

Questions that come up regarding --mode annex

bpoldrack commented Jun 22, 2020

sappelhoff commented Jun 22, 2020

bpoldrack commented Jun 23, 2020

bpoldrack commented Jun 23, 2020

bpoldrack commented Jun 23, 2020

bpoldrack commented Jun 23, 2020

sappelhoff commented Jun 23, 2020

sappelhoff commented Jun 23, 2020

sappelhoff commented Jun 24, 2020

allcontributors bot commented Jun 24, 2020

sappelhoff commented Jun 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mih commented Jun 25, 2020

mih commented Jun 26, 2020

mih commented Jun 26, 2020

sappelhoff commented Jun 27, 2020

mih commented Jun 27, 2020

sappelhoff commented Jun 27, 2020

sappelhoff commented Jun 22, 2020 •

edited

Loading

Questions that come up regarding `--mode export`

Questions that come up regarding `--mode annex`