Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

Team wise storage for NFS data storage #2204

Closed
scarlett2018 opened this issue Feb 25, 2019 · 3 comments
Closed

Team wise storage for NFS data storage #2204

scarlett2018 opened this issue Feb 25, 2019 · 3 comments

Comments

@scarlett2018
Copy link
Member

scarlett2018 commented Feb 25, 2019

Rescoping
For time and effort limitations, the belowing features will be limited in the context of NFS storage in v0.11.0

What would you like to be added:

  • PAI supports to set up multiple storage;
  • PAI admin can set up users default and accessible storage;
  • Storage between different teams can be isolated through settings;

Why is this needed:

  • security considerations: data isolation;

Without this feature, how does the current module work
all the teams are share one single storage set up

Components that may involve changes:
storage_plugin

@wangdian
Copy link
Member

PR created:
#2346

wangdian added a commit that referenced this issue Mar 20, 2019
Team wise storage management plugin:

Store server config and user config in k8s configmap
CommandLine tool to manage storage config in dev-box

#2204
@wangdian
Copy link
Member

wangdian commented Mar 22, 2019

Test Case:

Prerequisites:

  1. Set up nfs server, e.g: 10.0.0.1:/share/teamA with settings: (rw, sync, no_root_squash)
  2. Install PAI plugin submit-nfs-job [https://github.com/Microsoft/pai/tree/master/contrib/submit-nfs-job] [Plugin]Add simple NFS job #2358
  3. Start PAI dev-box

Steps:

  1. In dev-box, go to pai/contrib/storage_plugin, install jsonschema
    pip install jsonschema
  2. Create server config:
    python storagectl.py server set teamA 10.0.0.1 /share/teamA --sharedfolders data --privatefolders users
  3. Create user config:
    python storagectl.py user setdefault user1 teamA
  4. Go to nfs server, check if /share/teamA/data and /share/teamA/users/user1 folder was created
  5. Login with user1 in PAI, go to submit-nfs-job page, check if nfs mount section filled with right information
  6. Submit a job and check if the job files was created on nfs

@scarlett2018 scarlett2018 changed the title Team wise storage Team wise storage phase I - simplified implementation for NFS data storage Mar 22, 2019
@scarlett2018 scarlett2018 changed the title Team wise storage phase I - simplified implementation for NFS data storage Team wise storage for NFS data storage Mar 22, 2019
@ydye
Copy link
Contributor

ydye commented Mar 25, 2019

@wangdian When in steps2, come across this error.

root@xxxxxx:/pai/contrib/storage_plugin# python storagectl.py storage set default x.x.x.x /share/default --sharedfolders data --privatefolders user
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
usage: storagectl.py [-h] {init,server,user,push} ...
storagectl.py: error: invalid choice: 'storage' (choose from 'init', 'server', 'user', 'push')

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants