Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a base DPS image for users #966

Closed
sujen1412 opened this issue Apr 1, 2024 · 6 comments
Closed

Create a base DPS image for users #966

sujen1412 opened this issue Apr 1, 2024 · 6 comments
Assignees
Labels
ADE Algorithm Development Environment Subsystem DPS Data Processing Subsystem JPL JPL related issues

Comments

@sujen1412
Copy link
Collaborator

Currently, users are using (as base image) a pre-built base_image from https://github.com/MAAP-Project/maap-workspaces/tree/main/base_images.
When an algorithm is submitted from registration we are including additional steps for creating a user ops and re-doing all permissions which takes up a lot of time during registration.

TODO: Move up the work required at minimum by DPS to base images to save time during registration.

Future (not covered in this ticket) : How to optimize space taken by these images.

@sujen1412 sujen1412 added JPL JPL related issues ADE Algorithm Development Environment Subsystem DPS Data Processing Subsystem labels Apr 1, 2024
@sujen1412 sujen1412 self-assigned this Apr 1, 2024
@rtapella
Copy link
Collaborator

how does this relate to Stacks?

@sujen1412
Copy link
Collaborator Author

First pass, we will have a basic conda image as base image with no other packages installed.

@sujen1412
Copy link
Collaborator Author

sujen1412 commented Jun 24, 2024

@chuckwondo @wildintellect Please try mas.dit.maap-project.org/root/maap-workspaces/base_images/dps:dit. Built from https://github.com/MAAP-Project/maap-workspaces/blob/dps_base_image/dps_base_image/Dockerfile

@chuckwondo
Copy link
Collaborator

@sujen1412 and I wrangled things in DIT and saw the GEDI Subsetter image go from 10.4GB using the python:v4.0.0 base image down to 4.31GB using the pared down dps image, so we saw ~60% reduction in size.

@chuckwondo
Copy link
Collaborator

@sujen1412, is there more you need to do before you can push the stripped down DPS base image to OPS?

@sujen1412
Copy link
Collaborator Author

Yes, there are a few more tests I need to run to ensure that jobs run successfully on the workers. This is done to test user permissions on directories that are mounted within a container when the job runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ADE Algorithm Development Environment Subsystem DPS Data Processing Subsystem JPL JPL related issues
Projects
None yet
Development

No branches or pull requests

3 participants