Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have agents set configurable working directory on the runs they submit #2857

Closed
cicdw opened this issue Jun 24, 2020 · 1 comment
Closed
Labels
enhancement An improvement of an existing feature

Comments

@cicdw
Copy link
Member

cicdw commented Jun 24, 2020

This is motivated by the following scenario:

  • a user commits flow code to GitHub, and their script that defines the flow actually uses relative imports from other files in their Git repo
  • user uses the new GitHub Storage
  • user finds ImportErrors all over the place, because the relative imports break

We see similar issues with locally registered flows that are stored as pickles.

Ideally, at least for file-based storage, we can infer a "smart" place to run the flow from. In the case of GitHub Storage, we'd want to clone the repo and then use the directory that the flow-defining file lives in as the PWD for execution. This can be inferred entirely off the configuration on the storage object.

This location could also be overridable by the user if they choose.

@cicdw cicdw added the enhancement An improvement of an existing feature label Jun 24, 2020
@jcrist jcrist changed the title Have agents set configurable PWD on the runs they submit Have agents set configurable working directory on the runs they submit Jun 24, 2020
@joshmeek joshmeek self-assigned this Jun 29, 2020
@joshmeek
Copy link

joshmeek commented Jul 8, 2020

After getting a working version of this up and running I don't see a clear / clean pattern for achieving this. My initial implementation uses a recursive pattern for downloading each file in a set working directory of the repo however this adds a clump of latency and we definitely do not want to be downloading a large amount of files for each run. With GitHub's current API implementation there isn't a clean way to "clone" a repo so it would require some mix of using GitHub's API with native git bindings and that doesn't feel right / would definitely get weird when trying to work with private repositories.

Discussed this with @cicdw offline it seems like the issue this is trying to solve would be better suited with some sort of git sync that agents could use to mount files on flow runs. That way agent's could be "warmed up" with a repository's contents for use. The issue is still valid but I feel like this could be rolled into something else.

@joshmeek joshmeek removed their assignment Jul 9, 2020
@cicdw cicdw closed this as completed Aug 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

2 participants