-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Grantham/new methods flytefile and flytedirectory #2842
Grantham/new methods flytefile and flytedirectory #2842
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2842 +/- ##
===========================================
- Coverage 95.61% 45.55% -50.07%
===========================================
Files 31 196 +165
Lines 1094 20384 +19290
Branches 0 2633 +2633
===========================================
+ Hits 1046 9285 +8239
- Misses 48 10640 +10592
- Partials 0 459 +459 ☔ View full report in Codecov by Sentry. |
* Catch mistake in structured dataset (#2834) Signed-off-by: Yee Hing Tong <[email protected]> * Small change to clean up unit test. (#2835) * delete=true Signed-off-by: Yee Hing Tong <[email protected]> * close but don't delete on close Signed-off-by: Yee Hing Tong <[email protected]> * skip for 3.9 Signed-off-by: Yee Hing Tong <[email protected]> * 10 Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * Fix tree printing (#2837) Signed-off-by: Yee Hing Tong <[email protected]> * handle case where error may not have args (#2831) Signed-off-by: Blake Jackson <[email protected]> Co-authored-by: Blake Jackson <[email protected]> * Bump pyspark from 3.3.1 to 3.3.2 in /plugins/flytekit-greatexpectations (#2818) Bumps [pyspark](https://github.com/apache/spark) from 3.3.1 to 3.3.2. - [Commits](apache/spark@v3.3.1...v3.3.2) --- updated-dependencies: - dependency-name: pyspark dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pull secrets from environment when running locally (#2800) Signed-off-by: Thomas J. Fan <[email protected]> * Support executing launchplans from CLI (#2839) Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Ketan Umare <[email protected]> * Add top-level access to FlyteRemote, FlyteFile, and FlyteDirectory and convenience class methods for FlyteRemote (#2836) --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Blake Jackson <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Thomas J. Fan <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Blake Jackson <[email protected]> Co-authored-by: Blake Jackson <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Ketan Umare <[email protected]>
* Catch mistake in structured dataset (#2834) Signed-off-by: Yee Hing Tong <[email protected]> * Small change to clean up unit test. (#2835) * delete=true Signed-off-by: Yee Hing Tong <[email protected]> * close but don't delete on close Signed-off-by: Yee Hing Tong <[email protected]> * skip for 3.9 Signed-off-by: Yee Hing Tong <[email protected]> * 10 Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * Fix tree printing (#2837) Signed-off-by: Yee Hing Tong <[email protected]> * handle case where error may not have args (#2831) Signed-off-by: Blake Jackson <[email protected]> Co-authored-by: Blake Jackson <[email protected]> * Bump pyspark from 3.3.1 to 3.3.2 in /plugins/flytekit-greatexpectations (#2818) Bumps [pyspark](https://github.com/apache/spark) from 3.3.1 to 3.3.2. - [Commits](apache/spark@v3.3.1...v3.3.2) --- updated-dependencies: - dependency-name: pyspark dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pull secrets from environment when running locally (#2800) Signed-off-by: Thomas J. Fan <[email protected]> * Support executing launchplans from CLI (#2839) Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Ketan Umare <[email protected]> * Add top-level access to FlyteRemote, FlyteFile, and FlyteDirectory and convenience class methods for FlyteRemote (#2836) * Config for_endpoint doesn't respect config file (#2843) Signed-off-by: Yee Hing Tong <[email protected]> * Union/enum handling (#2845) * temp Signed-off-by: Yee Hing Tong <[email protected]> * delete test Signed-off-by: Yee Hing Tong <[email protected]> * allow strings Signed-off-by: Yee Hing Tong <[email protected]> * remove enum transformer change and tests Signed-off-by: Yee Hing Tong <[email protected]> * reset a test Signed-off-by: Yee Hing Tong <[email protected]> * lint Signed-off-by: Yee Hing Tong <[email protected]> * use .value for enums to handle dynamic case Signed-off-by: Yee Hing Tong <[email protected]> * use prior logic for 3.9 Signed-off-by: Yee Hing Tong <[email protected]> * add 3.9 testing Signed-off-by: Yee Hing Tong <[email protected]> * should always be a union type Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * update docs for FlyteRemote (#2847) --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Blake Jackson <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Thomas J. Fan <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Blake Jackson <[email protected]> Co-authored-by: Blake Jackson <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Ketan Umare <[email protected]>
I feel like it is verbose to get the working directory (
flytekit.current_context().working_directory
) in a task. I find myself doing this very, very frequently, primarily to write files. I use the working directory so that files won’t get written to my local working directory during local execution.I had thought about this for a while, and the more I thought about what exactly I wanted to accomplish, I feel like it should be accomplished with a few new classmethod for
FlyteFile
/FlyteDirectory
...This task...
Should ideally be replaced with something like this:
In this case,
FlyteFile.new
will create a FlyteFile with a path of the filename inside of the working directory (current_context().working_directory
).The same goes for FlyteDirectory...
Should be replaced with:
In this case,
FlyteDirectory.new
will create the directorymy_new_directory
undercurrent_context().working_directory
, and ensure that it exists (os.makedirs(dir.path)
)Additionally, I am overloading the
__truedivide__
operator toFlyteDirectory
for convenience: