Grantham/new methods flytefile and flytedirectory

[granthamtaylor]

I feel like it is verbose to get the working directory (flytekit.current_context().working_directory) in a task. I find myself doing this very, very frequently, primarily to write files. I use the working directory so that files won’t get written to my local working directory during local execution.

I had thought about this for a while, and the more I thought about what exactly I wanted to accomplish, I feel like it should be accomplished with a few new classmethod for FlyteFile / FlyteDirectory...

---

This task...

@task
def my_task() -> FlyteFile:
    local_path = os.path.join(current_context().working_directory, "data.txt")
    with open(local_path, mode="w") as file:
        file.write("Here is some sample data.")
    return FlyteFile(path=local_path)

Should ideally be replaced with something like this:

@task
def my_task() -> FlyteFile:

    ff = FlyteFile.new('myfile.txt')

    with open(ff, mode="w") as file:
        file.write("Here is some sample data.")

    return ff

In this case, FlyteFile.new will create a FlyteFile with a path of the filename inside of the working directory (current_context().working_directory).

---

The same goes for FlyteDirectory...

@task
def my_task() -> FlyteDirectory:

    p = os.path.join(current_context().working_directory, "my_new_directory")

    os.makedirs(p)

    with open(os.path.join(p, "file_1.txt"), 'w') as file:
        file.write("This is file 1.")

    return FlyteDirectory(p)

Should be replaced with:

@task
def my_task() -> FlyteDirectory:

    dir = FlyteDirectory.new('my_new_directory')

    with open(dir / "file_1.txt", 'w') as file:
        file.write("This is file 1.")

    return dir

In this case, FlyteDirectory.new will create the directory my_new_directory under current_context().working_directory, and ensure that it exists (os.makedirs(dir.path))

Additionally, I am overloading the __truedivide__ operator to FlyteDirectory for convenience:

class FlyteDirectory:

    ...

    def __truedivide__(self, other: str | os.PathLike) -> Path:
        """
        This is a convenience method to allow for easy concatenation of paths.
        """
        return Path(self.path) / other

[codecov]

Codecov Report

Attention: Patch coverage is 45.16129% with 34 lines in your changes missing coverage. Please review.

Project coverage is 45.55%. Comparing base (194300f) to head (6511968).
Report is 10 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/remote/remote.py	38.09%	13 Missing ⚠️
flytekit/types/directory/types.py	30.00%	7 Missing ⚠️
flytekit/types/file/file.py	28.57%	5 Missing ⚠️
flytekit/tools/fast_registration.py	0.00%	4 Missing ⚠️
flytekit/core/context_manager.py	83.33%	0 Missing and 2 partials ⚠️
flytekit/types/structured/structured_dataset.py	0.00%	2 Missing ⚠️
flytekit/core/base_task.py	50.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (194300f) and HEAD (6511968). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (194300f)	HEAD (6511968)
	5	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2842       +/-   ##
===========================================
- Coverage   95.61%   45.55%   -50.07%     
===========================================
  Files          31      196      +165     
  Lines        1094    20384    +19290     
  Branches        0     2633     +2633     
===========================================
+ Hits         1046     9285     +8239     
- Misses         48    10640    +10592     
- Partials        0      459      +459     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.