Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: package download is tied to the watcher service #927

Open
jraddaoui opened this issue Apr 19, 2024 · 1 comment
Open

Problem: package download is tied to the watcher service #927

jraddaoui opened this issue Apr 19, 2024 · 1 comment

Comments

@jraddaoui
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

The watcher service can watch multiple source locations and it's used as a dependency in the download activity to get the package from the watcher location that triggered the processing workflow. There are cases where the watcher is a filesystem watcher and downloading may not be needed, and I'm looking at options to run preprocessing child workflows remotely where the initial download would happen there and the preprocessing result (zipped bag) would be shared using a different bucket.

Describe the solution you'd like

I'd like a download activity that only relies on bucket(s), ideally in the temporal-activities repository, where it could be shared with those preprocessing child workflows.

Describe alternatives you've considered

Keep the current download activity while adding a new one as described in the solution above. Run them (or not) based on the preprocessing configuration.

Additional context

None.

@jraddaoui
Copy link
Collaborator Author

Re #874 and #867.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant