Genserver time out crash in long-running pipeline #235
starcraft66
started this conversation in
General
Replies: 1 comment
-
I would suggest firing off a Task instead or moving such heavy data processing to a separate process tree. Pipelines was designed for lightweight data processing. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I wrote a crawler that crawls media galleries to download them and download files uing
HTTPoison
in a custom pipeline. This seems to work fine for small files as they download quickly but if I try to crawl a really big file like a video, I will almost always get this crash:Is there a setting I can tweak to allow my pipeline to run for much longer? I didn't see anything easily tweakable in the config.
Beta Was this translation helpful? Give feedback.
All reactions