[BUG]: Adding a TritonInferenceStage to a pipeline adversely impacts dask performance

[dagardner-nv]

Version

24.10

Which installation method(s) does this occur on?

Source

Describe the bug.

This issue was discovered while migrating the ransomware_detection example pipeline to use C++ mode, and the bug only occurs with the C++ impl of the TritonInferenceStage.

The issue appears to be that AsyncioRunnable creates a new asyncio loop per thread, setting TritonInferenceStage._num_threads=1 works-around the issue.

Note I modified the CreateFeaturesRWStage from the ransomware_detection example to not actually emit messages, in part to illustrate that TritonInferenceStage never actually receives any messages.

Timings
No Triton	35-40s
With Triton	~3.5 minutes + segfault

repro scripts:

• https://github.com/dagardner-nv/Morpheus/blob/dask_triton_repro/ransom_dask_repo.py
• https://github.com/dagardner-nv/Morpheus/blob/dask_triton_repro/cf.py

Minimum reproducible example

Drop the two scripts into the root of a branch-24.10 Morpheus repo:

time python ransom_dask_repo.py --input_glob=./examples/data/appshield/*/snapshot-*/*.json

time python ransom_dask_repo.py --input_glob=./examples/data/appshield/*/snapshot-*/*.json --use_triton

Relevant log output

Click here to see error details

[Paste the error here, it will be hidden by default]

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report