Discussions

• 2 You must be logged in to vote

  🙏

  Ensemble/BLS models where individual steps are hosted on different machines (or clusters)?

  scalability Topics pertaining to scaling, multi-gpu, multi-node

  vadimkantorov asked Apr 26, 2024 in Q&A · Unanswered

  

  

  2
• 2 You must be logged in to vote

  🙏

  What would be the simplest way to use triton_python_backend_utils just to work with pb_utils.InferenceRequest and related abstractions?

  nybbles asked Apr 11, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to sepeifiy world_size for tensorrtllm backend?

  Saigut asked Apr 9, 2024 in Q&A · Unanswered

  

  

  

  

  3
• 1 You must be logged in to vote

  🙏

  What's the version of TensorRT-LLM in nvcr.io/nvidia/tritonserver:24.02-trtllm-python-py3?

  Saigut asked Apr 8, 2024 in Q&A · Closed · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Jetson Xavier AGX not using GPU

  roundPot asked Mar 25, 2024 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  💬

  How to decrypt models using Triton repository agent?

  wfd2022 started Apr 2, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  .

  jhassler asked Mar 26, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 2 You must be logged in to vote

  🙏

  In production ,is running multi triton-inference-server on 1 GPU a good idea?

  oymy asked Mar 27, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Understanding High Memory Usage When Loading Models in Triton Inference Server

  sikso1892 asked Mar 26, 2024 in Q&A · Closed · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  java sdk request tritonserver

  wuyonglei asked Mar 20, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💡

  Reduce image size and increase build frequency to cut high/medium vulnerabilities in tritonserver docker images

  mirekphd started Feb 19, 2023 in Ideas

  

  

  

  

  3
• 1 You must be logged in to vote

  💬

  Load fine-tuned model from local

  hshabbirh started Mar 4, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Why vllm-backend prepends output with the prompt?

  mkhludnev asked Feb 6, 2024 in Q&A · Answered

  

  

  

  

  9
• 1 You must be logged in to vote

  🙏

  In TritonServer, if I use gRPC requests with timeouts set, will the request continue to execute after the timeout?

  SunnyGhj asked Feb 26, 2024 in Q&A · Answered

  

  

  1
• 2 You must be logged in to vote

  💬

  Example/tutorial on infer LLMs deployed using onnx/onnxruntime

  mgiessing started Feb 26, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Custom model orchestration

  tingc9 asked Aug 4, 2023 in Q&A · Answered

  

  

  

  

  3
• 4 You must be logged in to vote

  🙏

  Triton server prediction streaming interface for LLM

  sepo-eng asked Jul 22, 2023 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  Can I run Triton Inference Server using multiple MIG instances?

  tunahanertekin started Oct 23, 2023 in General

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  [Question] What is the recommended way to run Triton ?

  question Further information is requested

  MatthieuToulemont asked Jun 22, 2023 in Q&A · Answered

  

  

  

  

  10
• 1 You must be logged in to vote

  🙏

  Existing Scaling Benchmarks and Model Grouping Pattern

  nastrofaction asked Jan 25, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Graceful Termination

  bro-adm asked Jan 12, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Shared memory regions are not available on Windows

  hahne asked Jan 4, 2024 in Q&A · Unanswered

  

  

  0
• 0 You must be logged in to vote

  🙏

  When max_batch_size=0, how to set cuda graph shape

  SunnyGhj asked Dec 29, 2023 in Q&A · Closed · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Build.py script fails: permissions error?

  bpickrel asked Nov 9, 2023 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  When I use the tition inference service with opentelemetry the inference performance is much worse than without adding opentelemetry. How to provide inference performance using opentelemtry？

  Leelongqi asked Dec 25, 2023 in Q&A · Unanswered

  

  

  1