Torch-TensorRT Versioning & Stack Support

[ncomly-nvidia]

This discussion is to make clear how versioning of Torch-TRT and sw stack support is determined

Definitions

Key Terms relevant to this discussion

Long Term Support - LTS / LWS

A release of the software that continues to be supported for longer than standard, even after new versions are released.

Example:

Assume v1.0 is a standard release. v1.0 is supported until v1.1 is released, then v1.0 is no longer supported.
Assume v1.5 is an LTS release. v1.5 continues to be supported until the next LTS version is defined, even as v1.6, v1.7, etc. are released.

SW Stack Support

All of the dependencies required to make the library functional, and their corresponding versions. For each release, there will be a defined SW stack that is compatible

Example:

Torch-TRT software stack includes PyTorch, TensorRT, cuDNN, CUDA, etc.
Torch-TRT release v1.0 has specific versions of the above that are required / supported, other versions are not guaranteed to work with v1.0.

Torch-TRT Releases & Versioning

There are multiple release mechanisms for Torch-TensorRT

Stable Releases

Torch-TRT releases numbered stable versions - GitHub Releases

• Follow semantic versioning
• Released shortly after a new PyTorch version

SW Stack

Stable Torch-TRT releases currently supports two SW stacks, Latest and JetPack, however, we are considering adding another support stack, Legacy. Torch-TRT would support all three of these stacks in a given release

Latest

This stack contains the latest compatible GA versions of all dependencies

JetPack

This stack contains the sw stack that is part of the latest JetPack SDK

Legacy

This stack is yet to be defined specifically, however the idea is as follows:

• A "Legacy" SW stack will be defined as a set of versions for all dependencies (PyT, TRT, CUDA, etc.)
• The legacy stack will be supported by new versions of Torch-TRT
• Legacy stack support will persist, regardless of the number new Torch-TRT releases, until a new legacy stack is defined.

Example:

Note: The specific versions are fictitious. This is just to demonstrate how stacks transition between releases.

stack	Torch-TensorRT v2.7	Torch-TensorRT v2.8	Torch-TensorRT v2.9	Torch-TensorRT v2.10	Torch-TensorRT v2.11
Latest	PyT 7.11, TRT 11.2	PyT 7.12, TRT 11.3	PyT 7.13, TRT 12	PyT 7.14, TRT 12.1	PyT 7.15, TRT 12.2
JetPack	JP 7.9	JP 8.0	JP 8.0	JP 8.1	JP 8.1
Legacy	PyT 7.11, TRT 11.2	PyT 7.11, TRT 11.2	PyT 7.11, TRT 11.2	PyT 7.11, TRT 11.2	PyT 7.14, TRT 12.1
Notes	Legacy Defined			Legacy Updated

Latest updates every release. JetPack updates when a new JetPack is released. Legacy stack is updated at a defined interval, O(1 year).

NGC Container Releases

Torch-TRT is also distributed in NGC PyT container.

There is approx 2 month lead time to DLFW container:

• ex: items complete by early May, will be available in the June container (22.06) late June

SW Stack

Torch-TRT will support the SW stack is in that months container.

LTS Versions

Currently Torch-TRT does not have any LTS versions

[ncomly-nvidia]

@frank-wei can you and team please provide feedback on this proposal? Any suggestions on alignment of the legacy stack?

[frank-wei]

@ncomly-nvidia , FX path is relative young and does not have many history burdens. I agree with the Latest and Jetpack.
For Legacy, there is some concerns like latest change/commit is based a feature in new version of Pytorch. For this situation, this commit could not be landed on the Legacy which is based older version of Pytorch. It will bring us some troubles to pick out those kind of commits.

[ncomly-nvidia]

@frank-wei we are not looking for retroactive support currently. An example of legacy would be when PyT 1.14 comes out say it is defined as the legacy stack. From then on PyT 1.14 would need to be supported in releases only, not main.

Is that possible in FX path?

[frank-wei]

That should be ok.