add int8 quantization support

[lanluo-nvidia]

Description

Add int8 quantization support

Fixes # (issue)

Type of change

Please delete options that are not relevant and/or add your own.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes
• I have added the relevant labels to my PR in so that relevant reviewers are notified

[peri044]

consider adding modelopt as optional dependency with the correct version

[github-actions]

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/examples/dynamo/simple_int8_ptq.py	2024-08-21 23:27:44.130840+00:00
+++ /home/runner/work/TensorRT/TensorRT/examples/dynamo/simple_int8_ptq.py	2024-08-21 23:28:02.118758+00:00
@@ -14,13 +14,15 @@
        x = self.linear1(x)
        x = torch.nn.ReLU()(x)
        x = self.linear2(x)
        return x

+
def calibrate_loop(model):
    """Simple calibration function for testing."""
    model(input_tensor)
+

input_tensor = torch.randn(1, 6).cuda()
model = SimpleNetwork().eval().cuda()
print(f"model before quantize: {model}")

[github-actions]

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/examples/dynamo/simple_int8_ptq.py	2024-08-21 23:28:32.341283+00:00
+++ /home/runner/work/TensorRT/TensorRT/examples/dynamo/simple_int8_ptq.py	2024-08-21 23:28:50.802608+00:00
@@ -14,13 +14,15 @@
        x = self.linear1(x)
        x = torch.nn.ReLU()(x)
        x = self.linear2(x)
        return x

+
def calibrate_loop(model):
    """Simple calibration function for testing."""
    model(input_tensor)
+

input_tensor = torch.randn(1, 6).cuda()
model = SimpleNetwork().eval().cuda()
print(f"model before quantize: {model}")

[lanluo-nvidia]

@dheerajperi @narendasan ready for review
tested with modelopt source from main with this PR, it is working for both int8 and fp8.

however in the testcase I has to change fromtorch.export.export()to torch.export._trace._export()
otherwise got the following error:
RuntimeError: Attempting to use FunctionalTensor on its own. Instead, please use it with a corresponding FunctionalTensorMode()
here is the slack discussion with Wei-Ming Chen from modelopt team: https://nvidia.slack.com/archives/C06K7JPMXE1/p1724280167574089

[github-actions]

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/examples/dynamo/simple_int8_ptq.py	2024-08-22 00:07:47.982949+00:00
+++ /home/runner/work/TensorRT/TensorRT/examples/dynamo/simple_int8_ptq.py	2024-08-22 00:08:05.777701+00:00
@@ -14,13 +14,15 @@
        x = self.linear1(x)
        x = torch.nn.ReLU()(x)
        x = self.linear2(x)
        return x

+
def calibrate_loop(model):
    """Simple calibration function for testing."""
    model(input_tensor)
+

input_tensor = torch.randn(1, 6).cuda()
model = SimpleNetwork().eval().cuda()
print(f"model before quantize: {model}")

[narendasan]

Shouldnt these be joined with a default set like enabled_precisions = {torch.float, torch.half}

[narendasan]

Seems good to me, one small optional idea for the docs