Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QNN] Optimize lowering for requantize and FixedPointMultiply. #4798

Merged
merged 3 commits into from
Feb 5, 2020
Merged

[QNN] Optimize lowering for requantize and FixedPointMultiply. #4798

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ecb5681
[QNN] Optimize lowering for requantize and FixedPointMultiply.
anijain2305 Feb 1, 2020
a6e2d34
Add check for requantize scale gt 1.
anijain2305 Feb 4, 2020
385a3d0
Added test case.
anijain2305 Feb 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/relay/qnn/op/requantize.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,11 @@ Expr RequantizeLower(const Expr& input_tensor, const Expr& input_scale,
shifted_int64_t = Add(Cast(output_zero_point, hp_dtype), scaled_int64_t);
}

// 4) Clip to the out_dtype min/max.
// 4) Clip to the out_dtype min/max. Skip clipping if out_dtype is Int32. The fixed point
// multiplication keeps the value in int32 range.
if (out_dtype == DataType::Int(32)) {
return Cast(shifted_int64_t, out_dtype);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you please share the insight here? I looked around, but lost a bit in the arithmetic here. :)

Copy link
Contributor Author

@anijain2305 anijain2305 Feb 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely, happy to explain :)

We approximate the floating point computation here with fixed point computation. This is done by representing the requantize_scale (input_scale/output_scale) as int32, where the decimal point is between 1st and 2nd bit - representing a number between 0.5 and 1. And then we multiply this fixed point number with the quantized tensor (another int32 tensor). So, to keep the precision higher, we perform multiplication in int64. But, we can safely say that the resulting number is still a fixed point int64 number, where the decimal part of the number is within int32 range. We, then perform, right shift etc to get the decimal portion.

So, if the requantize scale is less than 1, we can safely assume that the range will be within int32. (I forgot to add that check, but let me add that as a second commit).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thank you for the detailed explain!

auto q_min = GetQmin(out_dtype);
auto q_max = GetQmax(out_dtype);
auto clipped_t = Clip(shifted_int64_t, q_min, q_max);
Expand Down
10 changes: 7 additions & 3 deletions src/relay/qnn/util.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ Expr FixedPointMultiplyPerChannel(Expr tensor, std::vector<double> multipliers,
// 1) Calculating the integer multiplier and integer shift. These are calculated per axis/per
// channel.
std::vector<int32_t> fixed_pt_multipliers, lshifts, rshifts;
bool is_lshift_required = false;
for (auto multiplier : multipliers) {
int32_t fixed_pt_multiplier, shift;
std::tie(fixed_pt_multiplier, shift) = GetFixedPointMultiplierShift(multiplier);
Expand All @@ -157,12 +158,15 @@ Expr FixedPointMultiplyPerChannel(Expr tensor, std::vector<double> multipliers,
fixed_pt_multipliers.push_back(fixed_pt_multiplier);
lshifts.push_back(lshift);
rshifts.push_back(rshift);
is_lshift_required |= (lshift != 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we save the [?]= style operators for artimethic, but write boolean as it originally is.

}

// 2) Multiply the integer multiplier. Convert lefts shifts into expr and multiply.
auto lshift_expr = MakeConstantTensor(hp_dtype, {n_channels}, lshifts);
auto exp_lshift_expr = ExpandBiasToMatchAxis(lshift_expr, n_dim, {channel_axis});
tensor = LeftShift(tensor, exp_lshift_expr);
if (is_lshift_required) {
auto lshift_expr = MakeConstantTensor(hp_dtype, {n_channels}, lshifts);
auto exp_lshift_expr = ExpandBiasToMatchAxis(lshift_expr, n_dim, {channel_axis});
tensor = LeftShift(tensor, exp_lshift_expr);
}

// 3) Perform the multiplication in higher precision.
// The scalar is a fixed point value of int32 where the decimal point is
Expand Down