[Relay][Op] Add unbiased variance op and corresponding support in pytorch frontend

[shiwenloong]

Unbiased variance uses N-1 as the divisor in the calculation, where N represents the number of elements. torch.std and torch.var are unbiased by default in pytorch and these unbiased ops can't be converted.
This PR adds unbiased variance op and corresponding support in pytorch frontend.

@masahi @junrushao1994 Please help to review this PR. Thanks.

[masahi]

Do we need to introduce new Relay ops? Can we just add unbiased argument to existing ops like PyTorch?

[shiwenloong]

Do we need to introduce new Relay ops? Can we just add unbiased argument to existing ops like PyTorch?

The existing variance op is a reduce op, and all reduce ops take axis, keepdims and exclude as attributes. I think adding unbiased argument to variance op will change the consistency. It will also bring more work to isolate the variance op from reduce ops.

[masahi]

I think adding unbiased argument to variance op will change the consistency. It will also bring more work to isolate the variance op from reduce ops.

I'm not completely sure what you mean by above, but how about adding VarianceAttrs with default unbiased=false? I think that should fix the coupling between variance and reduce op you are referring to?

Anyway, adding two new ops just for the sake of adding unbiased support sounds a bit overkill to me.

[shiwenloong]

I think adding unbiased argument to variance op will change the consistency. It will also bring more work to isolate the variance op from reduce ops.

I'm not completely sure what you mean by above, but how about adding VarianceAttrs with default unbiased=false? I think that should fix the coupling between variance and reduce op you are referring to?

Anyway, adding two new ops just for the sake of adding unbiased support sounds a bit overkill to me.

If we follow this modification method, we need to add more than just VarianceAttrs. Many codes and tests related to the variance operator need to be changed. For example, we should also add variance_shape_func in python/tvm/relay/op/_reduce.py. std and variance should be isolated from python/tvm/relay/op/reduce.py because their parameters are different from other reduce ops.

In my opinion, although adding two unbiased operators seems a bit redundant, it has little impact on the code structure and is of a lower probability of causing bugs.

[masahi]

I mean, the new VarianceAttrs should also have all attributes of ReduceAttrs. The underlying implementation of variance op can continue to be the reduce op.

Isn't updating VarianceCompute (like you did) and MakeVariance (to add unbiased param with default False) with corresponding changes in reduce.py enough? I don't see why we'd need to register a new shape func to variance op. I don't think it would break anything either.

Sorry I didn't look into the details, so I may be missing something.

[shiwenloong]

I mean, the new VarianceAttrs should also have all attributes of ReduceAttrs. The underlying implementation of variance op can continue to be the reduce op.

Isn't updating VarianceCompute (like you did) and MakeVariance (to add unbiased param with default False) with corresponding changes in reduce.py enough? I don't see why we'd need to register a new shape func to variance op. I don't think it would break anything either.

Sorry I didn't look into the details, so I may be missing something.

I try the way of of adding VarianceAttrs in current implementation. There are fewer problems than I thought. But the inconsistency of reduce.py still exists because std and variance take one more parameter than all other functions in the file. I am not sure if this is a noteworthy issue.

[masahi]

But the inconsistency of reduce.py still exists because std and variance take one more parameter than all other functions in the file. I am not sure if this is a noteworthy issue.

They look fine to me. I don't think a small API change like this is a problem at all. Being in the same file doesn't mean they have to have the same signature.

[masahi]

Thanks @shiwenloong @leandron