Implement the lowering for HardSigmoid and HardSigmoidBackward

[JackCaoG]

test_hardsigmoid_grad_xla still won't get run because it was marked as @onlyOnCPUAndCUDA in pytorch code. That test seems to be super slow/hanging in

  File "test_nn.py", line 10275, in test_hardsigmoid_grad
    self.assertTrue(gradcheck(F.hardsigmoid, (inputs,)))
  File "/usr/local/google/home/jackcao/anaconda3/envs/pytorch2/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 291, in gradcheck
    numerical = get_numerical_jacobian(fn, tupled_inputs, eps=eps)
  File "/usr/local/google/home/jackcao/anaconda3/envs/pytorch2/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 130, in get_numerical_jacobian
    orig = x_tensor[x_idx].item()

[JackCaoG]

That test wasn't hanging, it is just super slow(15+ minutes already).. I will look into metirc once it is finished

[JackCaoG]

In https://github.com/pytorch/pytorch/blob/master/torch/autograd/gradcheck.py#L139, reshpae was in couple layer nested for loops and every reshape will cause a recompile. The test will take forever, I will leave test_hardsigmoid_grad_xla disabled. get_analytical_jacobian and get_numerical_jacobian will cause a total of 2048 recompile and these two function will be called multiple time.

[dlibenzi]

In pytorch/pytorch:torch/autograd/gradcheck.py@master#L139 , reshpae was in couple layer nested for loops and every reshape will cause a recompile. The test will take forever, I will leave test_hardsigmoid_grad_xla disabled. get_analytical_jacobian and get_numerical_jacobian will cause a total of 2048 recompile and these two function will be called multiple time.

Gradient tests should all be disabled until we fix the gradient-check code.
@ailzhang why are we suddenly get those?

[JackCaoG]

In pytorch/pytorch:torch/autograd/gradcheck.py@master#L139 , reshpae was in couple layer nested for loops and every reshape will cause a recompile. The test will take forever, I will leave test_hardsigmoid_grad_xla disabled. get_analytical_jacobian and get_numerical_jacobian will cause a total of 2048 recompile and these two function will be called multiple time.

Gradient tests should all be disabled until we fix the gradient-check code.
@ailzhang why are we suddenly get those?

The test was added and disabled last week because hardsigmoid_backward was not implemented for XLA. I implemented the lowering and tried to reenable the test and run into this slowness problem. The test is disabled so we should be good. I didn't know we should always skip Gradient tests.

[ailzhang]

@dlibenzi So this test was added to TestNN, and historically TestNN has some gradcheck tests. :(

[ailzhang]

@JackCaoG If gradcheck is the root cause, feel free to add it to the skipped test in pytorch/xla. And maybe submit a pr in Pytorch later to remove onlyOnCPUAndCUDA, so that we control when to turn it on.

[JackCaoG]

@JackCaoG If gradcheck is the root cause, feel free to add it to the skipped test in pytorch/xla. And maybe submit a pr in Pytorch later to remove onlyOnCPUAndCUDA, so that we control when to turn it on.

Sounds good! It is better we keep it in our file otherwise we will forget about it very soon

[dlibenzi]

Thanks @JackCaoG looks nice!