Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ttl.tensor.recip operation breaks with low PCC [Wormhole] [Doc] #3561

Closed
Tracked by #3657 ...
nemanjagrujic opened this issue Nov 2, 2023 · 6 comments
Closed
Tracked by #3657 ...
Assignees
Labels
bug Something isn't working op_cat: eltwise P2 WH

Comments

@nemanjagrujic
Copy link
Contributor

nemanjagrujic commented Nov 2, 2023

ttl.tensor.recip operation breaks with low PCC error in some test cases.

To Reproduce
Steps to reproduce the behavior:

  1. Checkout main branch
  2. Run unit test test_eltwise_recip.py using this command:
    pytest tests/tt_eager/python_api_testing/non_working_unit_tests/grayskull/test_eltwise_recip.py
    [Same path as for grayskull test]

Expected behavior
There are four test cases presented in the unit test test_eltwise_recip.py and they all are expected to fail with low PCC error.
For example, one of the tests is expected to fail with this result:
Max ATOL Delta: inf, Max RTOL Delta: inf, PCC: 1.0926756631169207e-06

Getting Additional info for the operation under test and its behavior
To get additional information and results for different combinations of input shapes, types, layouts and memory configs for which this operation was tested you can also run locally sweeps for ttl.tensor.recip and check the results. To do this you should:

  1. Follow the Getting Started page to setup the repo, environment variables and python-env
  2. Activate source build/python_env/bin/activate
  3. Run sweeps by using python tests/tt_eager/python_api_testing/sweep_tests/run_pytorch_test.py -i tests/tt_eager/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/pytorch_eltwise_recip_test.yaml -o ./result-sweeps
  4. After the run is completed all test sweeps results should be available inside specified output directory (in this case ./result-sweeps). There you will find eltwise_recip_sweep.csv which holds all executed sweeps, among which you can also find the ones that failed and were recreated by the unit test, which you can get by searching unique data_seed field.
@VirdhatchaniKN
Copy link
Contributor

Hi @nemanjagrujic , @jliangTT
Fix provided in branch Aswinmcw/fix_recip works .

Screenshot 2024-04-04 at 5 23 14 PM

@nemanjagrujic
Copy link
Contributor Author

@VirdhatchaniKN Tested, tests pass now (both unit and sweep). I will close the ticket once pull request is merged.

@ruthreshx
Copy link
Contributor

From the latest main, I'm facing the valid_page_size issue, currently debugging the same

@ruthreshx ruthreshx assigned ruthreshx and unassigned umadevimcw Aug 29, 2024
@ruthreshx ruthreshx mentioned this issue Aug 29, 2024
3 tasks
@ruthreshx
Copy link
Contributor

Hi @eyonland , @nemanjagrujic ,
The failure is related to range issues, that will be udpated during the doc task.
Thank you

@ruthreshx ruthreshx changed the title ttl.tensor.recip operation breaks with low PCC [Wormhole] ttl.tensor.recip operation breaks with low PCC [Wormhole] [Doc] Sep 18, 2024
@mcw-anasuya
Copy link
Contributor

All four test cases fail with the expected message.
Screenshot 2024-10-25 at 16 11 31
Hence closing

@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in External Requests and Reports Oct 25, 2024
@nemanjagrujic
Copy link
Contributor Author

@mcw-anasuya Just to double check. It is expected that 3rd case fails with:

PCC check failed

?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working op_cat: eltwise P2 WH
Projects
None yet
Development

No branches or pull requests

8 participants