Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TTRT - Golden Artifacts #1507

Closed
vprajapati-tt opened this issue Dec 4, 2024 · 1 comment · Fixed by #1544
Closed

TTRT - Golden Artifacts #1507

vprajapati-tt opened this issue Dec 4, 2024 · 1 comment · Fixed by #1544
Assignees
Labels
enhancement New feature or request ttrt

Comments

@vprajapati-tt
Copy link
Contributor

  • After running ttrt with the --golden option doesn't actually yield any specific artifacts beyond printing to stdout. It would be useful for end users (and also explorer) to store these results.
  • Currently Model Accuracy Overlay - TT-Explorer #1234 is blocked on this support being included.
@tapspatel
Copy link
Contributor

tapspatel commented Dec 9, 2024

Features to add:

  • store golden tensors to artifacts as torch tensors with name=loc
  • store intermediate tensors to artifacts as torch tensors with name=loc
  • track test pass/failure and raise an exception on mismatched pcc
  • print golden pcc dictionary of all intermediate/output tensor comparisons and their golden results
    pcc, atol, rtol, torch.allclose, torch.max(torch.abs(a - b)), torch.mean(torch.abs(a - b)), torch.sqrt(torch.mean((a - b) ** 2)), torch.nn.functional.cosine_similarity(a.unsqueeze(0), b.unsqueeze(0)),

tapspatel added a commit that referenced this issue Dec 9, 2024
…ate into report as well as store golden and device artifacts generated during runtime
tapspatel added a commit that referenced this issue Dec 10, 2024
…ate into report as well as store golden and device artifacts generated during runtime
tapspatel added a commit that referenced this issue Dec 11, 2024
…ate into report as well as store golden and device artifacts generated during runtime
tapspatel added a commit that referenced this issue Dec 11, 2024
…ate into report as well as store golden and device artifacts generated during runtime (#1544)
azecevicTT pushed a commit that referenced this issue Dec 17, 2024
…ate into report as well as store golden and device artifacts generated during runtime (#1544)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request ttrt
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants