Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(weave): include scores from scorers with illegal characters in name #2731

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

fix(weave): include scores from scorers with illegal characters in name #2731

wants to merge 5 commits into from
+52 −1

Conversation

gtarpenning
Copy link
Member

@gtarpenning gtarpenning commented Oct 17, 2024 •
edited
Loading

Description

This pr includes:

  • Hack frontend fix for scorers with illegal chars in the name. The scorer becomes a ref, which sanitizes the name, then when we look them up in the score summary the names don't match. Use a regex with a wildcard to match up the sanitized names to the raw summary value ones.
  • a python fix that sanitizes the name so that the name in the summary will match the artifact name.

Note: Scorers with names like v1 and v1! will be treated as identical...

Master:
Screenshot 2024-10-17 at 10 02 55 AM

This branch:
Screenshot 2024-10-17 at 10 02 02 AM

This is the UI for the above two runs, while the unsanitized scorers look different, we can still compare them in the UI.
Screenshot 2024-10-17 at 10 38 48 AM

Testing

Added test

@circle-job-mirror
Copy link

circle-job-mirror bot commented Oct 17, 2024 •
edited
Loading

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=b8ccd62ef681dc067c04f47a0c2886df00b5443d

gtarpenning
gtarpenning commented Oct 17, 2024
View reviewed changes
weave/flow/scorer.py
Comment on lines +107 to +108
if not scorer_name:
raise ValueError("Scorer name is required")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required by lint for some reason, should be impossible

@gtarpenning gtarpenning changed the title fix: include scores from scorers with spaces in name fix(weave): include scores from scorers with spaces in name Oct 17, 2024
@gtarpenning gtarpenning marked this pull request as ready for review October 17, 2024 17:40
@gtarpenning gtarpenning requested review from a team as code owners October 17, 2024 17:40
@gtarpenning gtarpenning changed the title fix(weave): include scores from scorers with spaces in name fix(weave): include scores from scorers with illegal characters in name Oct 17, 2024
@gtarpenning gtarpenning enabled auto-merge (squash) October 17, 2024 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tssweeney tssweeney tssweeney approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gtarpenning @tssweeney