Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pacify the smoke tests #716

Open
rogthefrog opened this issue Nov 26, 2024 · 2 comments
Open

Pacify the smoke tests #716

rogthefrog opened this issue Nov 26, 2024 · 2 comments
Assignees

Comments

@rogthefrog
Copy link
Contributor

rogthefrog commented Nov 26, 2024

https://github.com/mlcommons/modelbench/actions

@rogthefrog rogthefrog self-assigned this Nov 26, 2024
@rogthefrog
Copy link
Contributor Author

rogthefrog commented Nov 26, 2024

Issues

  • Missing keys in secrets:

scope='nvidia-nim-api' key='api_key' instructions='See https://build.nvidia.com/' ✅

scope='azure_phi_3_5_mini_endpoint' key='api_key' instructions='Ask MLCommons admin for permission.' ✅

scope='azure_phi_3_5_moe_endpoint' key='api_key' instructions='Ask MLCommons admin for permission.' ✅

scope='mistralai' key='api_key' instructions='MistralAI API key. See https://docs.mistral.ai/getting-started/quickstart/' ✅

scope='vertexai' key='project_id' instructions='Your Google Cloud Platform project ID.' ✅

  • Missing modellab file download token ✅

  • Google Auth

google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information. ✅

  • Date parsing issue

FAILED plugins/validation_tests/test_object_creation.py::test_all_suts_can_evaluate[mistral-nemo-instruct-2407-hf] - ValueError: Cannot parse '2024-11-16T00:27:02Z' as a datetime. Date string is expected to follow '%Y-%m-%dT%H:%M:%S.%fZ' pattern. ❎

^^ the date parsing issue is happening in the HF client. They're aware of it. I've disabled the mistral-nemo-instruct SUT in the meantime.

@rogthefrog
Copy link
Contributor Author

#718

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant