Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix smoke tests #718

Merged
merged 14 commits into from
Nov 27, 2024
Merged

Fix smoke tests #718

merged 14 commits into from
Nov 27, 2024

Conversation

rogthefrog
Copy link
Contributor

@rogthefrog rogthefrog commented Nov 26, 2024

Wins

  • Fix missing credentials
  • Add Google authentication for Vertex AI
  • Temporarily disable a SUT on Huggingface until the HF client can handle its own API's datetime fields 🙄
  • Add an option for fake secrets to use specified real secrets in case they are needed for some steps (in this case, downloading a prompt file from modellab, which requires an auth token)

Fails

I tried to extract the credential/secret logic into a dependent workflow so it could be shared between the app test and smoke test workflows. That didn't work because:

  • a secrets.toml file created in the runner file system in the dependent workflow doesn't get preserved when the caller workflow runs
  • the canonical way to share files between workflows is to upload them as artifacts. I didn't want to do with secrets.toml because it's full of secrets, and I couldn't find enough information about artifact ACLs to be comfortable uploading it as an artifact.
  • setting the contents of the secrets.toml file as an output of the dependent job does not work, because github actions helpfully strip out outputs that include secrets.

So I just C&P'ed the secrets section into a second workflow, like a neanderthal.

Copy link

github-actions bot commented Nov 26, 2024

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

…f you're not asked to preserve some of those secrets
…the test step. This isn't the prettiest way to do it, but using reusable workflows with outputs doesn't work because secret-containing outputs get scrubbed by Github.
@rogthefrog rogthefrog changed the title Fix smoke tests (2) Fix smoke tests Nov 27, 2024
@rogthefrog rogthefrog marked this pull request as ready for review November 27, 2024 03:59
@rogthefrog rogthefrog requested a review from a team as a code owner November 27, 2024 03:59
@rogthefrog rogthefrog merged commit df113b7 into main Nov 27, 2024
4 checks passed
@rogthefrog rogthefrog deleted the fix-716 branch November 27, 2024 17:32
@github-actions github-actions bot locked and limited conversation to collaborators Nov 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants