Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DOM-56006] Retry getting jwt token for reading timeout #141

Merged
merged 5 commits into from
Apr 3, 2024
Merged

[DOM-56006] Retry getting jwt token for reading timeout #141

merged 5 commits into from
Apr 3, 2024

Conversation

ddl-joyce-zhao
Copy link
Contributor

@ddl-joyce-zhao ddl-joyce-zhao commented Mar 28, 2024 •
edited
Loading

Description

Seeing the reading timeout in the nightly system test. It's an exception safe to retry.

Related Issue

https://dominodatalab.atlassian.net/browse/DOM-56006

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Checklist

  • I've read the CONTRIBUTING.md guide.
  • I've updated the code style using make codestyle.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in Google format for all the methods and classes that I used.

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 28, 2024 •
edited
Loading

✅ Result of Pytest Coverage

---------- coverage: platform linux, python 3.8.18-final-0 -----------

Name Stmts Miss Cover
domino_data/init.py 7 2 71%
domino_data/_feature_store/init.py 0 0 100%
domino_data/_feature_store/client.py 41 3 93%
domino_data/_feature_store/exceptions.py 9 0 100%
domino_data/_feature_store/git.py 43 1 98%
domino_data/_feature_store/logging.py 7 0 100%
domino_data/_feature_store/run.py 16 16 0%
domino_data/_feature_store/sync.py 98 9 91%
domino_data/auth.py 91 13 86%
domino_data/configuration_gen.py 232 0 100%
domino_data/data_sources.py 290 24 92%
domino_data/logging.py 10 0 100%
domino_data/meta.py 22 0 100%
domino_data/training_sets/init.py 0 0 100%
domino_data/training_sets/client.py 126 10 92%
domino_data/training_sets/model.py 42 0 100%
domino_data/transfer.py 37 0 100%
domino_data/vectordb.py 23 4 83%
TOTAL 1094 82 93%

~ 67 passed in 15.60s ~

@ddl-joyce-zhao
updated dependencies
b674c41
@ddl-joyce-zhao
Fixed the test failure
3a78c59
@ddl-joyce-zhao
fixed the style error
e547062
@ddl-joyce-zhao
fixed the vulnerability in fastapi
40c737a 
Have to loosen the version of openapi-client to solve pydantic version conflicts.
@ddl-joyce-zhao ddl-joyce-zhao marked this pull request as ready for review April 1, 2024 20:24
ddl-joyce-zhao
ddl-joyce-zhao commented Apr 2, 2024
View reviewed changes
pyproject.toml
@@ -79,11 +79,11 @@ types-python-dateutil = "^2.8.19.14"
vcrpy = "^5.0.0"

[tool.poetry.group.featurestore.dependencies]
feast = "^0.35.0"
feast = ">=0.36.0"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upgrade to the latest to fix the fastapi vulnerabiliy

ddl-joyce-zhao
ddl-joyce-zhao commented Apr 2, 2024
View reviewed changes
pyproject.toml
@@ -62,7 +62,7 @@ grpcio = "^1.56.2"
isort = {extras = ["colors"], version = "^5.12.0"}
mypy = "^1.4.1"
mypy-extensions = "^1.0.0"
openapi-python-client = "^0.11.6"
openapi-python-client = ">=0.11.6"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To solve the pydantic version conflicts with feast

ddl-joyce-zhao
ddl-joyce-zhao commented Apr 2, 2024
View reviewed changes
pyproject.toml
GitPython = "^3.1.41"

[tool.poetry.group.vectordbs.dependencies]
pinecone-client = ">=2.2.4"
pinecone-client = "^2.2.4"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DominoPineconeConfiguration is extended from OpenAPIConfiguration from pinecone client 2.x

@ddl-joyce-zhao ddl-joyce-zhao requested a review from a team April 2, 2024 15:43
ddl-joyce-zhao
ddl-joyce-zhao commented Apr 2, 2024
View reviewed changes
domino_data/auth.py
@@ -13,6 +13,7 @@


@backoff.on_exception(backoff.expo, httpx.HTTPStatusError, max_time=2)
@backoff.on_exception(backoff.expo, httpx.ReadTimeout, max_tries=2)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is for mitigating the read timeout exception when getting the jwt token.
All the other changes are for fixing the build failures caused by recent vulnerability.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should be able to use a tuple of exceptions instead of having two wrappers.

@backoff.on_exception(backoff.expo, (httpx.HTTPStatusError, httpx.ReadTimeout), max_time=2)
def ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that idea too. The two exception fail in different pace. HTTPStatusError fails fast, for ReadTimeout exception, it normally takes 5 seconds to fail.

@ddl-joyce-zhao ddl-joyce-zhao merged commit cecd0ac into main Apr 3, 2024
7 checks passed
@ddl-joyce-zhao ddl-joyce-zhao deleted the ddl-joyce-zhao.DOM-56006.retry-get-jwt-token branch April 3, 2024 13:08
@ddl-mcetin ddl-mcetin changed the title Retry getting jwt token for reading timeout [DOM-56006] Retry getting jwt token for reading timeout Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ddl-gabrielhaim ddl-gabrielhaim ddl-gabrielhaim approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ddl-joyce-zhao @ddl-gabrielhaim