Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci (ingest): elasticsearch test can fail due to timeout #1609

Closed
badGarnet opened this issue Oct 2, 2023 · 0 comments
Closed

ci (ingest): elasticsearch test can fail due to timeout #1609

badGarnet opened this issue Oct 2, 2023 · 0 comments
Assignees
@rbiseck3
Labels
github_actions Pull requests that update GitHub Actions code

Comments

@badGarnet
Copy link
Collaborator

Recently there is a spike in ingest elasticsearch test failure due to operation timeout, e.g., https://github.com/Unstructured-IO/unstructured/actions/runs/6372547014/job/17296165138 even after we have started to allow for retries.
Exploration suggest that add more timeout allowance and more retry allowance can mitigate this problem, which suggests the root issue is CI resource being tight.

We should profile the ingest CI to track resource usage for memory, CPU time, and storage (since a lot of test also creates local files) and investigate where the hotspots are. Specifically for this ticket right before elasticsearch test starts and after es host is created. If the root issue is indeed resource being too small we should followup to investigate if the problem is tests not releasing resource properly or the demand of a single ingest test can be larger than the CI instance allocation.
@badGarnet badGarnet added github_actions Pull requests that update GitHub Actions code ingest labels Oct 2, 2023
@rbiseck3 rbiseck3 self-assigned this Oct 2, 2023
@rbiseck3 rbiseck3 mentioned this issue Oct 2, 2023
Merged
@rbiseck3 rbiseck3 moved this to In Review in unstructured-ingest Oct 2, 2023
rbiseck3 added a commit that referenced this issue Oct 3, 2023
@rbiseck3
roman/es ingest test fixes (#1610)
b2e9976 
### Description
update elasticsearch docker setup to use docker-compose

Would close out
#1609
@rbiseck3 rbiseck3 moved this from In Review to Done in unstructured-ingest Oct 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rbiseck3 rbiseck3

Labels
github_actions Pull requests that update GitHub Actions code
Projects
unstructured-ingest
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@badGarnet @rbiseck3