Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid Java Temp Files in task creation #6178

Merged
merged 6 commits into from
May 3, 2022
Merged

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Apr 28, 2022

  • During task creation, we no longer use java’s own temporary file code (see reasons in linked issue below), but instead use a custom temporary file service.
  • The initial data requests are no longer sent in parallel to the tracingstore but instead sequentially, to avoid overloading the tracingstore (new function Fox.serialSequenceBox)

URL of deployed dev instance (used for testing):

Steps to test:

  • Create volume or hybrid task type
  • try to create (hundreds of) tasks with initial volume data
  • should not fail

Issues:


@fm3 fm3 self-assigned this Apr 28, 2022
@fm3 fm3 changed the title Avoid Java Temp Files in task creation [WIP] Avoid Java Temp Files in task creation Apr 28, 2022
@fm3 fm3 changed the title [WIP] Avoid Java Temp Files in task creation Avoid Java Temp Files in task creation May 2, 2022
@fm3 fm3 marked this pull request as ready for review May 2, 2022 12:55
@fm3 fm3 requested a review from jstriebel May 2, 2022 19:17
Copy link
Contributor

@jstriebel jstriebel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Props for digging into this, I think the solution is good. Maybe we should also add a comment about the bug in the code (e.g. in TempFileService)? See my two comments below, besides this LGTM 👍

app/models/task/TaskCreationService.scala Outdated Show resolved Hide resolved
app/oxalis/files/TempFileService.scala Show resolved Hide resolved
@fm3
Copy link
Member Author

fm3 commented May 3, 2022

Thanks for your feedback! I implemented the suggestions and also noticed another bug, where slashes in the prefix would lead to wrong file paths (so I am removing those now). Could you have another look?

@fm3 fm3 requested a review from jstriebel May 3, 2022 10:52
Copy link
Contributor

@jstriebel jstriebel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@fm3 fm3 enabled auto-merge (squash) May 3, 2022 10:53
@fm3 fm3 merged commit 7a953f9 into master May 3, 2022
@fm3 fm3 deleted the task-creation-no-temp-files branch May 3, 2022 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Server Errors when sending temporary files (JDK regression?)
2 participants