Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Companion+client stability fixes, error handling and retry #4734

Merged
merged 30 commits into from
Nov 2, 2023
Merged

Conversation

mifi
Copy link
Contributor

@mifi mifi commented Oct 13, 2023

  • respond with 500 if unhandled upload error
    400 seems inappropriate
  • fix companion download error
    • respond with correct response code when calling /get and request to the upstream server fails
    • also add a unit test for this
  • forward 429 from provider
    allows us to retry it
  • add a way to test refresh tokens
  • fix race condtiion with refreshing token that would cause many refresh-token calls to be made concurrently, possibly causing problems
  • implement retry and fix socket
    • Make sure we don't instantiate a new Provider for every file uploaded (reuse the original provider instead) - this caused the refresh token synchronization not to work
    • Retry most errors when uploading (/get) using companion, but for HTTP, only retry certain HTTP status codes
    • Reimplement the websocket code to be more stable and retry as well
    • if no websocket messages have been received in 5 minutes, fail the upload, so the upload doesn't get stuck forever in case something goes bad, TODO should this should be solved in the UI instead?
  • remove useFastRemoteRetry
    • because it isn't supported in Companion as far as I can tell, and the option effectively doesn't do anything at all

respond with correct response code when calling /get
and request to the upstream server fail

also add a unit test for this
allows us to retry it
- Make sure we don't instantiate a new Provider for every file uploaded (reuse the original provider instead) - this caused the refresh token synchronization not to work
- Retry most errors when uploading (`/get`) using companion, but for HTTP, only retry certain HTTP status codes
- Reimplement the websocket code to be more stable and retry as well
because it isn't supported in Companion as far as I can tell
@socket-security
Copy link

New dependencies detected. Learn more about Socket for GitHub ↗︎

Packages Version New capabilities Transitives Size Publisher
p-retry 6.1.0 None +2 24.3 kB sindresorhus

Copy link
Member

@Murderlon Murderlon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall very impressive!

packages/@uppy/companion-client/src/Provider.js Outdated Show resolved Hide resolved
})
if (!skipPostResponse) this.onReceiveResponse(response)
return handleJSONResponse(response)
return await pRetry(async () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a head up that is will be need to be refactored again in #4745, as we now have a util for fetching and retrying.

packages/@uppy/companion-client/src/RequestClient.js Outdated Show resolved Hide resolved
@Murderlon
Copy link
Member

CI is also failing and should be looked at

- rewrite so we can use p-retry for exponential backoff for socket reconnect too
- improve logging
- simplify/reuse code
@mifi
Copy link
Contributor Author

mifi commented Oct 23, 2023

pushed some improvements. e2e tests are back in business. i'm no longer retrying all http requests (as before, which caused the e2e to fail)

preventing socket.send when the socket is not in the connected state
catch errors in handlers
@mifi
Copy link
Contributor Author

mifi commented Oct 25, 2023

one e2e test flaked: should not create assembly when all individual files have been cancelled

@arturi arturi self-requested a review November 1, 2023 13:33
@mifi mifi merged commit 17826da into main Nov 2, 2023
18 checks passed
@mifi mifi deleted the companion-fixes branch November 2, 2023 12:31
Murderlon added a commit that referenced this pull request Nov 2, 2023
* main:
  Companion+client stability fixes, error handling and retry (#4734)
  add getBucket metadata argument (#4770)
  @uppy/core: simplify types with class generic (#4761)
github-actions bot added a commit that referenced this pull request Nov 8, 2023
| Package                | Version | Package                | Version |
| ---------------------- | ------- | ---------------------- | ------- |
| @uppy/aws-s3           |   3.5.0 | @uppy/provider-views   |   3.7.0 |
| @uppy/aws-s3-multipart |   3.9.0 | @uppy/react            |   3.2.0 |
| @uppy/companion        |  4.11.0 | @uppy/transloadit      |   3.4.0 |
| @uppy/companion-client |   3.6.0 | @uppy/tus              |   3.4.0 |
| @uppy/core             |   3.7.0 | @uppy/url              |   3.4.0 |
| @uppy/dashboard        |   3.7.0 | @uppy/utils            |   5.6.0 |
| @uppy/image-editor     |   2.3.0 | @uppy/xhr-upload       |   3.5.0 |
| @uppy/locales          |   3.4.0 | uppy                   |  3.19.0 |

- @uppy/dashboard: Remove uppy-Dashboard-isFixed when uppy.close() is invoked (Artur Paikin / #4775)
- @uppy/core,@uppy/dashboard: don't cancel all files when clicking "done" (Mikael Finstad / #4771)
- @uppy/utils: refactor to TS (Antoine du Hamel / #4699)
- @uppy/locales: locales: add ca_ES (ordago / #4772)
- @uppy/companion: Companion+client stability fixes, error handling and retry (Mikael Finstad / #4734)
- @uppy/companion: add getBucket metadata argument (Mikael Finstad / #4770)
- @uppy/core: simplify types with class generic (JokcyLou / #4761)
- @uppy/image-editor: More image editor improvements (Evgenia Karunus / #4676)
- @uppy/react: add useUppyState (Merlijn Vos / #4711)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants