Skip to content
This repository has been archived by the owner on Nov 8, 2024. It is now read-only.

refactor: customize HTTP transport for GCS client #4

Merged
merged 1 commit into from
Jul 13, 2018

Conversation

nblair
Copy link
Contributor

@nblair nblair commented Jun 28, 2018

This change set is the result of some investigation on the poor throughput issue (see #1).

It includes the following:

  • Upgrades google-cloud-storage to latest release
  • Setup the ApacheHttpTransport to use a pooling client connection manager. This addresses an issue observed with the default connection manager; after a period of inactivity, the client would fail to connect to the google API endpoint and not retry. This implementation does successfully retry when connections are severed. This change comes with a caveat in that it requires us to use deprecated HTTP Client constructors instead of their current equivalents.
  • Improve the performance of hard delete by skipping an unnecessary prior GET.
  • Lean down object read requests to only retrieve the MEDIA_LINK field (at this time all other GCS object attributes aren't used by this implementation).

This change set is the result of some investigation on the poor throughput issue (see #1).

It includes the following:

* Upgrades google-cloud-storage to latest release
* Setup the ApacheHttpTransport to use a pooling client connection manager. This addresses an issue observed with the default connection manager; after a period of inactivity, the client would fail to connect to the google API endpoint and not retry. This implementation does successfully retry when connections are severed. This change comes with a caveat in that it requires us to use deprecated HTTP Client constructors instead of their current equivalents.
* Improve the performance of hard delete by skipping an unnecessary prior GET.
* Lean down object read requests to only retrieve the MEDIA_LINK field (at this time all other GCS object attributes aren't used by this implementation).
@nblair
Copy link
Contributor Author

nblair commented Jun 28, 2018

I'll note here too that I don't believe this conclusively resolves #1, but casual observations show we are headed in the right direction. This project would benefit from some tooling to automate performance testing in a full NXRM deployment in Google Cloud.

@nblair
Copy link
Contributor Author

nblair commented Jul 12, 2018

googleapis/google-http-java-client#364 describes the issue necessitating the use of deprecated constructors - if we use the current means for constructing our own HttpClient to pass into the ApacheHttpTransport constructor, we get the same UnsupportedOperationException .

@nblair nblair merged commit 0915419 into master Jul 13, 2018
@nblair nblair deleted the investigate-storage-integration branch July 13, 2018 18:15
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant