Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document backoff semantics for create_profile #1498

Open
bitglue opened this issue Nov 12, 2020 · 2 comments
Open

Document backoff semantics for create_profile #1498

bitglue opened this issue Nov 12, 2020 · 2 comments
Labels
api: cloudprofiler Issues related to the Cloud Profiler API. type: question Request for information or clarification. Not an issue.

Comments

@bitglue
Copy link

bitglue commented Nov 12, 2020

Is your feature request related to a problem? Please describe.

I'm trying to implement an agent for Cloud Profiler, but I'm not sure how it should behave. According to the documentation, the server can request the client to back off through some GRPC metadata:

# CreateProfile creates a new profile resource in the online mode. The server
# ensures that the new profiles are created at a constant rate per deployment,
# so the creation request may hang for some time until the next profile session
# is available. The request may fail with ABORTED error if the creation is not
# available within ~1m, the response will indicate the duration of the backoff
# the client should take before attempting creating a profile again. The backoff
# duration is returned in google.rpc.RetryInfo extension on the response status.
# To a gRPC client, the extension will be return as a binary-serialized proto in
# the trailing metadata item named "google.rpc.retryinfo-bin".

However, I can't find anything that specifies how this maps to the JSON API.

Sometimes, I've received an HTTP 409 response with an error like "generic::aborted: action throttled, backoff for 44m0s". Maybe this is what the documentation is talking about? I'm not sure. Parsing the duration out of a string doesn't seem ideal. It's difficult to explore, because it's difficult to solicit this response.

Oddly, the documentation says this should happen in about 1 minute, but when I've had it happen, it's taken more like 230 seconds. This is longer than the default 60 second timeout on this method.

Furthermore, I don't always get the 409 response even with a longer timeout. I've tried increasing the timeout to as much as 1 hour, and usually I'll get <Google::Apis::TransmissionError: execution expired> at the end of that hour, instead of the HTTP 409.

I definitely want to respect the prescribed client behavior here, because the alternative is getting rate limited and that's no good.

Not sure if there's already logic in this library that's handling the retries or what.

Describe the solution you'd like

Document what the backoff response looks like over the JSON HTTP and the interface exposed by google-api-ruby-client.

Describe alternatives you've considered

I'd use an agent implemented by Google, but that doesn't seem to be going anywhere: googleapis/google-cloud-ruby#2116

I'd also use the GRPC API directly if it was easy, but it seems like https://github.com/googleapis/google-cloud-ruby would be the place that would happen, and there doesn't seem to be any support even for the raw API in there.

Additional context

@quartzmo quartzmo added api: cloudprofiler Issues related to the Cloud Profiler API. type: question Request for information or clarification. Not an issue. labels Nov 12, 2020
@quartzmo
Copy link
Member

I'm not sure if this helps, but here is where retry with backoff is implemented for the generated gRPC clients in google-cloud-ruby:

https://github.com/googleapis/gapic-generator-ruby/blob/master/gapic-common/lib/gapic/call_options/retry_policy.rb

@fhinkel fhinkel self-assigned this Dec 8, 2020
@fhinkel
Copy link

fhinkel commented Dec 8, 2020

Greetings, we're closing this. Looks like the issue got resolved. Please let us know if the issue needs to be reopened.

@fhinkel fhinkel closed this as completed Dec 8, 2020
@dazuma dazuma reopened this Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: cloudprofiler Issues related to the Cloud Profiler API. type: question Request for information or clarification. Not an issue.
Projects
None yet
Development

No branches or pull requests

4 participants