Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Allow custom HttpRules for REST LROs #1288

Merged
merged 92 commits into from
Mar 10, 2023
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
4c14e48
fix: Allow custom http bindings for LROs
lqiu96 Jan 31, 2023
8f5e313
fix: Update generator and GAX to support custom HTTP Bindings for ope…
lqiu96 Jan 31, 2023
a693085
fix: Use static Map for custom Operation REST Http Bindings
lqiu96 Jan 31, 2023
f4baa0a
chore: Update golden test cases
lqiu96 Jan 31, 2023
7bde2cf
chore: Update showcase tests
lqiu96 Jan 31, 2023
a3527e1
chore: Update golden ITs
lqiu96 Jan 31, 2023
2e1acbc
chore: Add back origin HttpJsonOperationStub.create() methods
lqiu96 Feb 1, 2023
5f3dfe2
fix(deps): update dependency com.google.auth:google-auth-library-bom …
renovate-bot Jan 31, 2023
d8761bc
doc: Update DEVELOPMENT.md for local development. (#1237)
blakeli0 Feb 1, 2023
d837f97
chore: Create a default mapping in OperationStub
lqiu96 Feb 1, 2023
dfd1e5a
chore: Do not generate custom bindings if there are none
lqiu96 Feb 1, 2023
0a891dc
chore: Update golden units
lqiu96 Feb 1, 2023
51bde57
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Feb 1, 2023
64fc726
chore: Update all test cases
lqiu96 Feb 1, 2023
09031c8
chore: Fix format issues
lqiu96 Feb 1, 2023
568ad44
fix: remove constant operation binding field
lqiu96 Feb 1, 2023
3a72781
chore: Clean up code
lqiu96 Feb 2, 2023
d7d1b12
chore: Resolve sonar comments
lqiu96 Feb 2, 2023
9b050d3
chore: DEVELOPMENT.md formatting fix (#1289)
meltsufin Feb 1, 2023
5b31d41
ci: use java-shared-dependencies in google-cloud-java repository for …
suztomo Feb 1, 2023
316621c
fix(java): initialize netty-shaded at run-time and add reflection con…
mpeddada1 Feb 1, 2023
d8e488f
ci(showcase): disable rest_numeric_enum for showcase testing (#1284)
mpeddada1 Feb 2, 2023
46156e0
chore(main): release 2.15.0 (#1269)
release-please[bot] Feb 2, 2023
8a3ba93
chore(main): release 2.15.1-SNAPSHOT (#1292)
release-please[bot] Feb 2, 2023
1cd51fb
chore: Pin Bazel version to 5.2.0 (#1304)
blakeli0 Feb 8, 2023
ec8ce91
build(deps): bump cryptography from 38.0.3 to 39.0.1 in /.kokoro (#1297)
dependabot[bot] Feb 8, 2023
443adec
chore: (cleanup) removing unused files (#1265)
ddixit14 Feb 8, 2023
c70d183
chore: removing unused Gradle files in api-common-java, gax-java, jav…
suztomo Feb 8, 2023
db34e1d
chore: updated gax-java contribution doc (#1334)
suztomo Feb 8, 2023
5fce328
fix: use pkg_tar from rules_pkg (#1303)
parthea Feb 9, 2023
5af299d
chore: Fix pre-commit. (#1294)
blakeli0 Feb 10, 2023
5f3e732
chore: update CONTRIBUTING.md (#1346)
JoeWang1127 Feb 13, 2023
78c6d89
fix(deps): update dependency io.grpc:grpc-bom to v1.53.0 (#1345)
renovate-bot Feb 13, 2023
472a3ba
chore(deps): update dependency org.apache.maven.plugins:maven-deploy-…
renovate-bot Feb 13, 2023
ebe84cc
chore(deps): update dependency org.apache.maven.plugins:maven-surefir…
renovate-bot Feb 14, 2023
6f8cbe0
chore: add rules_pkg to renovate bot ignoreDeps (#1349)
emmileaf Feb 14, 2023
1ae7726
chore: fix renovate bot ignoreDeps (#1353)
emmileaf Feb 14, 2023
e8d86df
fix(batcher): exceptions in unaryCaller bubble up (#1166)
diegomarquezp Feb 14, 2023
b8afde0
fix(deps): update dependency com.google.auth:google-auth-library-bom …
renovate-bot Feb 15, 2023
445769c
chore(deps): update dependency org.apache.maven.plugins:maven-javadoc…
renovate-bot Feb 15, 2023
9b42094
chore(main): release 2.15.1 (#1339)
release-please[bot] Feb 15, 2023
24dabca
chore: refactoring README and DEVELOPMENT.md (#1351)
suztomo Feb 15, 2023
dc40d5a
chore(main): release 2.15.2-SNAPSHOT (#1358)
release-please[bot] Feb 15, 2023
eeb5665
chore: renovate to group Protobuf artifacts (#1362)
suztomo Feb 15, 2023
40e49da
chore: README.md to explain service_config.proto (#1361)
suztomo Feb 21, 2023
4356783
chore: Telling owlbot to ignore these files & it's a monorepo (#1372)
ddixit14 Feb 22, 2023
9a636bb
fix: Change the default scope of gax from implementation to api in au…
blakeli0 Feb 22, 2023
1b3e292
chore: Update variable name
lqiu96 Feb 2, 2023
a64100f
chore: Fix format issues
lqiu96 Feb 2, 2023
e8fb415
fix: Use HttpRule as Value for Custom Bindings
lqiu96 Feb 22, 2023
09b787c
fix: Use HttpRule as Value for Custom Bindings
lqiu96 Feb 22, 2023
94c00fb
chore: Add comments
lqiu96 Feb 23, 2023
d5453b7
chore: Update tests
lqiu96 Feb 23, 2023
667c3cf
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Feb 23, 2023
e24b42f
fix(deps): update dependency com.google.auth:google-auth-library-bom …
renovate-bot Jan 31, 2023
62edb0e
doc: Update DEVELOPMENT.md for local development. (#1237)
blakeli0 Feb 1, 2023
c2cc0a8
chore: Add tests
lqiu96 Feb 23, 2023
4f9015f
chore: Cleanup files
lqiu96 Feb 23, 2023
c7631ba
chore: Format the files
lqiu96 Feb 23, 2023
674d814
chore: Add NoCredentialsProvider
lqiu96 Feb 23, 2023
52aaa23
chore: Fix sonar comments
lqiu96 Feb 23, 2023
1670957
chore: Add serviceyaml file for parsing for rest showcase tests
lqiu96 Feb 23, 2023
abebe3d
chore: Use service yaml file in test
lqiu96 Feb 23, 2023
0a70fe7
chore: Fix Echo showcase test
lqiu96 Feb 23, 2023
ff142e2
chore: Clean up tests
lqiu96 Feb 23, 2023
3f250ab
chore: Sort the map entry to get a consistent ordering for the test
lqiu96 Feb 24, 2023
71b1244
chore: Update showcase and integration tests
lqiu96 Feb 24, 2023
98c1cfb
chore: Resolve sonar comments
lqiu96 Feb 24, 2023
25acf97
chore: Update comments
lqiu96 Feb 24, 2023
f0a1692
chore: Remove a few public constructors
lqiu96 Feb 24, 2023
85bcf90
chore: test ci
lqiu96 Feb 24, 2023
166f6c4
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Feb 27, 2023
7af9882
chore: Remove the cache for java 8
lqiu96 Feb 27, 2023
a68927e
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Feb 28, 2023
0c4eec3
chore: Update from PR feedback
lqiu96 Feb 28, 2023
da232d2
chore: Update comments
lqiu96 Feb 28, 2023
1f40c11
chore: Fix sonar issue
lqiu96 Feb 28, 2023
88f7961
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Feb 28, 2023
8f7aed0
chore: Fix comment
lqiu96 Feb 28, 2023
31932b1
chore: Update to have multiple additional_bindings
lqiu96 Mar 7, 2023
88487e3
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Mar 7, 2023
4f6e39c
chore: Update grpcrest golden test to include httprule
lqiu96 Mar 8, 2023
8d4d54c
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Mar 9, 2023
9a4f437
chore: HttpJsonOperationsStub's MethodDescriptors are not static
lqiu96 Mar 9, 2023
350b811
chore: Add unit tests for HttpJson Operations logic
lqiu96 Mar 9, 2023
bfe3753
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Mar 9, 2023
1d5d967
chore: Resolve lint issues
lqiu96 Mar 9, 2023
fe6c4c7
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Mar 9, 2023
c15e2b5
chore: Clean up test code
lqiu96 Mar 9, 2023
2c489ff
chore: Resolve pr comments
lqiu96 Mar 10, 2023
78be800
chore: Add VisibleForTesting annotation
lqiu96 Mar 10, 2023
f179078
Merge branch 'main' into main-fix_custom_LRO_httpbindings
lqiu96 Mar 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@ public GapicClass generate(GapicContext context, Service service) {
Map<String, Message> messageTypes = context.messages();
List<Statement> classStatements =
createClassStatements(
context,
service,
protoMethodNameToDescriptorVarExprs,
callableClassMemberVarExprs,
Expand Down Expand Up @@ -278,6 +279,7 @@ protected List<Statement> createTypeRegistry(Service service) {
}

protected List<Statement> createClassStatements(
GapicContext context,
Service service,
Map<String, VariableExpr> protoMethodNameToDescriptorVarExprs,
Map<String, VariableExpr> callableClassMemberVarExprs,
Expand All @@ -298,6 +300,9 @@ protected List<Statement> createClassStatements(
classStatements.add(EMPTY_LINE_STATEMENT);
}

classStatements.addAll(createOperationCustomHttpBindingsMapDeclaration(context));
classStatements.add(EMPTY_LINE_STATEMENT);

classStatements.addAll(createClassMemberFieldDeclarations(callableClassMemberVarExprs));
classStatements.add(EMPTY_LINE_STATEMENT);

Expand All @@ -307,6 +312,10 @@ protected List<Statement> createClassStatements(
return classStatements;
}

protected List<Statement> createOperationCustomHttpBindingsMapDeclaration(GapicContext context) {
return new ArrayList<>();
}

protected List<Statement> createMethodDescriptorVariableDecls(
Service service,
Map<String, VariableExpr> protoMethodNameToDescriptorVarExprs,
Expand Down Expand Up @@ -649,6 +658,7 @@ protected List<MethodDefinition> createConstructorMethods(
if (generateOperationsStubLogic(service)) {
secondCtorExprs.addAll(
createOperationsStubInitExpr(
context,
service,
thisExpr,
operationsStubClassVarExpr,
Expand Down Expand Up @@ -758,6 +768,7 @@ protected List<MethodDefinition> createConstructorMethods(
}

protected List<Expr> createOperationsStubInitExpr(
GapicContext context,
Service service,
Expr thisExpr,
VariableExpr operationsStubClassVarExpr,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

package com.google.api.generator.gapic.composer.rest;

import com.google.api.HttpRule;
import com.google.api.core.InternalApi;
import com.google.api.gax.httpjson.ApiMethodDescriptor;
import com.google.api.gax.httpjson.ApiMethodDescriptor.MethodType;
Expand Down Expand Up @@ -63,6 +64,7 @@
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.BiMap;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.protobuf.TypeRegistry;
import java.util.ArrayList;
import java.util.Arrays;
Expand Down Expand Up @@ -90,6 +92,29 @@ public class HttpJsonServiceStubClassComposer extends AbstractTransportServiceSt
.build())
.build();

private static final VariableExpr OPERATION_CUSTOM_HTTP_BINDINGS =
VariableExpr.builder()
.setVariable(
Variable.builder()
.setName("operationCustomHttpBindings")
.setType(FIXED_REST_TYPESTORE.get(ImmutableMap.class.getSimpleName()))
.build())
.build();

private static final String LRO_LIST_OPERATIONS = "google.longrunning.Operations.ListOperations";
private static final String LRO_GET_OPERATION = "google.longrunning.Operations.GetOperation";
private static final String LRO_DELETE_OPERATION =
"google.longrunning.Operations.DeleteOperation";
private static final String LRO_CANCEL_OPERATION =
"google.longrunning.Operations.CancelOperation";

private static final Map<String, Function<HttpRule, String>> HTTP_RULE_OPERATION = ImmutableMap.<String, Function<HttpRule, String>>builder()
.put(LRO_LIST_OPERATIONS, HttpRule::getGet)
.put(LRO_GET_OPERATION, HttpRule::getGet)
.put(LRO_DELETE_OPERATION, HttpRule::getDelete)
.put(LRO_CANCEL_OPERATION, HttpRule::getPost).build();
private static final String LRO_NAME_PREFIX = "google.longrunning.Operations";

protected HttpJsonServiceStubClassComposer() {
super(RestContext.instance());
}
Expand All @@ -110,6 +135,7 @@ private static TypeStore createStaticTypes() {
HttpJsonOperationSnapshot.class,
HttpJsonStubCallableFactory.class,
Map.class,
ImmutableMap.class,
ProtoMessageRequestFormatter.class,
ProtoMessageResponseParser.class,
ProtoRestSerializer.class,
Expand All @@ -121,6 +147,65 @@ protected boolean generateOperationsStubLogic(Service service) {
return service.hasLroMethods();
}

@Override
protected List<Statement> createOperationCustomHttpBindingsMapDeclaration(GapicContext context) {
VariableExpr operationCustomHttpBindingsVarExpr =
VariableExpr.withVariable(
Variable.builder()
.setType(
TypeNode.withReference(
ConcreteReference.builder()
.setClazz(Map.class)
.setGenerics(
Arrays.asList(
ConcreteReference.withClazz(String.class),
ConcreteReference.withClazz(String.class)))
.build()))
.setName("operationCustomHttpBindings")
.build())
.toBuilder()
.setIsDecl(true)
.setIsStatic(true)
.setScope(ScopeNode.PRIVATE)
.setIsFinal(true)
.build();

Expr operationCustomHttpBindingsBuilderExpr =
MethodInvocationExpr.builder()
.setStaticReferenceType(FIXED_REST_TYPESTORE.get(ImmutableMap.class.getSimpleName()))
.setMethodName("builder")
.setGenerics(Arrays.asList(TypeNode.STRING.reference(), TypeNode.STRING.reference()))
.build();

for (Map.Entry<String, String> entrySet : parseCustomHttpBindings(context).entrySet()) {
String selector = entrySet.getKey();
String path = entrySet.getValue();
operationCustomHttpBindingsBuilderExpr =
MethodInvocationExpr.builder()
.setExprReferenceExpr(operationCustomHttpBindingsBuilderExpr)
.setMethodName("put")
.setArguments(
Arrays.asList(
ValueExpr.withValue(StringObjectValue.withValue(selector)),
ValueExpr.withValue(StringObjectValue.withValue(path))))
.build();
}

operationCustomHttpBindingsBuilderExpr =
MethodInvocationExpr.builder()
.setExprReferenceExpr(operationCustomHttpBindingsBuilderExpr)
.setMethodName("build")
.setReturnType(FIXED_REST_TYPESTORE.get(ImmutableMap.class.getSimpleName()))
.build();

return Collections.singletonList(
ExprStatement.withExpr(
AssignmentExpr.builder()
.setVariableExpr(operationCustomHttpBindingsVarExpr)
.setValueExpr(operationCustomHttpBindingsBuilderExpr)
.build()));
}

@Override
protected Statement createMethodDescriptorVariableDecl(
Service service,
Expand Down Expand Up @@ -1075,6 +1160,7 @@ private List<Expr> getMethodTypeExpr(Method protoMethod) {

@Override
protected List<Expr> createOperationsStubInitExpr(
GapicContext context,
Service service,
Expr thisExpr,
VariableExpr operationsStubClassVarExpr,
Expand All @@ -1088,6 +1174,7 @@ protected List<Expr> createOperationsStubInitExpr(
if (standardOpStub.equals(operationsStubType.reference().fullName())) {
arguments.add(TYPE_REGISTRY_VAR_EXPR);
}
arguments.add(OPERATION_CUSTOM_HTTP_BINDINGS);

return Collections.singletonList(
AssignmentExpr.builder()
Expand All @@ -1103,6 +1190,20 @@ protected List<Expr> createOperationsStubInitExpr(
.build());
}

private Map<String, String> parseCustomHttpBindings(GapicContext context) {
Map<String, String> customHttpBindings = new HashMap<>();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add all the default operation -> url mapping to this map and override then if needed. Basically we need to move as much code as possible to the generator from gax, as gax is a runtime dependency and it would be much harder to make changes there later.

Copy link
Contributor Author

@lqiu96 lqiu96 Feb 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add the defaults for bunch of mixins (or for only Operation module). I just saw that it is possible to have way more mixins (https://github.com/googleapis/gapic-showcase/blob/b94ecfc51059b49770e5bdb6f0d7ea07903158e8/schema/google/showcase/v1beta1/showcase_v1beta1.yaml#L46-L85) so I didn't want to just create a default list in the gapic-generator that needed to be manually updated later on.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I think that makes sense, we could do it the other way also, construct a map with all the bindings from the yaml, and populate default for the LRO operations. Or if we are confident about the list of mixins, we could pre-populate them as well, a related question, did you get a chance to see how mixins work for location and iam? Are we going to have similar problem for them?

Copy link
Contributor Author

@lqiu96 lqiu96 Feb 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can see, I think only the Operation proto is currently being used. I would imagine if Location and IAM are used, they would be created the same way the Operations Client was in GAX: googleapis/gax-java#1456 and we would reference it via https://github.com/googleapis/google-cloud-java/blob/10bb0cb494f64b864408ede46834e1046351370c/java-speech/google-cloud-speech/src/main/java/com/google/cloud/speech/v1/SpeechClient.java#L164-L166.

I was running under the assumption that they would be added in sometime in the future, but I'm not sure about it. Plus each generated client (operation/ iam/ location) would have the initial defaults set from the proto file: https://github.com/googleapis/gapic-generator-java/blob/d1a16195937df041f65a52717ddf9dc6ceb09b4f/gax-java/gax-httpjson/src/main/java/com/google/api/gax/httpjson/longrunning/stub/HttpJsonOperationsStub.java#L84

com.google.api.Service service = context.serviceYamlProto();
if (service != null && service.getHttp() != null) {
for (HttpRule httpRule : service.getHttp().getRulesList()) {
String selector = httpRule.getSelector();
if (selector.contains(LRO_NAME_PREFIX)) {
customHttpBindings.put(selector, HTTP_RULE_OPERATION.get(selector).apply(httpRule));
}
}
}
return customHttpBindings;
}

@Override
protected List<Statement> createLongRunningClient(Service service, TypeStore typeStore) {
Method pollingMethod = service.operationPollingMethod();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import com.google.api.gax.rpc.ClientContext;
import com.google.api.gax.rpc.OperationCallable;
import com.google.api.gax.rpc.ServerStreamingCallable;
import com.google.api.gax.rpc.UnaryCallable;
import com.google.common.collect.ImmutableMap;
import com.google.longrunning.Operation;
import com.google.protobuf.TypeRegistry;
import com.google.showcase.grpcrest.v1beta1.BlockRequest;
Expand Down Expand Up @@ -330,6 +331,9 @@ public class HttpJsonEchoStub extends EchoStub {
.build())
.build();

private static final Map<String, String> operationCustomHttpBindings =
ImmutableMap.<String, String>builder().build();

private final UnaryCallable<EchoRequest, EchoResponse> echoCallable;
private final ServerStreamingCallable<ExpandRequest, EchoResponse> expandCallable;
private final UnaryCallable<PagedExpandRequest, PagedExpandResponse> pagedExpandCallable;
Expand Down Expand Up @@ -382,7 +386,8 @@ public class HttpJsonEchoStub extends EchoStub {
throws IOException {
this.callableFactory = callableFactory;
this.httpJsonOperationsStub =
HttpJsonOperationsStub.create(clientContext, callableFactory, typeRegistry);
HttpJsonOperationsStub.create(
clientContext, callableFactory, typeRegistry, operationCustomHttpBindings);

HttpJsonCallSettings<EchoRequest, EchoResponse> echoTransportSettings =
HttpJsonCallSettings.<EchoRequest, EchoResponse>newBuilder()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import com.google.api.gax.httpjson.ProtoMessageResponseParser;
import com.google.api.gax.httpjson.ProtoRestSerializer;
import com.google.api.gax.rpc.ClientContext;
import com.google.api.gax.rpc.UnaryCallable;
import com.google.common.collect.ImmutableMap;
import com.google.protobuf.TypeRegistry;
import com.google.showcase.v1beta1.EnumRequest;
import com.google.showcase.v1beta1.EnumResponse;
Expand Down Expand Up @@ -350,6 +351,9 @@ public class HttpJsonComplianceStub extends ComplianceStub {
.build())
.build();

private static final Map<String, String> operationCustomHttpBindings =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't want to create unused variables if we can avoid it in the generator.

ImmutableMap.<String, String>builder().build();

private final UnaryCallable<RepeatRequest, RepeatResponse> repeatDataBodyCallable;
private final UnaryCallable<RepeatRequest, RepeatResponse> repeatDataBodyInfoCallable;
private final UnaryCallable<RepeatRequest, RepeatResponse> repeatDataQueryCallable;
Expand Down
Loading