Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Data Catalog createEntry quickstart samples and tests. #1731

Conversation

mesmacosta
Copy link
Contributor

I would like to add samples for Data Catalog create_entry method, released on 0.28.0-alpha google-cloud-datacatalog version.

Since we already have samples for createEntry and createEntryGroup, I would like to submit this PR as an integrated version, which would be used here:
https://cloud.google.com/data-catalog/docs/how-to/filesets#data-catalog-fileset-java

@mesmacosta mesmacosta requested a review from a team November 13, 2019 18:58
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 13, 2019
@mesmacosta
Copy link
Contributor Author

@tswast

@lesv lesv removed their assignment Nov 14, 2019

// Store names for clean up on teardown
String expectedEntryGroupName =
EntryGroupName.of(PROJECT_ID, LOCATION, entryGroupId).toString();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: I think these helpers are being deprecated. At least, they were deprecated in Python (googleapis/gapic-generator#3012) and the same reasoning applies to the Java generator (it's not possible with new proto annotations for configuring the generator).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know the recommended way?
I see two alternatives...

  1. Use EntryGroupName.Builder....setProject(project).setLocation(location).setEntryGroup(entryGroup).build()
  2. Build the string manually

Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least in Python (2) is recommended, as the resource builders are going away there. I haven't seen any discussion about it outside of Python, though.

I don't know why, as the same configuration changes affect all languages.

Copy link
Contributor Author

@mesmacosta mesmacosta Nov 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, it makes sense going that way if this configuration will affect all languages...
@kurtisvg Could you point me the recommended way in Java ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, they aren't going away for Java. They are generally preferred to manually formatting the string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tswast WDYT can we keep the helpers?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have doubts that they will actually stay in Java, based on my conversations with Luke on the ACtools team. I'll check with him if the resource helper deprecation is supposed to affect more than just Python.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will wait for your confirmation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can proceed with using the resource builders. I misunderstood the reasoning for removal in Python.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so AFAIK the PR is ready to be merged, @tswast are you able to review it and merge if everything is ok?

@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 4, 2019
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 4, 2019
@tswast
Copy link
Contributor

tswast commented Dec 4, 2019

I just added the label to run our CI. Once that passes, I think this is OK to merge.

@tswast
Copy link
Contributor

tswast commented Dec 4, 2019

Test failures:

******************** TESTING PROJECTS ********************
------------------------------------------------------------
- testing datacatalog/cloud-client
------------------------------------------------------------
[ERROR] Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.966 s <<< FAILURE! - in com.example.datacatalog.CreateEntryTests
[ERROR] testCreateFilesetEntry(com.example.datacatalog.CreateEntryTests)  Time elapsed: 0.55 s  <<< ERROR!
com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: "my_bucket" is an invalid value for CreateEntryRequest.entry.gcs_fileset_spec.file_patterns. Bucket name must contain only lowercase letters, numbers, dashes (-), underscores (_), and dots (.). See https://cloud.google.com/storage/docs/naming for details.
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at com.google.cloud.datacatalog.v1beta1.DataCatalogClient.createEntry(DataCatalogClient.java:1005)
		at com.example.datacatalog.CreateFilesetEntry.createEntry(CreateFilesetEntry.java:108)
		at com.example.datacatalog.CreateEntryTests.testCreateFilesetEntry(CreateEntryTests.java:89)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
		at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
		at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
		at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
		at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
		at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
		at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
		at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
		at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
		at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
		at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
		at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
		at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
		at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
		at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
		at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
		at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
		at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
		at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
		at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
		at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
		at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: "my_bucket" is an invalid value for CreateEntryRequest.entry.gcs_fileset_spec.file_patterns. Bucket name must contain only lowercase letters, numbers, dashes (-), underscores (_), and dots (.). See https://cloud.google.com/storage/docs/naming for details.
	at io.grpc.Status.asRuntimeException(Status.java:533)
	... 24 more

[ERROR] com.example.datacatalog.CreateEntryTests  Time elapsed: 0.555 s  <<< FAILURE!
java.lang.AssertionError: Something went wrong, no entries were generated
	at org.junit.Assert.fail(Assert.java:89)
	at com.example.datacatalog.CreateEntryTests.tearDownClass(CreateEntryTests.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[ERROR] Failures:
[ERROR]   CreateEntryTests.tearDownClass:67 Something went wrong, no entries were generated
[ERROR] Errors:
[ERROR]   CreateEntryTests.testCreateFilesetEntry » InvalidArgument io.grpc.StatusRuntim...
[ERROR] Tests run: 6, Failures: 1, Errors: 1, Skipped: 0
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project datacatalog-google-cloud-samples: There are test failures.
[ERROR]
[ERROR] Please refer to /tmpfs/src/github/java-docs-samples/datacatalog/cloud-client/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

 Testing failed: Maven returned a non-zero exit code.

.setDisplayName("My Fileset")
.setDescription("This fileset consists of ....")
.setGcsFilesetSpec(
GcsFilesetSpec.newBuilder().addFilePatterns("gs://my_bucket/*").build())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to use a real bucket here, such as gs://cloud-samples-data

Copy link
Contributor Author

@mesmacosta mesmacosta Dec 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the docs, underscore should be accepted, I filled a bug to check this, but using a real bucket is better, so made the change, thanks.

@mesmacosta
Copy link
Contributor Author

ping @tswast

@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 10, 2019
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 10, 2019
@tswast tswast merged commit 43764f8 into GoogleCloudPlatform:master Dec 10, 2019
@mesmacosta mesmacosta deleted the task/add-dc-fileset-quickstart-sample branch December 11, 2019 13:12
charlieyu1996 pushed a commit that referenced this pull request Nov 11, 2022
…1731)

* ADD sample for create fileset entry quickstart

* RAN google java format

* CHANGE exception comment

* FIX lint issues

* Split quickstart into a new folder

* Rename files since class name will appear on samples

* Ran java formatter

* revert region tags change

* remove extra space

* change to real bucket name
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants