feat: Introducing Tracing with OpenTelemetry API #1537 (#1576)

* feat: Adding TraceUtil interface and its implementation to enable Tracing controls via DatastoreOptions (#1431) * Adding EnabledTraceUtil, DisabledTraceUtil and TraceUtilTest * Annotating DatastoreOpenTelemetryOptions to be transient as they're not serializable * Adding google-auth-library-credentials dependency due to https://github.com/googleapis/java-datastore/actions/runs/8944472794/job/24571458116?pr=1431 * feat: Adding Lookup RPC OpenTelemetry Tracing (#1437) * feat: Adding Lookup RPC OpenTelemetry Tracing - Removed OpenCensus Tracing - Added E2E tests with Global and Local OTel SDK - Moved OTel SDK setup to RemoteDatastoreHelper - Fixed pom to depend on BOM for all shared dependencies * feat: Adding Commit RPC Trace Instrumentation (#1440) - Added end-to-end test for Datastore operationsput, add, update and delete. - Updated E2E Test to use the namespace correctly for efficient clean-up of test data * feat: RunQuery trace instrumentation (#1441) * feat: RunQuery trace instrumentation * feat: RunAggregationQuery instrumentation (#1447) * feat: RunQuery trace instrumentation * Formatting * Formatting * Refactor: s/RUNQUERY/RUN_QUERY * feat: RunAggregationQuery Trace Instrumentation * Build: retiring test assertions for OpenCensus spans - will be replacing this in hermetic integration tests for OpenTelemetry using in-memory span exports (in addition to ITE2ETraceTest.java). * Formatting * Fixing @test annotation missed after merge * Formatting * feat: RunQuery trace instrumentation * Formatting * Formatting * Refactor: s/RUNQUERY/RUN_QUERY * feat: RunAggregationQuery Trace Instrumentation * Build: retiring test assertions for OpenCensus spans - will be replacing this in hermetic integration tests for OpenTelemetry using in-memory span exports (in addition to ITE2ETraceTest.java). * Formatting * Fixing @test annotation missed after merge * Formatting * feat: Add Transaction tracing test: transactionalLookupTest * test: Transaction test for RunInTransaction - need to fix trace instrumentation for RunIn.. * Adding transaction span names * TransactionLookupTest * feat: support for transactional operations - tested using newTransaction() and runInTransaction() * Revert "feat: support for transactional operations" This reverts commit 10341c0. * feat: support for transactional operations (#1468) * feat: support for transactional operations - tested using newTransaction() and runInTransaction() * feat: Allocateid tracing (#1488) * feat: Adding tracing for AllocateIds RPC * formatting * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * feat: Add tracing for ReserveIds operation (#1490) - added end-to-end test * fix: Fixed Span nesting for `ReadWriteTransactionCallable` by using parent SpanContext instead of just parent Context (#1495) * fix: Fixed the TraceUtil.startSpan method to use `SpanContext` for linking with the parent instead of `Context`. - This fixes the hierarchy of Spans appearing in a transaction under a Run method. - Tested using existing transaction test * Fixed commit reordering and typos * fix: lint errors * fix: Refactored the ReadWriteTransactioncallable.call method to use startSpan idiomatically - TraceUtil.startSpan needs more debugging - return DefaultTracerProvider instance (no-op) when initializing DisabledTraceUtil - this fixes the unit tests in DatastoreTest.testRunInTransactionWithReadWriteOption * feat: Added tracing for Transaction.RunQuery (#1499) * feat: Added span for Transactional RunQuery - tested * fix: lint * fix: patch apply issues * fix: refactor using boolean flag * fix: s/startSpan/startSpanWithParentContext * test: Additional Transaction Testing and cleanup OpenCensus usage (#1505) * test: newTransactionReadWriteTraceTest * fix: test literal * feat: Added tests for transaction cases * fix: Cleanup OpenCensus dead code * fix: updating version from 2.20.1 -> 2.21.0 * fix: reverting version from 2.21.0 -> 2.20.1 * fix: Adding an exception to the clirr-maven-plugin for an internal API parameter change from com.google.cloud.datastore.TraceUtil -> com.google.cloud.datastore.telemetry.TraceUtil * fix: Fixing the differenceType in clirr exception * fix: add an exception for removal of an internal class (com.google.cloud.datastore.TraceUtil) * fix: fixing incomplete difference details for type 7005 * fix: Fixing `to` of the difference to be the entire signature * fix: typo * test: Adding ITTracingTest to verify events and span attributes (whic… (#1514) * test: Adding ITTracingTest to verify events and span attributes (which are not verified in ITE2ETracingTest) due to TraceClient API limitations. - This test uses InMemorySpanExporter to read the generated Otel span data by the test process to verify generated span data as it were before exporting to a backend. None of the span data is exported to a durable backend. - This test is still an E2E test as it requires a project to send RPCs to. * fix: fixing compilation error due to missing pom dependency. * test: Test for AllocateId and ReserveId rpcs * test: Commit/Put/Update/Delete tests * test: Added fixes and test for RunQuery event * test: Additional Transaction tests and AggregationQuery test (#1518) * test: ReadWrite Transaction test * test: Added test for Transactional RunQuery and Transaction Rollback * test: runInTransaction API tracing test - Fixed setting of common span attributes to spans in runInTransaction - Removed some gRPC related channel attributes that are not present in this Datastore version, yet. * fix: Undelete gRPC upgrade docs * fix: Undo merge mistakes * fix: Updating span event strings (#1539) * fix: Fixing user-facing span names in line with go/firestore-client-trace-catalog * fix: updating bom dependency version to fix https://github.com/googleapis/java-datastore/actions/runs/10256441634/job/28375496112?pr=1539 * Fix: typo in test causing integration test failure (#1556) https://btx.cloud.google.com/invocations/c11a2e8b-4494-4ddc-a77e-cf2bcbcf5254/targets/cloud-devrel%2Fclient-libraries%2Fjava%2Fjava-datastore%2Fpresubmit%2Fintegration;config=default/log * fix: opentelemetry-sdk should only be used as a Test Dependency * fix: Update opentelemetry.version - this also fixes the tests failing in https://github.com/googleapis/java-datastore/actions/runs/10891578591/job/30222786908 * fix: Replacing attribute key values w/ constants * fix: opentelemetry.version to fix RequireUpperBoundDeps check https://github.com/googleapis/java-datastore/actions/runs/10892403348/job/30225154043?pr=1576 * fix: Create Span hierarchy using parent Span (#1580) * fix: Replace use of TraceUtil.SpanContext w/ TraceUtil.Context * fix: Fixing how span hierarchy is created across threads - using Span instead of Context * fix: cleaning up startSpan(spanName, parentContext) variant * fix: add TracedReadWriteTransactionCallable to bifurcate tracing enabled/disabled paths for the Transaction callback. - This change implements the idiomatic way to express nested spans as described in https://opentelemetry.io/docs/languages/java/instrumentation/#create-nested-spans * fix: cleanup * fix: cleanup * fix: cleanup * fix: formatting and import refactoring * chore: generate libraries at Thu Sep 19 18:35:54 UTC 2024 --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: cloud-java-bot <[email protected]>
googleapis · Sep 19, 2024 · 5440c22 · 5440c22
1 parent 224b137
commit 5440c22
Show file tree

Hide file tree

Showing 22 changed files with 3,216 additions and 256 deletions.
diff --git a/README.md b/README.md
@@ -42,7 +42,7 @@ If you are using Maven without the BOM, add this to your dependencies:
 <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-datastore</artifactId>
-  <version>2.21.2</version>
+  <version>2.21.3</version>
 </dependency>
 
 ```

diff --git a/google-cloud-datastore/clirr-ignored-differences.xml b/google-cloud-datastore/clirr-ignored-differences.xml
@@ -55,4 +55,16 @@
         <method>java.lang.Object execute(com.google.cloud.datastore.Query, com.google.cloud.datastore.ReadOption[])</method>
         <differenceType>7004</differenceType>
     </difference>
+    <difference>
+        <className>com/google/cloud/datastore/RetryAndTraceDatastoreRpcDecorator</className>
+        <method>RetryAndTraceDatastoreRpcDecorator(com.google.cloud.datastore.spi.v1.DatastoreRpc, com.google.cloud.datastore.TraceUtil, com.google.api.gax.retrying.RetrySettings, com.google.cloud.datastore.DatastoreOptions)</method>
+        <to>RetryAndTraceDatastoreRpcDecorator(com.google.cloud.datastore.spi.v1.DatastoreRpc, com.google.cloud.datastore.telemetry.TraceUtil, com.google.api.gax.retrying.RetrySettings, com.google.cloud.datastore.DatastoreOptions)</to>
+        <differenceType>7005</differenceType>
+    </difference>
+
+    <!-- Class removed -->
+    <difference>
+        <className>com/google/cloud/datastore/TraceUtil</className>
+        <differenceType>8001</differenceType>
+    </difference>
 </differences>
diff --git a/google-cloud-datastore/pom.xml b/google-cloud-datastore/pom.xml
@@ -16,7 +16,20 @@
   </parent>
   <properties>
     <site.installationModule>google-cloud-datastore</site.installationModule>
+    <opentelemetry.version>1.42.0</opentelemetry.version>
   </properties>
+  <dependencyManagement>
+    <dependencies>
+      <dependency>
+        <!-- Artifacts from google-cloud-java monorepo -->
+        <groupId>com.google.cloud</groupId>
+        <artifactId>gapic-libraries-bom</artifactId>
+        <version>1.37.0</version>
+        <type>pom</type>
+        <scope>import</scope>
+      </dependency>
+    </dependencies>
+  </dependencyManagement>
   <dependencies>
     <dependency>
       <groupId>com.google.api.grpc</groupId>
@@ -38,6 +51,10 @@
       <groupId>com.google.cloud.datastore</groupId>
       <artifactId>datastore-v1-proto-client</artifactId>
     </dependency>
+    <dependency>
+      <groupId>com.google.auth</groupId>
+        <artifactId>google-auth-library-credentials</artifactId>
+    </dependency>
     <dependency>
       <groupId>io.grpc</groupId>
       <artifactId>grpc-api</artifactId>
@@ -111,14 +128,26 @@
       <artifactId>jsr305</artifactId>
     </dependency>
 
+    <!-- OpenTelemetry -->
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-api</artifactId>
+      <version>${opentelemetry.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-context</artifactId>
+      <version>${opentelemetry.version}</version>
+    </dependency>
+    <!-- END OpenTelemetry -->
+
     <!-- Test dependencies -->
     <dependency>
       <groupId>${project.groupId}</groupId>
       <artifactId>google-cloud-core</artifactId>
       <type>test-jar</type>
       <scope>test</scope>
     </dependency>
-
     <dependency>
       <groupId>com.google.guava</groupId>
       <artifactId>guava-testlib</artifactId>
@@ -160,6 +189,61 @@
       <version>1.4.4</version>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>com.google.testparameterinjector</groupId>
+      <artifactId>test-parameter-injector</artifactId>
+      <version>1.16</version>
+      <scope>test</scope>
+    </dependency>
+    <!-- OpenTelemetry -->
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-sdk</artifactId>
+      <version>${opentelemetry.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-sdk-common</artifactId>
+      <version>${opentelemetry.version}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-sdk-testing</artifactId>
+      <version>${opentelemetry.version}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-sdk-trace</artifactId>
+      <version>${opentelemetry.version}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-semconv</artifactId>
+      <version>1.1.0-alpha</version>
+      <scope>test</scope>
+    </dependency>
+    <!-- END OpenTelemetry -->
+    <!-- Cloud Ops -->
+    <dependency>
+      <groupId>com.google.cloud.opentelemetry</groupId>
+      <artifactId>exporter-trace</artifactId>
+      <version>0.15.0</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>com.google.cloud</groupId>
+      <artifactId>google-cloud-trace</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>com.google.api.grpc</groupId>
+      <artifactId>proto-google-cloud-trace-v1</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <!-- END Cloud Ops -->
   </dependencies>
 
   <build>