Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: instrument with traces for observability #2079

Closed
odeke-em opened this issue Jul 13, 2024 · 1 comment · Fixed by #2118
Closed

all: instrument with traces for observability #2079

odeke-em opened this issue Jul 13, 2024 · 1 comment · Fixed by #2118
Labels
api: spanner Issues related to the googleapis/nodejs-spanner API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@odeke-em
Copy link
Contributor

Customers are asking for traces to be added to this package so that they can properly diagnose latency issues and have observability for their systems.

I am tasked with implementing tracing for this library and this issue serves as the reference. I have implemented this functionality in my fork odeke-em#3 and I shall progressively send code changes to this repository to make the software development process much more scalable and easier to merge.

Samples of results

Screenshot 2024-07-14 at 1 52 25 AM

with PII acknowledgement and annotations added
Screenshot 2024-07-14 at 1 52 47 AM

@odeke-em odeke-em added priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. labels Jul 13, 2024
@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/nodejs-spanner API. label Jul 13, 2024
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Jul 14, 2024
This change introduces an OpenTelemetry tracing helper
that shall then be used to generate spans that can then
be exported for examination.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Jul 14, 2024
This change introduces an OpenTelemetry tracing helper
that shall then be used to generate spans that can then
be exported for examination.

For background, please see the wholesome PR
#3
containing all the changes combined.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Jul 14, 2024
This change introduces an OpenTelemetry tracing helper
that shall then be used to generate spans that can then
be exported for examination.

For background, please see the wholesome PR
#3
containing all the changes combined.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 1, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 4, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 8, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 17, 2024
This change adds the scaffolding for instrumentation
along with tests for the tracing mechanisms that'll
be used to trace the entire package.

Built from googleapis#2087
Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 17, 2024
This change adds the scaffolding for instrumentation
along with tests for the tracing mechanisms that'll
be used to trace the entire package.

Built from googleapis#2087
Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 30, 2024
This change adds the scaffolding for instrumentation
along with tests for the tracing mechanisms that'll
be used to trace the entire package.

Built from googleapis#2087
Updates googleapis#2079
gcf-merge-on-green bot pushed a commit that referenced this issue Aug 30, 2024
This change adds the scaffolding for instrumentation along with tests for the
tracing mechanisms that'll be used to trace the entire package.

Built from #2087
Updates #2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 30, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079

observability-test: more tests for startTrace
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 30, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079

observability-test: more tests for startTrace
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 30, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079

observability-test: more tests for startTrace
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 30, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079

observability-test: more tests for startTrace
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Aug 30, 2024
This change implements tracing of RPC calls using
OpenTelemetry to aid in providing observability.

Fixes googleapis#2079

observability-test: more tests for startTrace
gcf-merge-on-green bot pushed a commit that referenced this issue Oct 10, 2024
…l.createSessions (#2145)

This change adds tracing for Database.batchCreateSessions
as well as SessionPool.createSessions which was raised
as a big need.
This change is a premise to finishing up tracing Transaction.

While here, also folded in the async/await fix to avoid day+ long
code review lag and then 3+ hours just to run tests per PR:
OpenTelemetry cannot work correctly for async/await if there isn't
a set AsyncHooksManager, but we should not burden our customers with
this type of specialist knowledge, their code should just work and
this change performs such a check. Later on we shall file a feature
request with the OpenTelemetry-JS API group to give us a hook to detect
if we've got a live asyncHooksManager instead of this mandatory
comparison to ROOT_CONTEXT each time.

Fixes #2146
Updates #2079
Spun out of PR #2122
Supersedes PR #2147
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 10, 2024
…d to aid in quick debugging

With this change customers shall always be able to identify which
database is being connected to.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 10, 2024
This change documents an end-to-end observability tracing sample
using OpenTelemetry, which then exports trace spans to Google Cloud Trace.

Updates googleapis#2079
gcf-merge-on-green bot pushed a commit that referenced this issue Oct 10, 2024
…d to aid in quick debugging (#2155)

With this change customers shall always be able to identify which database is being connected to.

Updates #2079

## Exhibit
<img width="1581" alt="Screenshot 2024-10-08 at 10 52 56 PM" src="https://togithub.com/user-attachments/assets/8a4bcb5d-b346-4faf-823a-34d87e4b6ef5">
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 11, 2024
This change traces Database.batchWriteAtLeastOnce. Sadly though
MockSpanner doesn't yet support batch writes hence no end-to-end
test with it.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 11, 2024
This change traces Database.batchWriteAtLeastOnce. Sadly though
MockSpanner doesn't yet support batch writes hence no end-to-end
test with it.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 14, 2024
This change traces Database.batchWriteAtLeastOnce. Sadly though
MockSpanner doesn't yet support batch writes hence no end-to-end
test with it.

Updates googleapis#2079
surbhigarg92 pushed a commit that referenced this issue Oct 17, 2024
This change traces Database.batchWriteAtLeastOnce. Sadly though
MockSpanner doesn't yet support batch writes hence no end-to-end
test with it.

Updates #2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 21, 2024
Extracted out of PR googleapis#2158, this change traces
Database.runTransactionAsync.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 21, 2024
Extracted out of PR googleapis#2158, this change traces
Database.runTransactionAsync.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 25, 2024
Extracted out of PR googleapis#2158, this change traces
Database.runTransactionAsync.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 28, 2024
This change traces Database.runPartitionedUpdate along with
the appropriate tests for it with and without errors.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 28, 2024
This change traces Database.runPartitionedUpdate along with
the appropriate tests for it with and without errors.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 28, 2024
This change traces Database.runPartitionedUpdate along with
the appropriate tests for it with and without errors.

Updates googleapis#2079
gcf-merge-on-green bot pushed a commit that referenced this issue Oct 28, 2024
This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates #2079
@odeke-em
Copy link
Contributor Author

And completed!

surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 28, 2024
…s#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 29, 2024
…s#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 29, 2024
…s#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates googleapis#2079
odeke-em added a commit to odeke-em/nodejs-spanner that referenced this issue Oct 29, 2024
This change documents an end-to-end observability tracing sample
using OpenTelemetry, which then exports trace spans to Google Cloud Trace.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 29, 2024
…s#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 29, 2024
…s#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates googleapis#2079
surbhigarg92 pushed a commit to surbhigarg92/nodejs-spanner that referenced this issue Oct 29, 2024
…s#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates googleapis#2079
surbhigarg92 added a commit that referenced this issue Oct 30, 2024
* feat: (observability, samples): add tracing end-to-end sample

This change documents an end-to-end observability tracing sample
using OpenTelemetry, which then exports trace spans to Google Cloud Trace.

Updates #2079

* Update with code review comments

* Minimize observability sample

* Tailor samples to match rubric

* Add OBSERVABILITY.md docs file

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* system-test: add sample test

* samples: remove demo of User Defined Traces

* Address review feedback

* Remove the vestige of tag until the team decides an appropriate one

* Address Surbhi's review feedback

* Update samples/system-test/spanner.test.js

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: surbhigarg92 <[email protected]>
gcf-merge-on-green bot pushed a commit that referenced this issue Oct 30, 2024
🤖 I have created a release *beep* *boop*
---


## [7.15.0](https://togithub.com/googleapis/nodejs-spanner/compare/v7.14.0...v7.15.0) (2024-10-30)


### Features

* (observability, samples): add tracing end-to-end sample ([#2130](https://togithub.com/googleapis/nodejs-spanner/issues/2130)) ([66d99e8](https://togithub.com/googleapis/nodejs-spanner/commit/66d99e836cd2bfbb3b0f78980ec2b499f9e5e563))
* (observability) add spans for BatchTransaction and Table ([#2115](https://togithub.com/googleapis/nodejs-spanner/issues/2115)) ([d51aae9](https://togithub.com/googleapis/nodejs-spanner/commit/d51aae9c9c3c0e6319d81c2809573ae54675acf3)), closes [#2114](https://togithub.com/googleapis/nodejs-spanner/issues/2114)
* (observability) Add support for OpenTelemetry traces and allow observability options to be passed.  ([#2131](https://togithub.com/googleapis/nodejs-spanner/issues/2131)) ([5237e11](https://togithub.com/googleapis/nodejs-spanner/commit/5237e118befb4b7fe4aea76a80a91e822d7a22e4)), closes [#2079](https://togithub.com/googleapis/nodejs-spanner/issues/2079)
* (observability) propagate database name for every span generated to aid in quick debugging ([#2155](https://togithub.com/googleapis/nodejs-spanner/issues/2155)) ([0342e74](https://togithub.com/googleapis/nodejs-spanner/commit/0342e74721a0684d8195a6299c3a634eefc2b522))
* (observability) trace Database.batchCreateSessions + SessionPool.createSessions ([#2145](https://togithub.com/googleapis/nodejs-spanner/issues/2145)) ([f489c94](https://togithub.com/googleapis/nodejs-spanner/commit/f489c9479fa5402f0c960cf896fd3be0e946f182))
* (observability): trace Database.runPartitionedUpdate ([#2176](https://togithub.com/googleapis/nodejs-spanner/issues/2176)) ([701e226](https://togithub.com/googleapis/nodejs-spanner/commit/701e22660d5ac9f0b3e940ad656b9ca6c479251d)), closes [#2079](https://togithub.com/googleapis/nodejs-spanner/issues/2079)
* (observability): trace Database.runTransactionAsync ([#2167](https://togithub.com/googleapis/nodejs-spanner/issues/2167)) ([d0fe178](https://togithub.com/googleapis/nodejs-spanner/commit/d0fe178623c1c48245d11bcea97fcd340b6615af)), closes [#207](https://togithub.com/googleapis/nodejs-spanner/issues/207)
* Allow multiple KMS keys to create CMEK database/backup ([#2099](https://togithub.com/googleapis/nodejs-spanner/issues/2099)) ([51bc8a7](https://togithub.com/googleapis/nodejs-spanner/commit/51bc8a7445ab8b3d2239493b69d9c271c1086dde))
* **observability:** Fix bugs found from product review + negative cases ([#2158](https://togithub.com/googleapis/nodejs-spanner/issues/2158)) ([cbc86fa](https://togithub.com/googleapis/nodejs-spanner/commit/cbc86fa80498af6bd745eebb9443612936e26d4e))
* **observability:** Trace Database methods ([#2119](https://togithub.com/googleapis/nodejs-spanner/issues/2119)) ([1f06871](https://togithub.com/googleapis/nodejs-spanner/commit/1f06871f7aca386756e8691013602b069697bb87)), closes [#2114](https://togithub.com/googleapis/nodejs-spanner/issues/2114)
* **observability:** Trace Database.batchWriteAtLeastOnce ([#2157](https://togithub.com/googleapis/nodejs-spanner/issues/2157)) ([2a19ef1](https://togithub.com/googleapis/nodejs-spanner/commit/2a19ef1af4f6fd1b81d08afc15db76007859a0b9)), closes [#2079](https://togithub.com/googleapis/nodejs-spanner/issues/2079)
* **observability:** Trace Transaction ([#2122](https://togithub.com/googleapis/nodejs-spanner/issues/2122)) ([a464bdb](https://togithub.com/googleapis/nodejs-spanner/commit/a464bdb5cbb7856b7a08dac3ff48132948b65792)), closes [#2114](https://togithub.com/googleapis/nodejs-spanner/issues/2114)


### Bug Fixes

* Exact staleness timebound ([#2143](https://togithub.com/googleapis/nodejs-spanner/issues/2143)) ([f01516e](https://togithub.com/googleapis/nodejs-spanner/commit/f01516ec6ba44730622cfb050c52cd93f30bba7a)), closes [#2129](https://togithub.com/googleapis/nodejs-spanner/issues/2129)
* GetMetadata for Session ([#2124](https://togithub.com/googleapis/nodejs-spanner/issues/2124)) ([2fd63ac](https://togithub.com/googleapis/nodejs-spanner/commit/2fd63acb87ce06a02d7fdfa78d836dbd7ad59a26)), closes [#2123](https://togithub.com/googleapis/nodejs-spanner/issues/2123)

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
surbhigarg92 added a commit that referenced this issue Oct 30, 2024
* chore: integration test fix

* traces tests refactoring

* feat: (observability): trace Database.runPartitionedUpdate (#2176)

This change traces Database.runPartitionedUpdate along with the appropriate tests for it with and without errors.

Updates #2079

* moving additional attributes to separate PR

---------

Co-authored-by: Emmanuel T Odeke <[email protected]>
Co-authored-by: alkatrivedi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/nodejs-spanner API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
1 participant