Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(aws-sdk): add s3 and kinesis service extensions for aws-sdk instrumentation #2361

Merged
merged 12 commits into from
Nov 1, 2024

Conversation

jj22ee
Copy link
Contributor

@jj22ee jj22ee commented Jul 31, 2024

Which problem is this PR solving?

Short description of the changes

  • Add service extensions for S3 and Kinesis in AWS SDK instrumentation.
    • For S3, add bucket name to span if present
    • For Kinesis, add stream name to span if present
  • This is a simple starting point for these new service extensions. The bucket/stream names are most important, but in the future, perhaps more data can be collected as span attributes.

Regarding some added constants to the utils file:

export const _AWS_S3_BUCKET = 'aws.s3.bucket';
export const _AWS_KINESIS_STREAM_NAME = 'aws.kinesis.stream.name';

Testing

  1. Unit tests
  2. Tested with:
  • Kinesis Client: observed 'aws.kinesis.stream.name': myNewKinesisStream
    const client = new KinesisClient({ region: 'us-east-1' });
    const input = { // CreateStreamInput
      StreamName: "myNewKinesisStream", // required
      ShardCount: Number(3),
      StreamModeDetails: { // StreamModeDetails
        StreamMode: "ON_DEMAND", // required
      },
    };
    const command = new CreateStreamCommand(input);
    await client.send(command).then(data => {
      console.log(data);
    })
  • S3 Client: observed 'aws.s3.bucket': test-bucket-not-exist-or-accessible
  const s3Client = new S3Client({ region: 'us-east-1' });
  const bucketName = 'test-bucket-not-exist-or-accessible';
  try {
    await s3Client.send(
      new ListObjectsCommand({
        Bucket: bucketName,
      }),
    ).then((data) => {
      console.log(data);
    });
  } catch (e) {
    if (e instanceof Error) {
      console.error('Exception thrown: ', e.message);
    }
  } finally {
    res.send('done aws sdk s3 request');
  }

Copy link

codecov bot commented Jul 31, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.85%. Comparing base (ad8c581) to head (e689114).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2361      +/-   ##
==========================================
+ Coverage   90.83%   90.85%   +0.01%     
==========================================
  Files         159      161       +2     
  Lines        7831     7859      +28     
  Branches     1610     1614       +4     
==========================================
+ Hits         7113     7140      +27     
- Misses        718      719       +1     
Files with missing lines Coverage Δ
...opentelemetry-instrumentation-aws-sdk/src/enums.ts 100.00% <100.00%> (ø)
...ntation-aws-sdk/src/services/ServicesExtensions.ts 96.29% <100.00%> (-3.71%) ⬇️
...ry-instrumentation-aws-sdk/src/services/kinesis.ts 100.00% <100.00%> (ø)
...lemetry-instrumentation-aws-sdk/src/services/s3.ts 100.00% <100.00%> (ø)
...opentelemetry-instrumentation-aws-sdk/src/utils.ts 97.29% <ø> (ø)

@jj22ee
Copy link
Contributor Author

jj22ee commented Jul 31, 2024

Pinging @blumamir for review!

Unrelated heads-up, I'm looking to take over component ownership of AWS JS components from @carolabadeer. I think I just need to make a PR like #1464.

@trentm
Copy link
Contributor

trentm commented Aug 9, 2024

I think I just need to make a PR like #1464.

@jj22ee Yup that's correct. Great to have you helping.

Copy link
Member

@blumamir blumamir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution and the info in the PR.

Left few comments. Some of them I wrote on s3 implementation but they also apply to kinesis, didn't want to duplicate the same comments twice :)

import { AwsInstrumentation } from '../src';
registerInstrumentationTesting(new AwsInstrumentation());

import * as AWS from 'aws-sdk';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome to have tests for that.

Please note that those tests are for v2 which will soon be end of life.
Consider also adding assertions on these new attributes to v3 tests in test/aws-sdk-v3.test.ts here

jj22ee added a commit to aws-observability/aws-otel-js-instrumentation that referenced this pull request Aug 15, 2024
*Issue #, if available:*

*Description of changes:*
1. `aws-attribute-keys.ts`:
- Update values for keys: `AWS_BUCKET_NAME, AWS_QUEUE_URL,
AWS_QUEUE_NAME, AWS_STREAM_NAME, AWS_TABLE_NAME` in order to match the
actual attribute collected from AWS SDK auto-instrumentation

2. `aws-metric-attribute-generator.ts`:
- [Similarly to
Python](https://github.com/aws-observability/aws-otel-python-instrumentation/blob/2c00bd07eaaf703880a24a2fcfe874cdb4196678/aws-opentelemetry-distro/src/amazon/opentelemetry/distro/_aws_metric_attribute_generator.py#L378-L380),
accommodates the fact that AWS_ATTRIBUTE_KEYS.AWS_TABLE_NAMES [has an
array of table
names](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/931318cac21ee3707f3735a64ac751566ee37182/plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/dynamodb.ts#L99-L105)
from aws-sdk dynamodb client instrumentation

3. `aws-opentelemetry-configurator.ts`:
    - Takes and sets instrumentations in constructor.

4. `instrumentation-patch.ts`:
    - Applies patches for AWS SDK Instrumentation

5. `register.ts`:
- Applies patches from (4.) by default, unless `AWS_APPLY_PATCHES` env
var is not `'true'`

The following files are copied from upstream:
```
aws-distro-opentelemetry-node-autoinstrumentation/src/patches/aws/services/ServiceExtension.ts
```
The following files are being contributed to upstream:
See:
open-telemetry/opentelemetry-js-contrib#2361
```
aws-distro-opentelemetry-node-autoinstrumentation/src/patches/aws/services/kinesis.ts
aws-distro-opentelemetry-node-autoinstrumentation/src/patches/aws/services/s3.ts
```

*Testing:*
- Add unit tests

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
@jj22ee jj22ee force-pushed the aws-sdk-s3-kinesis-extensions branch from 7794f36 to ed94e6c Compare August 19, 2024 08:12
@jj22ee jj22ee force-pushed the aws-sdk-s3-kinesis-extensions branch from f382a46 to b97d73a Compare September 16, 2024 16:12
@jj22ee jj22ee requested a review from a team as a code owner September 26, 2024 20:05
@jj22ee
Copy link
Contributor Author

jj22ee commented Sep 26, 2024

Hi, looking for another round of review! I removed some of the aws-sdk v2 tests I had previously given their end-of-life and soon-to-be-deprecated statuses, and removed the latency with the Kinesis Client test (enforced HTTP instead of HTTP2 which isn't supported by nock)

Copy link
Member

@blumamir blumamir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 💯

Thank you for the contribution, the code and tests looks great and the description on the PR is very helpful.

@blumamir blumamir merged commit a5b5614 into open-telemetry:main Nov 1, 2024
22 of 23 checks passed
@dyladan dyladan mentioned this pull request Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants