support latest in distribution build urls #1569

tianleh · 2022-01-27T22:28:16Z

Signed-off-by: Tianle Huang [email protected]

Description

Add the logic to scan S3 bucket to get the max build number. The version and bucket name are hardcoded for demo purpose. Also move the Lambda function logic to a local js file instead of inline code.

Issues Resolved

#1492

Test

able to get the max build number in my test account.
INFO maxBuildNumber 21

The S3 structure is similar to distribution bucket.

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

peternied

Thanks for sharing, good progress on this space.

Do you know why the package-lock.json change is so big? I wouldn't expect much of any change to it.

deployment/lib/cf-url-rewriter.js

kavilla · 2022-01-29T08:04:40Z

Will this enable us to hit latest plugins as well?

deployment/lib/cf-url-rewriter.js

tianleh · 2022-01-31T08:41:00Z

Will this enable us to hit latest plugins as well?

This feature will only replace the latest with the max build number. Thus whatever it has with the build number shall still work with the latest keyword.

codecov-commenter · 2022-02-03T06:21:46Z

Codecov Report

Merging #1569 (9fea470) into main (986736d) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1569      +/-   ##
============================================
- Coverage     94.62%   94.61%   -0.02%     
  Complexity       14       14              
============================================
  Files           164      165       +1     
  Lines          3459     3489      +30     
  Branches         21       25       +4     
============================================
+ Hits           3273     3301      +28     
- Misses          183      185       +2     
  Partials          3        3

Impacted Files	Coverage Δ
...loyment/lambdas/cf-url-rewriter/cf-url-rewriter.ts	`100.00% <100.00%> (+50.00%)`	⬆️
deployment/lambdas/cf-url-rewriter/https-get.ts	`100.00% <100.00%> (ø)`
src/system/temporary_directory.py	`83.87% <0.00%> (-12.91%)`	⬇️
src/system/os.py	`92.30% <0.00%> (-7.70%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 986736d...9fea470. Read the comment docs.

tianleh · 2022-02-03T06:28:14Z

Experiment a bit about this comment about pagination support #1569 (comment)

I wrote a script to put >1K objects under a version ("1.7.1") in my personal AWS account to simulate the situation where distribution build generates more than 1K builds for a version.

Without making the change to support pagination, I run the Lambda function and can indeed see the limitation of ListObjectsV2 API about CommonPrefixes field as documented https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-CommonPrefixes

After my change, I am able to see the full result being retrieved. The CloudWatch log snippet shows:

tianleh · 2022-02-06T01:04:15Z

Added the logic to upload index.yml but it failed with the following S3 exception.

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

tianleh · 2022-02-06T04:37:34Z

Added the logic to upload index.yml but it failed with the following S3 exception.

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

Created a Github issue to track the S3 permission which could be worked in parallel. #1601

I temporarily upload to $BUCKET/distribution-build-opensearch/1.2.4/dist instead of $BUCKET/distribution-build-opensearch/1.2.4 to continue working of this PR.

tianleh · 2022-02-07T06:19:02Z

Tuning the Lambda function to get latest replaced by a build number in my personal AWS account.

request.uri = request.uri.replace('latest', '147');

The one with real number works
d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/147/test.png

But the latest one is not
d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/latest/test.png

Need to dig more why the Lambda is not working.

seraphjiang · 2022-02-07T06:23:16Z

is result cached by cloudfront cdn?

tianleh · 2022-02-07T08:07:19Z

is result cached by cloudfront cdn?

Thanks for the reminder. After digging more with @seraphjiang , the new logic needs to go through new version publish and deployment to Lambda@Edge to take effect. Now the latest url is working
https://d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/latest/test.png

Next step: read yml to get the build number instead of hardcoding mapping latest to 147

dblock · 2022-02-07T13:25:02Z

I suggest redirecting latest to an actual build number with a 302, which avoids the whole cloudfront caching business.

peternied · 2022-02-07T23:45:46Z

Today I had some time so I've created two PRs to help with this one, @tianleh please take a look at them and consider merging them or building off of them.

Add permissions to upload index.yml files for the bundle role #3
Switch url rewriter from js to typescipt #4

tianleh · 2022-02-09T03:02:42Z

I suggest redirecting latest to an actual build number with a 302, which avoids the whole cloudfront caching business.

Sure. I was able to make this happen in my Lambda.

https://d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/latest/test.png

    const redirectResponse = {
          status: '302',
          statusDescription: 'Moved temporarily',
          headers: {
            'location': [{
              key: 'Location',
              value: request.uri.replace('latest', '151'),
            }],
            'cache-control': [{
              key: 'Cache-Control',
              value: "max-age=3600"
            }],
          },
      };

tianleh · 2022-02-09T10:02:33Z

experimenting on how to get the index.yml

Option 1: download (e.g use JavaScript library 'https') the yml file from the URL d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/dist/index.yml to a Lambda local file system. Read the yml and get the build number.

Option 2: get the S3 keys from the url https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.1.1/latest/linux/x64/dist/opensearch/manifest.yml However, we do not know the S3 bucket name. May need a way to figure this out.

tianleh · 2022-02-10T08:56:52Z

experimenting on how to get the index.yml

Option 1: download (e.g use JavaScript library 'https') the yml file from the URL d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/dist/index.yml to a Lambda local file system. Read the yml and get the build number.

Option 2: get the S3 keys from the url https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.1.1/latest/linux/x64/dist/opensearch/manifest.yml However, we do not know the S3 bucket name. May need a way to figure this out.

struggled with Option 1 to download from url. Based on my research, the https library cannot be promisified. https://stackoverflow.com/questions/65306617/async-await-for-node-js-https-get

Will try to see if there is other library to handle the file download.

tianleh · 2022-02-10T09:49:11Z

experimenting on how to get the index.yml
Option 1: download (e.g use JavaScript library 'https') the yml file from the URL d271n53yru5cr7.cloudfront.net/distribution-build-opensearch/2.0.0/dist/index.yml to a Lambda local file system. Read the yml and get the build number.
Option 2: get the S3 keys from the url https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.1.1/latest/linux/x64/dist/opensearch/manifest.yml However, we do not know the S3 bucket name. May need a way to figure this out.

struggled with Option 1 to download from url. Based on my research, the https library cannot be promisified. https://stackoverflow.com/questions/65306617/async-await-for-node-js-https-get

Will try to see if there is other library to handle the file download.

Received the following error when tuning inside Lambda console. I suspect there is some version upgrade needed to support such in Lambda. Researching...

tianleh · 2022-02-28T04:57:13Z

Notes for reviewers:

~~This current PR contains another open PR fix failed jest by unifying cdk versions #1675 which is for fixing jest CI flow. I would like to have the related PR merged first before merging this PR.~~
Since our CI runs yarn https://github.com/opensearch-project/opensearch-build/runs/5355023313?check_suite_focus=true , I am also including deployment/yarn.lock in the commits.

Signed-off-by: Tianle Huang <[email protected]>

The bundle role is the one used by the distribution-build jobs when vars/uploadArtifacts.groovy is processes. This change allows the uploading of an index.yml Signed-off-by: Peter Nied <[email protected]>

I pulled in a new library @aws-cdk/aws-lambda-nodejs that handles the transformation of javascript to typescript. Had to patch up a couple of things which made this change a little more complex that intended. - All of the CDK components were not version locked, that meant depending on the order of npm installs they could drift and get into a dependancy mismatch spiral. The version itself isn't important, its uniformity is all that matters. - Added permissions to the lambda role for interacting with cloudfronts distribution system, these were handled by the lambda function during deployment, but it seems this new library while handling the js -> ts brought some limitations and I had to manually implement the permssions see for details https://tinyurl.com/5xy3n8zt - Need to fix some typescript compliation issues, please feel free to re-work all of this and just keep the method signature Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Tianle Huang <[email protected]> Revert "support js-yaml" This reverts commit c78e701. Signed-off-by: Tianle Huang <[email protected]> add js-yaml Signed-off-by: Tianle Huang <[email protected]>

Signed-off-by: Tianle Huang <[email protected]> clean up Signed-off-by: Tianle Huang <[email protected]> update lock json Signed-off-by: Tianle Huang <[email protected]> update Signed-off-by: Tianle Huang <[email protected]> update the s3 location for index.json Signed-off-by: Tianle Huang <[email protected]> delete unused from rebase Signed-off-by: Tianle Huang <[email protected]>

Signed-off-by: Tianle Huang <[email protected]>

dblock

I am good with this if @peternied is good with this.

peternied

Thank you for this change!

peternied · 2022-03-01T12:54:24Z

Can @opensearch-project/engineering-effectiveness update this PR after the change has been deployed?

I know there are a bunch of folks that would love to use 'latest' in their codebases.

gaiksaya · 2022-03-01T18:20:29Z

Can @opensearch-project/engineering-effectiveness update this PR after the change has been deployed?

I know there are a bunch of folks that would love to use 'latest' in their codebases.

Looking into it. Will update this PR once the change is deployed. Thanks!

gaiksaya · 2022-03-01T19:38:58Z

These changes have been deployed. @tianleh Can you check if it is working as expected?
Thanks!

tianleh requested review from peternied, dblock, kavilla, seraphjiang and ohltyler January 27, 2022 22:28

tianleh mentioned this pull request Jan 27, 2022

[Feature] Add support for urls to substitute latest for the most recent build number #1492

Closed

kavilla linked an issue Jan 27, 2022 that may be closed by this pull request

[Feature] Add support for urls to substitute latest for the most recent build number #1492

Closed

peternied suggested changes Jan 27, 2022

View reviewed changes

dblock reviewed Jan 30, 2022

View reviewed changes

deployment/lib/cf-url-rewriter.js Outdated Show resolved Hide resolved

seraphjiang approved these changes Jan 30, 2022

View reviewed changes

seraphjiang approved these changes Jan 31, 2022

View reviewed changes

davidlago mentioned this pull request Feb 2, 2022

Use latest artifacts for OpenSearch and OpenSearch Security Plugin during integration tests opensearch-project/security-dashboards-plugin#886

Closed

tianleh force-pushed the support-latest branch from bfa2284 to 97e7bdc Compare February 5, 2022 21:51

tianleh mentioned this pull request Feb 6, 2022

Allow S3 uploading of index.yml file besides builds/dist #1601

Closed

peternied mentioned this pull request Feb 7, 2022

Switch url rewriter from js to typescipt tianleh/opensearch-build#4

Merged

tianleh and others added 18 commits February 28, 2022 19:19

support latest in distribution build urls

68da29d

Signed-off-by: Tianle Huang <[email protected]>

add support for pagination

aff53b5

Signed-off-by: Tianle Huang <[email protected]>

upload index yaml with latest build number

83bdb77

Signed-off-by: Tianle Huang <[email protected]>

check why AmazonS3Exception: Access Denied

946007c

Signed-off-by: Tianle Huang <[email protected]>

Add permisions to upload index.yml files for the bundle role

47f62eb

The bundle role is the one used by the distribution-build jobs when vars/uploadArtifacts.groovy is processes. This change allows the uploading of an index.yml Signed-off-by: Peter Nied <[email protected]>

support js-yaml

2300eaf

Signed-off-by: Tianle Huang <[email protected]> Revert "support js-yaml" This reverts commit c78e701. Signed-off-by: Tianle Huang <[email protected]> add js-yaml Signed-off-by: Tianle Huang <[email protected]>

address some feedback

ba48f73

Signed-off-by: Tianle Huang <[email protected]>

add error handling

830ca50

Signed-off-by: Tianle Huang <[email protected]>

replace slash as well

bef8a48

Signed-off-by: Tianle Huang <[email protected]>

replace

5f81401

Signed-off-by: Tianle Huang <[email protected]>

sync from upstream main

d8fad2e

Signed-off-by: Tianle Huang <[email protected]>

more sync from upstream

3b57bb0

Signed-off-by: Tianle Huang <[email protected]>

more sync

7304e9e

Signed-off-by: Tianle Huang <[email protected]>

add unit tests

8708082

Signed-off-by: Tianle Huang <[email protected]>

add unit tests

f446f2a

Signed-off-by: Tianle Huang <[email protected]>

add one more test

96b4670

Signed-off-by: Tianle Huang <[email protected]>

tianleh force-pushed the support-latest branch from c747319 to 96b4670 Compare February 28, 2022 19:24

rebase

9fea470

Signed-off-by: Tianle Huang <[email protected]>

dblock approved these changes Feb 28, 2022

View reviewed changes

peternied approved these changes Mar 1, 2022

View reviewed changes

peternied merged commit a0e25d8 into opensearch-project:main Mar 1, 2022

tianleh added the ci-test-automation label Mar 2, 2022

tianleh mentioned this pull request Mar 2, 2022

add documentation about latest url #1689

Closed

2 tasks

amitgalitz mentioned this pull request Mar 29, 2022

Make older OpenSearch releases (1.1, 1.2 and etc) available to fetch with latest capability #1845

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support latest in distribution build urls #1569

support latest in distribution build urls #1569

tianleh commented Jan 27, 2022

peternied left a comment

kavilla commented Jan 29, 2022

tianleh commented Jan 31, 2022 •

edited

Loading

codecov-commenter commented Feb 3, 2022 •

edited

Loading

tianleh commented Feb 3, 2022

tianleh commented Feb 6, 2022

tianleh commented Feb 6, 2022 •

edited

Loading

tianleh commented Feb 7, 2022

seraphjiang commented Feb 7, 2022

tianleh commented Feb 7, 2022

dblock commented Feb 7, 2022 •

edited

Loading

peternied commented Feb 7, 2022

tianleh commented Feb 9, 2022 •

edited

Loading

tianleh commented Feb 9, 2022

tianleh commented Feb 10, 2022

tianleh commented Feb 10, 2022

tianleh commented Feb 28, 2022 •

edited

Loading

dblock left a comment

peternied left a comment

peternied commented Mar 1, 2022

gaiksaya commented Mar 1, 2022

gaiksaya commented Mar 1, 2022

support latest in distribution build urls #1569

support latest in distribution build urls #1569

Conversation

tianleh commented Jan 27, 2022

Description

Issues Resolved

Test

Check List

peternied left a comment

Choose a reason for hiding this comment

kavilla commented Jan 29, 2022

tianleh commented Jan 31, 2022 • edited Loading

codecov-commenter commented Feb 3, 2022 • edited Loading

Codecov Report

tianleh commented Feb 3, 2022

tianleh commented Feb 6, 2022

tianleh commented Feb 6, 2022 • edited Loading

tianleh commented Feb 7, 2022

seraphjiang commented Feb 7, 2022

tianleh commented Feb 7, 2022

dblock commented Feb 7, 2022 • edited Loading

peternied commented Feb 7, 2022

tianleh commented Feb 9, 2022 • edited Loading

tianleh commented Feb 9, 2022

tianleh commented Feb 10, 2022

tianleh commented Feb 10, 2022

tianleh commented Feb 28, 2022 • edited Loading

dblock left a comment

Choose a reason for hiding this comment

peternied left a comment

Choose a reason for hiding this comment

peternied commented Mar 1, 2022

gaiksaya commented Mar 1, 2022

gaiksaya commented Mar 1, 2022

tianleh commented Jan 31, 2022 •

edited

Loading

codecov-commenter commented Feb 3, 2022 •

edited

Loading

tianleh commented Feb 6, 2022 •

edited

Loading

dblock commented Feb 7, 2022 •

edited

Loading

tianleh commented Feb 9, 2022 •

edited

Loading

tianleh commented Feb 28, 2022 •

edited

Loading