Download data from S3 location using gateway endpoint and no internet

[himanshu-amazon]

**Is your feature request related to a problem? Please describe. We want to run the benchmark test from an Amazon EC2 instance in private subnet with no connectivity to internet but it can download files from S3 using Gateway endpoint.

A clear and concise description of what the problem is. We want to run the benchmark test from an Amazon EC2 instance in private subnet with no connectivity to internet but it can download files from S3 using Gateway endpoint.

Describe the solution you'd like
We want to ability to be able to download files from S3 even without internet as connectivity to S3 is there using gateway endpoint. Currently OSB using HTTP url which can only be accessed using internet for downloads.

Describe alternatives you've considered

I cloned the OSB code and updated workload/loader.py file to use boto3 and download the files without internet

Additional context

Add any other context or screenshots about the feature request here.

[IanHoang]

@himanshu-amazon Thanks for bringing attention to this. This would certainly be useful for individuals with similar testing setups. You're more than welcome to implement this, create a PR, and tag maintainers once it's ready to be reviewed!

[gkamat]

@himanshu-amazon please provide any updates on this if you have any information. This would be a useful feature, but it should be implemented in an open-source-compliant way, since we would not want vendor-specific functionality in core OSB. Thanks.

[himanshu-amazon]

Hi Govind, I work for customer specific projects and would not have time currently to contribute at the moment. Please feel free to make changes in the code as needed. Requirement was to be able access S3 bucket without internet and most of the enterprise customer want to avoid internet access and want to use private endpoints. With S3 we can create gateway or interface endpoints to access S3 buckets without internet. https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html Thanks, Himanshu Dixit Cloud Infrastructure Architect AWS Professional Services 980.318.1864 - mobile [signature_4200569804] <http://www.aws.amazon.com/> [signature_3873578238] [signature_2210450787] [signature_1957359825] [signature_168463157] [signature_400507593] [signature_3553748852] Thoughts on our interaction? Provide feedback here<https://feedback.aws.amazon.com/?ea=hdixit&fn=Himanshu&ln=Dixit>. From: Govind Kamat ***@***.***> Reply-To: opensearch-project/opensearch-benchmark ***@***.***> Date: Tuesday, October 15, 2024 at 4:50 PM To: opensearch-project/opensearch-benchmark ***@***.***> Cc: "Dixit, Himanshu" ***@***.***>, Mention ***@***.***> Subject: Re: [opensearch-project/opensearch-benchmark] Download data from S3 location using gateway endpoint and no internet (Issue #472) @himanshu-amazon<https://github.com/himanshu-amazon> please provide any updates on this if you have any information. This would be a useful feature, but it should be implemented in an open-source-compliant way, since we would not want vendor-specific functionality in core OSB. Thanks. — Reply to this email directly, view it on GitHub<#472 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BGSCCBKI74GXBISYC3UBXTLZ3V5V7AVCNFSM6AAAAABP77VEFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJVGA3TIOBUGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>