Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS credentials for loading S3 files in pipeline.yaml to data-prepper-config.yaml #2771

Open
asifsmohammed opened this issue May 26, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@asifsmohammed
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Right now, to load files from S3 in pipelines.yaml in any plugin pipeline author has to provide AWS credentials in each plugin. This is making the pipeline look noisy. Currently we support loading ISM policy(ism_policy_file ) and index template(template_file) from S3.

Describe the solution you'd like
Leverage the extensions work and create an s3_configuration extension in data-prepper-config.yaml.

aws:
  # New configuration
  s3_configurations:
    authentication: my-custom-role
  credentials:
    default:
      sts_role_arn: arn:aws:iam::99123456789:role/OsiDataPlaneRole
      region: us-east-1
    my-custom-role:
      sts_role_arn:
      region:

Now, pipeline author don't have to configure the credentials in pipeline and just use S3 URI path to load files from S3.

log-pipeline:
  s3:
    aws:
      sts_role_arn: arn:aws:iam::123456789012:role/MyRole
      region: us-east-1
  sink:
    - opensearch:
        hosts: [""]
        ism_policy_file: "s3://filepath"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants