Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't allow empty S3-key parts #98

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Dunedan
Copy link

@Dunedan Dunedan commented Jul 4, 2017

dynamodb-replicator uses / as delimiter to hierarchically structure
the data in S3. Up to now it did a naive join of all path parts when
constructing paths and key names for S3. That produced different paths
in S3 whether or not a trailing slash was provided when specifying the
target. For example the following two commands produced different output
files in S3:

incremental-backfill us-east-1/table s3://s3-bucket/foo
incremental-backfill us-east-1/table s3://s3-bucket/foo/

As that's different from other tools which interact with S3, like the
official awscli this commit changes that behavior to work like
path.join does, but by using an external package to ensure that also
under Windows paths are generated with forward slashes.

Daniel Roschka added 2 commits July 4, 2017 13:58
`dynamodb-replicator` uses `/` as delimiter to hierarchically structure
the data in S3. Up to now it did a naive join of all path parts when
constructing paths and key names for S3. That produced different paths
in S3 whether or not a trailing slash was provided when specifying the
target. For example the following two commands produced different output
files in S3:

    incremental-backfill us-east-1/table s3://s3-bucket/foo
    incremental-backfill us-east-1/table s3://s3-bucket/foo/

As that's different from other tools which interact with S3, like the
official `awscli` this commit changes that behavior to work like
`path.join` does, but by using an external package to ensure that also
under Windows paths are generated with forward slashes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant