Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added CI testing against S3 bucket. #1648

Merged
merged 2 commits into from
Oct 15, 2019
Merged

Added CI testing against S3 bucket. #1648

merged 2 commits into from
Oct 15, 2019

Conversation

bwplotka
Copy link
Member

  • Again, only for master/release builds and PRs from person with write-access to Thanos.

Signed-off-by: Bartek Plotka [email protected]

@bwplotka
Added CI testing against S3 bucket.
f84735f 
* Again, only for master/release builds and PRs from person with write-access to Thanos.

Signed-off-by: Bartek Plotka <[email protected]>
@brancz
Copy link
Member

brancz commented Oct 15, 2019 •
edited
Loading

Looks like we've got some failures. Happy to review once resolved. (looks good though so far)

@bwplotka
Copy link
Member Author

bwplotka commented Oct 15, 2019 •
edited
Loading

Yea, the reason is that minio-go client does not work if you don't pass os.File as io.Reader which is really bad. We touched this problem at some point ago, we should either:

  • Be clear we require the size of upload before op: add ReaderSizer interface or something. Now it is super hidden. In fact we always upload files, we never stream without knowing size, so that would do... This will block though any attempts to do some streaming more in future.
  • Fix minio-go to work without file size (exploring)
  • Just burn all of it and use AWS SDK or better.. go-cloud! (wonder if go cloud S3 driver works for other S3-compatible backends EDIT: it works)

@bwplotka bwplotka mentioned this pull request Oct 15, 2019
Closed
@bwplotka bwplotka merged commit 5d107d1 into master Oct 15, 2019
@bwplotka bwplotka deleted the added-s3-ci branch October 15, 2019 12:50
@bwplotka
Copy link
Member Author

Resolved it quickly.

Let's discuss moving to go-cloud here: #443.

GiedriusS pushed a commit that referenced this pull request Oct 28, 2019
@bwplotka @GiedriusS
Added CI testing against S3 bucket. (#1648)
a492579 
* Added CI testing against S3 bucket.

* Again, only for master/release builds and PRs from person with write-access to Thanos.

Signed-off-by: Bartek Plotka <[email protected]>

* Fixed multipart upload for S3 for smaller objects.

Signed-off-by: Bartek Plotka <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@povilasv povilasv povilasv approved these changes

@jojohappy jojohappy Awaiting requested review from jojohappy

@brancz brancz Awaiting requested review from brancz

@domgreen domgreen Awaiting requested review from domgreen

@GiedriusS GiedriusS Awaiting requested review from GiedriusS

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bwplotka @brancz @povilasv