If encryption is required on s3 bucket through bucket policy, output cannot start

[hijakk]

Working on Logstash 5.4.1 on OSX
Config file:

output {
   s3{
     server_side_encryption => true
     server_side_encryption_algorithm => "AES256"
     access_key_id => "mykey"
     secret_access_key => "mykey"
     region => "my-region"
     bucket => "mybucket"
     codec => "json"
      }
}

If bucket policy is set up like:

{
    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "mybucketarn/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "mybucketarn/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

I will consistently get an error like:
[2017-06-16T13:53:38,921][ERROR][logstash.outputs.s3 ] Error validating bucket write permissions! {:message=>"Access Denied", :class=>"Aws::S3::Errors::AccessDenied"}

I believe this is because logstash-output-s3/lib/logstash/outputs/s3/write_bucket_permission_validator.rb attempts to create a file to check that it has write permissions. Due to the bucket policy, since the file that write_bucket_permission_validator.rb attempts to write is not encrypted, it is rejected and logstash decides that it can't output to s3.

If I remove the bucket policy, I am able to write encrypted data to the bucket.

It would be valuable if write_bucket_permission_validator and/or writable_directory_validator.rb were configured to take encryption settings in the output parameterization into account when validating write access.

[hijakk]

We can skip this check using validate_credentials_on_root_bucket => false - not sure if this is an appropriate workaround, though? Having the validation might still be useful in the case of the encrypted buckets.

[jmcarp]

Duplicate of #132.

[jakelandis]

I believe that was fixed in 4.0.12. Please re-open if you find this to not be the case.