Initial attempt at direct s3 remote cache

.gitattributes

@@ -4,3 +4,8 @@ site/* linguist-documentation
 
 # Files that should not use CRLF line endings, even on Windows.
 tools/genrule/genrule-setup.sh -text
+third_party/aws-sig-v4-test-suite/**/*.authz -text
+third_party/aws-sig-v4-test-suite/**/*.creq -text
+third_party/aws-sig-v4-test-suite/**/*.req -text
+third_party/aws-sig-v4-test-suite/**/*.sreq -text
+third_party/aws-sig-v4-test-suite/**/*.sts -text

CODEOWNERS

@@ -59,6 +59,10 @@ android/ @ahumesky @jin
 
 /src/main/java/com/google/devtools/build/lib/remote/ @buchgr @ishikhman @ola-rozenfeld
 
+/src/main/java/com/google/devtools/build/lib/authentication/ @buchgr
+
+/src/main/java/com/google/devtools/build/lib/grpc/ @buchgr
+
 /src/test/shell/bazel/remote/ @buchgr @ishikhman @ola-rozenfeld
 
 /src/tools/execlog/ @buchgr @ola-rozenfeld

scripts/bootstrap/compile.sh

@@ -44,7 +44,7 @@ if [ "$ERROR_PRONE_INDEX" -lt "$GUAVA_INDEX" ]; then
   LIBRARY_JARS="${LIBRARY_JARS_ARRAY[*]}"
 fi
 
-DIRS=$(echo src/{java_tools/singlejar/java/com/google/devtools/build/zip,main/java} tools/java/runfiles third_party/java/dd_plist/java ${OUTPUT_DIR}/src)
+DIRS=$(echo src/{java_tools/singlejar/java/com/google/devtools/build/zip,main/java} tools/java/runfiles third_party/java/dd_plist/java third_party/aws-sdk-auth-lite ${OUTPUT_DIR}/src)
 EXCLUDE_FILES="src/main/java/com/google/devtools/build/lib/server/GrpcServerImpl.java src/java_tools/buildjar/java/com/google/devtools/build/buildjar/javac/testing/*"
 # Exclude whole directories under the bazel src tree that bazel itself
 # doesn't depend on.

site/docs/remote-caching.md

@@ -18,6 +18,7 @@ make builds significantly faster.
     * [nginx](#nginx)
     * [Bazel Remote Cache](#bazel-remote-cache)
     * [Google Cloud Storage](#google-cloud-storage)
+    * [AWS S3 Storage](#aws-s3-storage)
     * [Other servers](#other-servers)
 * [Authentication](#authentication)
 * [HTTP Caching Protocol](#http-caching-protocol)
@@ -98,6 +99,7 @@ include:
 * [nginx](#nginx)
 * [Bazel Remote Cache](#bazel-remote-cache)
 * [Google Cloud Storage](#google-cloud-storage)
+* [AWS S3 Storage](#aws-s3-storage)
 
 ### nginx
 
@@ -173,11 +175,59 @@ to/from your GCS bucket.
 5. You can configure Cloud Storage to automatically delete old files. To do so, see
 [Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles).
 
+### AWS S3 Storage
+
+[AWS S3] is a fully managed object store which provides an
+HTTP API that is compatible with Bazel's remote caching protocol. It requires
+that you have an AWS account.
+
+To use S3 as the cache:
+
+1. [Create a bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html).
+Ensure that you select a bucket location that's closest to you, as network bandwidth
+is important for the remote cache.
+
+2. Generate the relevant auth token and pass it to Bazel for authentication.
+Store the key securely, as anyone with the key can read and write arbitrary data
+to/from your S3 bucket.
+
+3. Connect to S3 by adding the following flags to your Bazel command:
+   * Pass the following URL to Bazel by using the flag:
+     `--remote_http_cache=https://bucket-name.s3-website-region.amazonaws.com`
+     where `bucket-name` is the name of your storage bucket, and `region` is the
+     name of the chosen region. You will need to use the HTTP/HTTPS url that
+     points to your bucket. Presently `s3://` style urls are _not_ supported due
+     to a lack of any consistent standard.
+   * Pass the authentication key using the flags:
+     `--aws_access_key_id=$ACCESS_KEY`
+     `--aws_secret_access_key=$SECRET_KEY`
+     Alternatively, AWS credentials can be configured using standard methods
+     such as instance roles and environment variables. Use this flag to
+     authenticate with instance credentials: `--aws_default_credentials`
+     A specific credentials profile can be specified using `--aws_profile=$PROFILE`
+
+4. You can configure S3 to automatically delete old files. To do so, see
+[Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
+
+#### Virtual hosted and accelerated buckets
+
+Buckets that have been configured under a virtual host name, or are using the s3
+cloudfront acceleration proxy require some additional configuration.
+
+These buckets cannot determine the region that should be used from the URL alone.
+to specify the region for these buckets directly use the
+`--aws_region=$AWS_REGION` flag.
+
+For those using s3 accelerated buckets, the remote http cache url will be of the
+following form:
+`--remote_http_cache=https://bucket-name.s3-accelerate.amazonaws.com` where
+`bucket-name` is the name of the bucket to use.
+
 ### Other servers
 
 You can set up any HTTP/1.1 server that supports PUT and GET as the cache's
 backend. Users have reported success with caching backends such as [Hazelcast],
-[Apache httpd], and [AWS S3].
+[AWS S3] and [Apache httpd].
 
 ## Authentication
 

src/main/java/com/google/devtools/build/lib/BUILD

@@ -16,7 +16,10 @@ filegroup(
         "//src/main/java/com/google/devtools/build/lib/analysis/platform:srcs",
         "//src/main/java/com/google/devtools/build/lib/analysis/skylark/annotations:srcs",
         "//src/main/java/com/google/devtools/build/lib/analysis/skylark/annotations/processor:srcs",
-        "//src/main/java/com/google/devtools/build/lib/authandtls:srcs",
+        "//src/main/java/com/google/devtools/build/lib/authentication:srcs",
+        "//src/main/java/com/google/devtools/build/lib/authentication/aws:srcs",
+        "//src/main/java/com/google/devtools/build/lib/authentication/google:srcs",
+        "//src/main/java/com/google/devtools/build/lib/grpc:srcs",
         "//src/main/java/com/google/devtools/build/lib/bazel/repository/cache:srcs",
         "//src/main/java/com/google/devtools/build/lib/bazel/repository/downloader:srcs",
         "//src/main/java/com/google/devtools/build/lib/buildeventservice/client:srcs",
@@ -840,6 +843,8 @@ java_library(
         ":bazel-rules",
         ":bazel/BazelRepositoryModule",
         ":build-base",
+        "//src/main/java/com/google/devtools/build/lib/authentication/aws",
+        "//src/main/java/com/google/devtools/build/lib/authentication/google",
         "//src/main/java/com/google/devtools/build/lib/bazel/debug:workspace-rule-module",
         "//src/main/java/com/google/devtools/build/lib/buildeventservice",
         "//src/main/java/com/google/devtools/build/lib/dynamic",