Merge latest changes from rate_limiting to master (envoyproxy#221)

* Point to googleapi in service control client. (envoyproxy#91) * Point to googleapi in service control client. * Use git repository for service-control-client. * Merge latest changes from master (envoyproxy#104) * Get attributes from envoy config. (envoyproxy#87) * Send all attributes. * Remove unused const strings. * Address comment. * updated SHA to point to newer envoy with RDS API feature (envoyproxy#94) * Disable travis on stable branches (envoyproxy#96) * Publish debug binaries (no release yet) (envoyproxy#98) * Copies the binary instead of linking for release (envoyproxy#102) * Extract quota config from service config. (envoyproxy#101) * Add metric_cost in config. * Remove group rules. * Call loadQuotaConfig in config::create. * Update latest update from master branch (envoyproxy#106) * Get attributes from envoy config. (envoyproxy#87) * Send all attributes. * Remove unused const strings. * Address comment. * updated SHA to point to newer envoy with RDS API feature (envoyproxy#94) * Disable travis on stable branches (envoyproxy#96) * Publish debug binaries (no release yet) (envoyproxy#98) * Copies the binary instead of linking for release (envoyproxy#102) * Added quota contoll without the service control client library (envoyproxy#93) * Added quota contoll without the service control client library * Applied code review * Applied code review * Resolve conflicts * Resolve conflicts * Fixed format error reported by script/check-style * Fixed a bug at Aggregated::GetAuthToken that causes Segmentation Fault * Changed usage of template funcion * Applied latest changes from the repo * Applied latest changes from the repo * Applied latest changes from the repo * Adde comments * Updated log information * Applied envoyproxy#101 * Changed metric_cost_map to metric_cost_vector * Fixed test case compilation error * Fixed test case compilation error * Add unit test for quota config. (envoyproxy#108) * Add unit test for quota config. * Add comments. * Update test specifics. * Merge latest changes from master branch (envoyproxy#112) * Get attributes from envoy config. (envoyproxy#87) * Send all attributes. * Remove unused const strings. * Address comment. * updated SHA to point to newer envoy with RDS API feature (envoyproxy#94) * Disable travis on stable branches (envoyproxy#96) * Publish debug binaries (no release yet) (envoyproxy#98) * Copies the binary instead of linking for release (envoyproxy#102) * Not to use api_key if its service is not actived. (envoyproxy#109) * If QuotaControl service is not available, return utils::Status::OK (envoyproxy#113) * If QuotaControl service is not available, return utils::Status::OK * Updated comment * Return HTTP status code 429 on google.rpc.Code.RESOURCE_EXHAUSTED (envoyproxy#119) * Fixed incorrectly resolved conflicts (envoyproxy#123) * Added unit test cases for rate limiting (envoyproxy#124) * Fixed incorrectly resolved conflicts * Added unit test cases for rate limiting * Added unit test cases for rate limiting * Added unit test cases for rate limiting * Added unit test cases for rate limiting * Added unit test cases for rate limiting * Added unit test cases for rate limiting * Rename response.http.code (envoyproxy#125) (envoyproxy#128) * Added handling of error code QUOTA_SYSTEM_UNAVAILABLE (envoyproxy#148) * Integrated service control client library with quota cache aggregation (envoyproxy#149) * Fixed error on merge (envoyproxy#151) * Integrated service control client library with quota cache aggregation * Fixed error on merge * Fixed the compatibility issue with the latest update on esp (envoyproxy#152) * Removed copied proto files (envoyproxy#208) * Set default allocate quota request timeout to 1sec and applied latest service control client library change (envoyproxy#211) * Merged key_restriction related changes from master (envoyproxy#213) * Merge latest changes from master branch (envoyproxy#217) * Not call report if decodeHeaders is not called. (envoyproxy#150) * Update mixerclient with sync-ed grpc write and fail-fast. (envoyproxy#155) * Update mixerclient with sync-ed write and fail-fast. * Update to latest test. * Update again * Update envoy to PR553 (envoyproxy#156) * Update envoy to PR553 * Update libevent to 2.1.8 * Uses a specific version of the Shared Pipeline lib (envoyproxy#158) * Update lyft/envoy commit Id to latest. (envoyproxy#161) * Update lyft/envoy commit Id to latest. * Remove the comment about pull request * Add new line - will delete in next commit. * Update repositories.bzl (envoyproxy#169) * Always set response latency (envoyproxy#172) * Update mixerclient to sync_transport change. (envoyproxy#178) * Use opaque config to turn on/off forward attribute and mixer filter (envoyproxy#179) * Modify mixer filter * Swap defaults * Make the filter decoder only * cache mixer disabled decision * Fix a bug in opaque config change and test it out (envoyproxy#182) * Fix a bug and test it out * Update filter type * Update README.md * Update mixer client to mixer api with gogoproto. (envoyproxy#184) * Move .bazelrc to tools/bazel.rc (envoyproxy#186) * Move .bazelrc to tools/bazel.rc * Update Jenkinsfile with latest version of pipeline * Support apikey based traffic restriction (envoyproxy#189) * b/36368559 support apikey based traffic restriction * Fixed code formatting * Fix crash in unreachable/overloaded RDS (envoyproxy#190) * Add mixer client end to end integration test. (envoyproxy#177) * Add mixer client end to end integration test. * Split some repositories into a separate file. * use real mixer for fake mixer_server. * Test repository * use mixer bzl file. * Use mixer repositories * Not to use mixer repository. * Add return line at the end of WORKSPACE. * Fix broken link (envoyproxy#193) * Make quota call (envoyproxy#192) * hookup quota call * Make quota call. * Update indent. * Update envoy and update configs (envoyproxy#195) * Update envoy and update configs * Use gcc-4.9 for travis * Use bazel 0.4.5 * Fix SHA of lightstep-tracer-common * Enable check cache and refactory mixer config loading (envoyproxy#197) * Refactory the mixer config loading. * fix format * Add integration test. * updated README.md * s/send/sent/ * Split into separate tests. (envoyproxy#201) * Update README on how to enable check cache. (envoyproxy#204) * Update README on how to enable check cache. * Update the comment. * build: support Envoy native Bazel build. (envoyproxy#210) * build: support Envoy native Bazel build. This patch switches the Envoy build from src/envoy/repositories.bzl to using the upstream native build. See envoyproxy#663 for the corresponding changes on the Envoy side. * Use Envoy master with BUILD.wip rename merged. * Fix clang-format issues. * Fixes bazel.rc issues (envoyproxy#212) * Fixes bazel rc issues * Update Jenkins to latest pipeline version * Updated the commit id of cloudendpoints/service-control-client-cxx (envoyproxy#218) * Update commitid of cloudendpoints/service-control-client-cxx repo (envoyproxy#220)
rshriram · Apr 7, 2017 · 1ef1b4a · 1ef1b4a
1 parent 987223f
commit 1ef1b4a
Show file tree

Hide file tree

Showing 32 changed files with 1,145 additions and 51 deletions.
diff --git a/contrib/endpoints/include/api_manager/method.h b/contrib/endpoints/include/api_manager/method.h
@@ -89,6 +89,10 @@ class MethodInfo {
 
   // Get the names of url system parameters
   virtual const std::set<std::string> &system_query_parameter_names() const = 0;
+
+  // Get quota metric cost vector
+  virtual const std::vector<std::pair<std::string, int>> &metric_cost_vector()
+      const = 0;
 };
 
 }  // namespace api_manager

diff --git a/contrib/endpoints/repositories.bzl b/contrib/endpoints/repositories.bzl
@@ -211,12 +211,9 @@ def googleapis_repositories(protobuf_repo="@protobuf_git//", bind=True):
 #
 ################################################################################
 #
-
 licenses(["notice"])
 
-load("{}:protobuf.bzl", "cc_proto_library")
-
-exports_files(glob(["google/**"]))
+load("@protobuf_git//:protobuf.bzl", "cc_proto_library")
 
 cc_proto_library(
     name = "servicecontrol",
@@ -259,9 +256,13 @@ cc_proto_library(
         "google/api/log.proto",
         "google/api/logging.proto",
         "google/api/metric.proto",
+        "google/api/experimental/experimental.proto",
+        "google/api/experimental/authorization_config.proto",
         "google/api/monitored_resource.proto",
         "google/api/monitoring.proto",
+        "google/api/quota.proto",
         "google/api/service.proto",
+        "google/api/source_info.proto",
         "google/api/system_parameter.proto",
         "google/api/usage.proto",
     ],
@@ -290,10 +291,9 @@ cc_proto_library(
 )
 """.format(protobuf_repo)
 
-
     native.new_git_repository(
         name = "googleapis_git",
-        commit = "db1d4547dc56a798915e0eb2c795585385922165",
+        commit = "2fe0050bd2a6d4c6ba798c0311f0b149b8997314",
         remote = "https://github.com/googleapis/googleapis.git",
         build_file_content = BUILD,
     )
@@ -324,7 +324,7 @@ def servicecontrol_client_repositories(bind=True):
 
     native.git_repository(
         name = "servicecontrol_client_git",
-        commit = "d739d755365c6a13d0b4164506fd593f53932f5d",
+        commit = "3d1a30d9221e700542eeaaf20eab69faddb63894",
         remote = "https://github.com/cloudendpoints/service-control-client-cxx.git",
     )
 
@@ -333,3 +333,11 @@ def servicecontrol_client_repositories(bind=True):
             name = "servicecontrol_client",
             actual = "@servicecontrol_client_git//:service_control_client_lib",
         )
+        native.bind(
+            name = "quotacontrol",
+            actual = "@servicecontrol_client_git//proto:quotacontrol",
+        )
+        native.bind(
+            name = "quotacontrol_genproto",
+            actual = "@servicecontrol_client_git//proto:quotacontrol_genproto",
+        )
diff --git a/contrib/endpoints/src/api_manager/BUILD b/contrib/endpoints/src/api_manager/BUILD
@@ -81,6 +81,8 @@ cc_library(
         "method_impl.cc",
         "path_matcher.cc",
         "path_matcher_node.cc",
+        "quota_control.cc",
+        "quota_control.h",
         "request_handler.cc",
     ],
     linkopts = select({

diff --git a/contrib/endpoints/src/api_manager/auth/service_account_token.h b/contrib/endpoints/src/api_manager/auth/service_account_token.h
@@ -64,6 +64,7 @@ class ServiceAccountToken {
   enum JWT_TOKEN_TYPE {
     JWT_TOKEN_FOR_SERVICE_CONTROL = 0,
     JWT_TOKEN_FOR_CLOUD_TRACING,
+    JWT_TOKEN_FOR_QUOTA_CONTROL,
     JWT_TOKEN_TYPE_MAX,
   };
   // Set audience.  Only calcualtes JWT token with specified audience.

diff --git a/contrib/endpoints/src/api_manager/check_workflow.cc b/contrib/endpoints/src/api_manager/check_workflow.cc
@@ -18,6 +18,7 @@
 #include "contrib/endpoints/src/api_manager/check_auth.h"
 #include "contrib/endpoints/src/api_manager/check_service_control.h"
 #include "contrib/endpoints/src/api_manager/fetch_metadata.h"
+#include "contrib/endpoints/src/api_manager/quota_control.h"
 
 using ::google::api_manager::utils::Status;
 
@@ -33,6 +34,8 @@ void CheckWorkflow::RegisterAll() {
   Register(CheckAuth);
   // Checks service control.
   Register(CheckServiceControl);
+  // Quota control
+  Register(QuotaControl);
 }
 
 void CheckWorkflow::Register(CheckHandler handler) {

diff --git a/contrib/endpoints/src/api_manager/config.cc b/contrib/endpoints/src/api_manager/config.cc
@@ -113,6 +113,23 @@ MethodInfoImpl *Config::GetOrCreateMethodInfoImpl(const string &name,
   return i->second.get();
 }
 
+bool Config::LoadQuotaRule(ApiManagerEnvInterface *env) {
+  for (const auto &rule : service_.quota().metric_rules()) {
+    auto method = utils::FindOrNull(method_map_, rule.selector());
+    if (method) {
+      for (auto &metric_cost : rule.metric_costs()) {
+        (*method)->add_metric_cost(metric_cost.first, metric_cost.second);
+      }
+    } else {
+      env->LogError("Metric rule with selector " + rule.selector() +
+                    "is mismatched.");
+      return false;
+    }
+  }
+
+  return true;
+}
+
 bool Config::LoadHttpMethods(ApiManagerEnvInterface *env,
                              PathMatcherBuilder *pmb) {
   std::set<std::string> all_urls, urls_with_options;
@@ -443,6 +460,9 @@ std::unique_ptr<Config> Config::Create(ApiManagerEnvInterface *env,
   if (!config->LoadBackends(env)) {
     return nullptr;
   }
+  if (!config->LoadQuotaRule(env)) {
+    return nullptr;
+  }
   return config;
 }
 

diff --git a/contrib/endpoints/src/api_manager/config.h b/contrib/endpoints/src/api_manager/config.h
@@ -25,6 +25,7 @@
 #include "contrib/endpoints/src/api_manager/method_impl.h"
 #include "contrib/endpoints/src/api_manager/path_matcher.h"
 #include "contrib/endpoints/src/api_manager/proto/server_config.pb.h"
+#include "google/api/quota.pb.h"
 #include "google/api/service.pb.h"
 
 namespace google {
@@ -111,6 +112,8 @@ class Config {
   // Load SystemParameters info to MethodInfo.
   bool LoadSystemParameters(ApiManagerEnvInterface *env);
 
+  bool LoadQuotaRule(ApiManagerEnvInterface *env);
+
   // Gets the MethodInfoImpl creating it if necessary
   MethodInfoImpl *GetOrCreateMethodInfoImpl(const std::string &name,
                                             const std::string &api_name,

diff --git a/contrib/endpoints/src/api_manager/config_test.cc b/contrib/endpoints/src/api_manager/config_test.cc
@@ -870,6 +870,90 @@ TEST(Config, TestCorsDisabled) {
   ASSERT_EQ(nullptr, method1);
 }
 
+TEST(Config, TestInvalidMetricRules) {
+  MockApiManagerEnvironmentWithLog env;
+  // There is no http.rule or api.method to match the selector.
+  static const char config_text[] = R"(
+name: "Service.Name"
+quota {
+   metric_rules {
+      selector: "GetShelves"
+      metric_costs {
+         key: "test.googleapis.com/operation/read_book"
+         value: 100
+      }
+   }
+}
+)";
+
+  std::unique_ptr<Config> config = Config::Create(&env, config_text, "");
+  EXPECT_EQ(nullptr, config);
+}
+
+TEST(Config, TestMetricRules) {
+  MockApiManagerEnvironmentWithLog env;
+  static const char config_text[] = R"(
+name: "Service.Name"
+http {
+  rules {
+   selector: "DeleteShelf"
+   delete: "/shelves"
+  }
+  rules {
+   selector: "GetShelves"
+   get: "/shelves"
+  }
+}
+quota {
+   metric_rules {
+      selector: "GetShelves"
+      metric_costs {
+         key: "test.googleapis.com/operation/get_shelves"
+         value: 100
+      }
+      metric_costs {
+          key: "test.googleapis.com/operation/request"
+          value: 10
+       }
+   }
+   metric_rules {
+      selector: "DeleteShelf"
+      metric_costs {
+         key: "test.googleapis.com/operation/delete_shelves"
+         value: 200
+      }
+   }
+}
+)";
+
+  std::unique_ptr<Config> config = Config::Create(&env, config_text, "");
+  ASSERT_TRUE(config);
+
+  const MethodInfo *method1 = config->GetMethodInfo("GET", "/shelves");
+  ASSERT_NE(nullptr, method1);
+
+  std::vector<std::pair<std::string, int>> metric_cost_vector =
+      method1->metric_cost_vector();
+  std::sort(metric_cost_vector.begin(), metric_cost_vector.end());
+  ASSERT_EQ(2, metric_cost_vector.size());
+  ASSERT_EQ("test.googleapis.com/operation/get_shelves",
+            metric_cost_vector[0].first);
+  ASSERT_EQ(100, metric_cost_vector[0].second);
+
+  ASSERT_EQ("test.googleapis.com/operation/request",
+            metric_cost_vector[1].first);
+  ASSERT_EQ(10, metric_cost_vector[1].second);
+
+  const MethodInfo *method2 = config->GetMethodInfo("DELETE", "/shelves");
+  ASSERT_NE(nullptr, method1);
+
+  metric_cost_vector = method2->metric_cost_vector();
+  ASSERT_EQ(1, metric_cost_vector.size());
+  ASSERT_EQ("test.googleapis.com/operation/delete_shelves",
+            metric_cost_vector[0].first);
+  ASSERT_EQ(200, metric_cost_vector[0].second);
+}
+
 }  // namespace
 
 }  // namespace api_manager

diff --git a/contrib/endpoints/src/api_manager/context/request_context.cc b/contrib/endpoints/src/api_manager/context/request_context.cc
@@ -241,6 +241,16 @@ void RequestContext::FillCheckRequestInfo(
   request_->FindHeader(kXIosBundleId, &info->ios_bundle_id);
 }
 
+void RequestContext::FillAllocateQuotaRequestInfo(
+    service_control::QuotaRequestInfo *info) {
+  FillOperationInfo(info);
+
+  info->client_ip = request_->GetClientIP();
+  info->method_name = this->method_call_.method_info->name();
+  info->metric_cost_vector =
+      &this->method_call_.method_info->metric_cost_vector();
+}
+
 void RequestContext::FillReportRequestInfo(
     Response *response, service_control::ReportRequestInfo *info) {
   FillOperationInfo(info);

diff --git a/contrib/endpoints/src/api_manager/context/request_context.h b/contrib/endpoints/src/api_manager/context/request_context.h
@@ -66,6 +66,9 @@ class RequestContext {
   // Fill CheckRequestInfo
   void FillCheckRequestInfo(service_control::CheckRequestInfo *info);
 
+  // FillAllocateQuotaRequestInfo
+  void FillAllocateQuotaRequestInfo(service_control::QuotaRequestInfo *info);
+
   // Fill ReportRequestInfo
   void FillReportRequestInfo(Response *response,
                              service_control::ReportRequestInfo *info);

diff --git a/contrib/endpoints/src/api_manager/method_impl.h b/contrib/endpoints/src/api_manager/method_impl.h
@@ -18,6 +18,7 @@
 #include <map>
 #include <memory>
 #include <set>
+#include <vector>
 
 #include "contrib/endpoints/include/api_manager/method.h"
 #include "contrib/endpoints/src/api_manager/utils/stl_util.h"
@@ -62,6 +63,10 @@ class MethodInfoImpl : public MethodInfo {
 
   const std::string &backend_address() const { return backend_address_; }
 
+  const std::vector<std::pair<std::string, int>> &metric_cost_vector() const {
+    return metric_cost_vector_;
+  }
+
   const std::string &rpc_method_full_name() const {
     return rpc_method_full_name_;
   }
@@ -90,6 +95,10 @@ class MethodInfoImpl : public MethodInfo {
     url_query_parameters_[name].push_back(url_query_parameter);
   }
 
+  void add_metric_cost(const std::string &metric, int64_t cost) {
+    metric_cost_vector_.push_back(std::make_pair(metric, cost));
+  }
+
   // After add all system parameters, lookup some of them to cache
   // their lookup results.
   void process_system_parameters();
@@ -139,13 +148,13 @@ class MethodInfoImpl : public MethodInfo {
   // such as API Key)?
   bool allow_unregistered_calls_;
   // Issuers to allowed audiences map.
-  std::map<std::string, std::set<std::string> > issuer_audiences_map_;
+  std::map<std::string, std::set<std::string>> issuer_audiences_map_;
 
   // system parameter map of parameter name to http_header name.
-  std::map<std::string, std::vector<std::string> > http_header_parameters_;
+  std::map<std::string, std::vector<std::string>> http_header_parameters_;
 
   // system parameter map of parameter name to url query parameter name.
-  std::map<std::string, std::vector<std::string> > url_query_parameters_;
+  std::map<std::string, std::vector<std::string>> url_query_parameters_;
 
   // all the names of system query parameters
   std::set<std::string> system_query_parameter_names_;
@@ -175,6 +184,9 @@ class MethodInfoImpl : public MethodInfo {
 
   // Whether the response is streaming or not.
   bool response_streaming_;
+
+  // map of metric and its cost
+  std::vector<std::pair<std::string, int>> metric_cost_vector_;
 };
 
 typedef std::unique_ptr<MethodInfoImpl> MethodInfoImplPtr;

diff --git a/contrib/endpoints/src/api_manager/mock_method_info.h b/contrib/endpoints/src/api_manager/mock_method_info.h
@@ -48,6 +48,8 @@ class MockMethodInfo : public MethodInfo {
   MOCK_CONST_METHOD0(response_streaming, bool());
   MOCK_CONST_METHOD0(system_query_parameter_names,
                      const std::set<std::string>&());
+  MOCK_CONST_METHOD0(metric_cost_vector,
+                     const std::vector<std::pair<std::string, int>>&());
 };
 
 }  // namespace api_manager

diff --git a/contrib/endpoints/src/api_manager/proto/server_config.proto b/contrib/endpoints/src/api_manager/proto/server_config.proto
@@ -66,6 +66,13 @@ message ServiceControlConfig {
   // The intermediate reports for streaming calls should not be more frequent
   // than this value (in seconds)
   int32 intermediate_report_min_interval = 7;
+
+  // Quota aggregator config
+  QuotaAggregatorConfig quota_aggregator_config = 8;
+
+  // Timeout in milliseconds on service control allocate quota requests.
+  // If the value is <= 0, default timeout is 5000 milliseconds.
+  int32 quota_timeout_ms = 9;
 }
 
 // Check aggregator config
@@ -82,6 +89,17 @@ message CheckAggregatorConfig {
   int32 response_expiration_ms = 3;
 }
 
+// Quota aggregator config
+message QuotaAggregatorConfig {
+  // The maximum number of cache entries that can be kept in the aggregation
+  // cache. Cache is disabled when entries <= 0.
+  int32 cache_entries = 1;
+
+  // The maximum milliseconds before aggregated quota requests are refreshed to
+  // the server.
+  int32 refresh_interval_ms = 2;
+}
+
 // Report aggregator config
 message ReportAggregatorConfig {
   // The maximum number of cache entries that can be kept in the aggregation